Warning, /frameworks/syndication/autotests/atom/bug411626.xml.expected is written in an unsupported language. File is not indexed.
0001 # Feed begin ######################
0002 title: #Real Python#
0003 link: #https://realpython.com/#
0004 # Person begin ####################
0005 name: #Real Python#
0006 # Person end ######################
0007 # Item begin ######################
0008 id: #https://realpython.com/python-kwargs-and-args/#
0009 title: #Python args and kwargs: Demystified#
0010 link: #https://realpython.com/python-kwargs-and-args/#
0011 description: #In this step-by-step tutorial, you'll learn how to use args and kwargs in Python to add more flexibility to your functions. You'll also take a closer look at the single and double-asterisk unpacking operators, which you can use to unpack any iterable object in Python.#
0012 content: #<p>Sometimes, when you look at a function definition in Python, you might see that it takes two strange arguments: <strong><code>*args</code></strong> and <strong><code>**kwargs</code></strong>. If you’ve ever wondered what these peculiar variables are, or why your IDE defines them in <code>main()</code>, then this article is for you. You’ll learn how to use args and kwargs in Python to add more flexibility to your functions.</p>
0013 <p><strong>By the end of the article, you’ll know:</strong></p>
0014 <ul>
0015 <li>What <code>*args</code> and <code>**kwargs</code> actually mean</li>
0016 <li>How to use <code>*args</code> and <code>**kwargs</code> in function definitions</li>
0017 <li>How to use a single asterisk (<code>*</code>) to unpack iterables</li>
0018 <li>How to use two asterisks (<code>**</code>) to unpack dictionaries</li>
0019 </ul>
0020 <p>This article assumes that you already know how to define Python functions and work with <a href="https://realpython.com/lessons/mutable-data-structures-lists-dictionaries/">lists and dictionaries</a>.</p>
0021 <div class="alert alert-warning" role="alert"><p><strong>Free Bonus:</strong> <a href="" class="alert-link" data-toggle="modal" data-target="#modal-python-cheat-sheet-shortened" data-focus="false">Click here to get a Python Cheat Sheet</a> and learn the basics of Python 3, like working with data types, dictionaries, lists, and Python functions.</p></div>
0022
0023 <h2 id="passing-multiple-arguments-to-a-function">Passing Multiple Arguments to a Function</h2>
0024 <p><strong><code>*args</code></strong> and <strong><code>**kwargs</code></strong> allow you to pass multiple arguments or keyword arguments to a function. Consider the following example. This is a simple function that takes two arguments and returns their sum:</p>
0025 <div class="highlight python"><pre><span></span><span class="k">def</span> <span class="nf">my_sum</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">):</span>
0026 <span class="k">return</span> <span class="n">a</span> <span class="o">+</span> <span class="n">b</span>
0027 </pre></div>
0028
0029 <p>This function works fine, but it’s limited to only two arguments. What if you need to sum a varying number of arguments, where the specific number of arguments passed is only determined at runtime? Wouldn’t it be great to create a function that could sum <em>all</em> the integers passed to it, no matter how many there are?</p>
0030 <h2 id="using-the-python-args-variable-in-function-definitions">Using the Python args Variable in Function Definitions</h2>
0031 <p>There are a few ways you can pass a varying number of arguments to a function. The first way is often the most intuitive for people that have experience with collections. You simply pass a list or a <a href="https://realpython.com/python-sets/">set</a> of all the arguments to your function. So for <code>my_sum()</code>, you could pass a list of all the integers you need to add:</p>
0032 <div class="highlight python"><pre><span></span><span class="c1"># sum_integers_list.py</span>
0033 <span class="k">def</span> <span class="nf">my_sum</span><span class="p">(</span><span class="n">my_integers</span><span class="p">):</span>
0034 <span class="n">result</span> <span class="o">=</span> <span class="mi">0</span>
0035 <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">my_integers</span><span class="p">:</span>
0036 <span class="n">result</span> <span class="o">+=</span> <span class="n">x</span>
0037 <span class="k">return</span> <span class="n">result</span>
0038
0039 <span class="n">list_of_integers</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span>
0040 <span class="nb">print</span><span class="p">(</span><span class="n">my_sum</span><span class="p">(</span><span class="n">list_of_integers</span><span class="p">))</span>
0041 </pre></div>
0042
0043 <p>This implementation works, but whenever you call this function you’ll also need to create a list of arguments to pass to it. This can be inconvenient, especially if you don’t know up front all the values that should go into the list.</p>
0044 <p>This is where <code>*args</code> can be really useful, because it allows you to pass a varying number of positional arguments. Take the following example:</p>
0045 <div class="highlight python"><pre><span></span><span class="c1"># sum_integers_args.py</span>
0046 <span class="k">def</span> <span class="nf">my_sum</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">):</span>
0047 <span class="n">result</span> <span class="o">=</span> <span class="mi">0</span>
0048 <span class="c1"># Iterating over the Python args tuple</span>
0049 <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">args</span><span class="p">:</span>
0050 <span class="n">result</span> <span class="o">+=</span> <span class="n">x</span>
0051 <span class="k">return</span> <span class="n">result</span>
0052
0053 <span class="nb">print</span><span class="p">(</span><span class="n">my_sum</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">))</span>
0054 </pre></div>
0055
0056 <p>In this example, you’re no longer passing a list to <code>my_sum()</code>. Instead, you’re passing three different positional arguments. <code>my_sum()</code> takes all the parameters that are provided in the input and packs them all into a single iterable object named <code>args</code>.</p>
0057 <p>Note that <strong><code>args</code> is just a name.</strong> You’re not required to use the name <code>args</code>. You can choose any name that you prefer, such as <code>integers</code>:</p>
0058 <div class="highlight python"><pre><span></span><span class="c1"># sum_integers_args_2.py</span>
0059 <span class="k">def</span> <span class="nf">my_sum</span><span class="p">(</span><span class="o">*</span><span class="n">integers</span><span class="p">):</span>
0060 <span class="n">result</span> <span class="o">=</span> <span class="mi">0</span>
0061 <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">integers</span><span class="p">:</span>
0062 <span class="n">result</span> <span class="o">+=</span> <span class="n">x</span>
0063 <span class="k">return</span> <span class="n">result</span>
0064
0065 <span class="nb">print</span><span class="p">(</span><span class="n">my_sum</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">))</span>
0066 </pre></div>
0067
0068 <p>The function still works, even if you pass the iterable object as <code>integers</code> instead of <code>args</code>. All that matters here is that you use the <strong>unpacking operator</strong> (<code>*</code>).</p>
0069 <p>Bear in mind that the iterable object you’ll get using the unpacking operator <code>*</code> is <a href="https://realpython.com/python-lists-tuples/">not a <code>list</code> but a <code>tuple</code></a>. A <code>tuple</code> is similar to a <code>list</code> in that they both support slicing and iteration. However, tuples are very different in at least one aspect: lists are <a href="https://realpython.com/courses/immutability-python/">mutable</a>, while tuples are not. To test this, run the following code. This script tries to change a value of a list:</p>
0070 <div class="highlight python"><pre><span></span><span class="c1"># change_list.py</span>
0071 <span class="n">my_list</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span>
0072 <span class="n">my_list</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="mi">9</span>
0073 <span class="nb">print</span><span class="p">(</span><span class="n">my_list</span><span class="p">)</span>
0074 </pre></div>
0075
0076 <p>The value located at the very first index of the list should be updated to <code>9</code>. If you execute this script, you will see that the list indeed gets modified:</p>
0077 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python change_list.py
0078 <span class="go">[9, 2, 3]</span>
0079 </pre></div>
0080
0081 <p>The first value is no longer <code>0</code>, but the updated value <code>9</code>. Now, try to do the same with a tuple:</p>
0082 <div class="highlight python"><pre><span></span><span class="c1"># change_tuple.py</span>
0083 <span class="n">my_tuple</span> <span class="o">=</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span>
0084 <span class="n">my_tuple</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="mi">9</span>
0085 <span class="nb">print</span><span class="p">(</span><span class="n">my_tuple</span><span class="p">)</span>
0086 </pre></div>
0087
0088 <p>Here, you see the same values, except they’re held together as a tuple. If you try to execute this script, you will see that the Python interpreter returns an <a href="https://realpython.com/python-exceptions/">error</a>:</p>
0089 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python change_tuple.py
0090 <span class="go">Traceback (most recent call last):</span>
0091 <span class="go"> File "change_tuple.py", line 3, in <module></span>
0092 <span class="go"> my_tuple[0] = 9</span>
0093 <span class="go">TypeError: 'tuple' object does not support item assignment</span>
0094 </pre></div>
0095
0096 <p>This is because a tuple is an immutable object, and its values cannot be changed after assignment. Keep this in mind when you’re working with tuples and <code>*args</code>.</p>
0097 <h2 id="using-the-python-kwargs-variable-in-function-definitions">Using the Python kwargs Variable in Function Definitions</h2>
0098 <p>Okay, now you’ve understood what <code>*args</code> is for, but what about <code>**kwargs</code>? <code>**kwargs</code> works just like <code>*args</code>, but instead of accepting positional arguments it accepts keyword (or <strong>named</strong>) arguments. Take the following example:</p>
0099 <div class="highlight python"><pre><span></span><span class="c1"># concatenate.py</span>
0100 <span class="k">def</span> <span class="nf">concatenate</span><span class="p">(</span><span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
0101 <span class="n">result</span> <span class="o">=</span> <span class="s2">""</span>
0102 <span class="c1"># Iterating over the Python kwargs dictionary</span>
0103 <span class="k">for</span> <span class="n">arg</span> <span class="ow">in</span> <span class="n">kwargs</span><span class="o">.</span><span class="n">values</span><span class="p">():</span>
0104 <span class="n">result</span> <span class="o">+=</span> <span class="n">arg</span>
0105 <span class="k">return</span> <span class="n">result</span>
0106
0107 <span class="nb">print</span><span class="p">(</span><span class="n">concatenate</span><span class="p">(</span><span class="n">a</span><span class="o">=</span><span class="s2">"Real"</span><span class="p">,</span> <span class="n">b</span><span class="o">=</span><span class="s2">"Python"</span><span class="p">,</span> <span class="n">c</span><span class="o">=</span><span class="s2">"Is"</span><span class="p">,</span> <span class="n">d</span><span class="o">=</span><span class="s2">"Great"</span><span class="p">,</span> <span class="n">e</span><span class="o">=</span><span class="s2">"!"</span><span class="p">))</span>
0108 </pre></div>
0109
0110 <p>When you execute the script above, <code>concatenate()</code> will iterate through the Python kwargs <a href="https://realpython.com/python-dicts/">dictionary</a> and concatenate all the values it finds:</p>
0111 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python concatenate.py
0112 <span class="go">RealPythonIsGreat!</span>
0113 </pre></div>
0114
0115 <p>Like <code>args</code>, <code>kwargs</code> is just a name that can be changed to whatever you want. Again, what is important here is the use of the <strong>unpacking operator</strong> (<code>**</code>).</p>
0116 <p>So, the previous example could be written like this:</p>
0117 <div class="highlight python"><pre><span></span><span class="c1"># concatenate_2.py</span>
0118 <span class="k">def</span> <span class="nf">concatenate</span><span class="p">(</span><span class="o">**</span><span class="n">words</span><span class="p">):</span>
0119 <span class="n">result</span> <span class="o">=</span> <span class="s2">""</span>
0120 <span class="k">for</span> <span class="n">arg</span> <span class="ow">in</span> <span class="n">words</span><span class="o">.</span><span class="n">values</span><span class="p">():</span>
0121 <span class="n">result</span> <span class="o">+=</span> <span class="n">arg</span>
0122 <span class="k">return</span> <span class="n">result</span>
0123
0124 <span class="nb">print</span><span class="p">(</span><span class="n">concatenate</span><span class="p">(</span><span class="n">a</span><span class="o">=</span><span class="s2">"Real"</span><span class="p">,</span> <span class="n">b</span><span class="o">=</span><span class="s2">"Python"</span><span class="p">,</span> <span class="n">c</span><span class="o">=</span><span class="s2">"Is"</span><span class="p">,</span> <span class="n">d</span><span class="o">=</span><span class="s2">"Great"</span><span class="p">,</span> <span class="n">e</span><span class="o">=</span><span class="s2">"!"</span><span class="p">))</span>
0125 </pre></div>
0126
0127 <p>Note that in the example above the iterable object is a standard <code>dict</code>. If you <a href="https://realpython.com/iterate-through-dictionary-python/">iterate over the dictionary</a> and want to return its values, like in the example shown, then you must use <code>.values()</code>.</p>
0128 <p>In fact, if you forget to use this method, you will find yourself iterating through the <strong>keys</strong> of your Python kwargs dictionary instead, like in the following example:</p>
0129 <div class="highlight python"><pre><span></span><span class="c1"># concatenate_keys.py</span>
0130 <span class="k">def</span> <span class="nf">concatenate</span><span class="p">(</span><span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
0131 <span class="n">result</span> <span class="o">=</span> <span class="s2">""</span>
0132 <span class="c1"># Iterating over the keys of the Python kwargs dictionary</span>
0133 <span class="k">for</span> <span class="n">arg</span> <span class="ow">in</span> <span class="n">kwargs</span><span class="p">:</span>
0134 <span class="n">result</span> <span class="o">+=</span> <span class="n">arg</span>
0135 <span class="k">return</span> <span class="n">result</span>
0136
0137 <span class="nb">print</span><span class="p">(</span><span class="n">concatenate</span><span class="p">(</span><span class="n">a</span><span class="o">=</span><span class="s2">"Real"</span><span class="p">,</span> <span class="n">b</span><span class="o">=</span><span class="s2">"Python"</span><span class="p">,</span> <span class="n">c</span><span class="o">=</span><span class="s2">"Is"</span><span class="p">,</span> <span class="n">d</span><span class="o">=</span><span class="s2">"Great"</span><span class="p">,</span> <span class="n">e</span><span class="o">=</span><span class="s2">"!"</span><span class="p">))</span>
0138 </pre></div>
0139
0140 <p>Now, if you try to execute this example, you’ll notice the following output:</p>
0141 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python concatenate_keys.py
0142 <span class="go">abcde</span>
0143 </pre></div>
0144
0145 <p>As you can see, if you don’t specify <code>.values()</code>, your function will iterate over the keys of your Python kwargs dictionary, returning the wrong result.</p>
0146 <h2 id="ordering-arguments-in-a-function">Ordering Arguments in a Function</h2>
0147 <p>Now that you have learned what <code>*args</code> and <code>**kwargs</code> are for, you are ready to start writing functions that take a varying number of input arguments. But what if you want to create a function that takes a changeable number of both positional <em>and</em> named arguments?</p>
0148 <p>In this case, you have to bear in mind that <strong>order counts</strong>. Just as non-default arguments have to precede default arguments, so <code>*args</code> must come before <code>**kwargs</code>.</p>
0149 <p>To recap, the correct order for your parameters is:</p>
0150 <ol>
0151 <li>Standard arguments</li>
0152 <li><code>*args</code> arguments</li>
0153 <li><code>**kwargs</code> arguments</li>
0154 </ol>
0155 <p>For example, this function definition is correct:</p>
0156 <div class="highlight python"><pre><span></span><span class="c1"># correct_function_definition.py</span>
0157 <span class="k">def</span> <span class="nf">my_function</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
0158 <span class="k">pass</span>
0159 </pre></div>
0160
0161 <p>The <code>*args</code> variable is appropriately listed before <code>**kwargs</code>. But what if you try to modify the order of the arguments? For example, consider the following function:</p>
0162 <div class="highlight python"><pre><span></span><span class="c1"># wrong_function_definition.py</span>
0163 <span class="k">def</span> <span class="nf">my_function</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">):</span>
0164 <span class="k">pass</span>
0165 </pre></div>
0166
0167 <p>Now, <code>**kwargs</code> comes before <code>*args</code> in the function definition. If you try to run this example, you’ll receive an error from the interpreter:</p>
0168 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python wrong_function_definition.py
0169 <span class="go"> File "wrong_function_definition.py", line 2</span>
0170 <span class="go"> def my_function(a, b, **kwargs, *args):</span>
0171 <span class="go"> ^</span>
0172 <span class="go">SyntaxError: invalid syntax</span>
0173 </pre></div>
0174
0175 <p>In this case, since <code>*args</code> comes after <code>**kwargs</code>, the Python interpreter throws a <code>SyntaxError</code>.</p>
0176 <h2 id="unpacking-with-the-asterisk-operators">Unpacking With the Asterisk Operators: <code>*</code> & <code>**</code></h2>
0177 <p>You are now able to use <code>*args</code> and <code>**kwargs</code> to define Python functions that take a varying number of input arguments. Let’s go a little deeper to understand something more about the <strong>unpacking operators</strong>.</p>
0178 <p>The single and double asterisk unpacking operators were introduced in Python 2. As of the 3.5 release, they have become even more powerful, thanks to <a href="https://www.python.org/dev/peps/pep-0448/">PEP 448</a>. In short, the unpacking operators are operators that unpack the values from iterable objects in Python. The single asterisk operator <code>*</code> can be used on any iterable that Python provides, while the double asterisk operator <code>**</code> can only be used on dictionaries.</p>
0179 <p>Let’s start with an example:</p>
0180 <div class="highlight python"><pre><span></span><span class="c1"># print_list.py</span>
0181 <span class="n">my_list</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span>
0182 <span class="nb">print</span><span class="p">(</span><span class="n">my_list</span><span class="p">)</span>
0183 </pre></div>
0184
0185 <p>This code defines a list and then prints it to the standard output:</p>
0186 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python print_list.py
0187 <span class="go">[1, 2, 3]</span>
0188 </pre></div>
0189
0190 <p>Note how the list is printed, along with the corresponding brackets and commas.</p>
0191 <p>Now, try to prepend the unpacking operator <code>*</code> to the name of your list:</p>
0192 <div class="highlight python"><pre><span></span><span class="c1"># print_unpacked_list.py</span>
0193 <span class="n">my_list</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span>
0194 <span class="nb">print</span><span class="p">(</span><span class="o">*</span><span class="n">my_list</span><span class="p">)</span>
0195 </pre></div>
0196
0197 <p>Here, the <code>*</code> operator tells <code>print()</code> to unpack the list first.</p>
0198 <p>In this case, the output is no longer the list itself, but rather <em>the content</em> of the list:</p>
0199 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python print_unpacked_list.py
0200 <span class="go">1 2 3</span>
0201 </pre></div>
0202
0203 <p>Can you see the difference between this execution and the one from <code>print_list.py</code>? Instead of a list, <code>print()</code> has taken three separate arguments as the input.</p>
0204 <p>Another thing you’ll notice is that in <code>print_unpacked_list.py</code>, you used the unpacking operator <code>*</code> to call a function, instead of in a function definition. In this case, <code>print()</code> takes all the items of a list as though they were single arguments.</p>
0205 <p>You can also use this method to call your own functions, but if your function requires a specific number of arguments, then the iterable you unpack must have the same number of arguments.</p>
0206 <p>To test this behavior, consider this script:</p>
0207 <div class="highlight python"><pre><span></span><span class="c1"># unpacking_call.py</span>
0208 <span class="k">def</span> <span class="nf">my_sum</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="n">c</span><span class="p">):</span>
0209 <span class="nb">print</span><span class="p">(</span><span class="n">a</span> <span class="o">+</span> <span class="n">b</span> <span class="o">+</span> <span class="n">c</span><span class="p">)</span>
0210
0211 <span class="n">my_list</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span>
0212 <span class="n">my_sum</span><span class="p">(</span><span class="o">*</span><span class="n">my_list</span><span class="p">)</span>
0213 </pre></div>
0214
0215 <p>Here, <code>my_sum()</code> explicitly states that <code>a</code>, <code>b</code>, and <code>c</code> are required arguments.</p>
0216 <p>If you run this script, you’ll get the sum of the three numbers in <code>my_list</code>:</p>
0217 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python unpacking_call.py
0218 <span class="go">6</span>
0219 </pre></div>
0220
0221 <p>The 3 elements in <code>my_list</code> match up perfectly with the required arguments in <code>my_sum()</code>.</p>
0222 <p>Now look at the following script, where <code>my_list</code> has 4 arguments instead of 3:</p>
0223 <div class="highlight python"><pre><span></span><span class="c1"># wrong_unpacking_call.py</span>
0224 <span class="k">def</span> <span class="nf">my_sum</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="n">c</span><span class="p">):</span>
0225 <span class="nb">print</span><span class="p">(</span><span class="n">a</span> <span class="o">+</span> <span class="n">b</span> <span class="o">+</span> <span class="n">c</span><span class="p">)</span>
0226
0227 <span class="n">my_list</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">]</span>
0228 <span class="n">my_sum</span><span class="p">(</span><span class="o">*</span><span class="n">my_list</span><span class="p">)</span>
0229 </pre></div>
0230
0231 <p>In this example, <code>my_sum()</code> still expects just three arguments, but the <code>*</code> operator gets 4 items from the list. If you try to execute this script, you’ll see that the Python interpreter is unable to run it:</p>
0232 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python wrong_unpacking_call.py
0233 <span class="go">Traceback (most recent call last):</span>
0234 <span class="go"> File "wrong_unpacking_call.py", line 6, in <module></span>
0235 <span class="go"> my_sum(*my_list)</span>
0236 <span class="go">TypeError: my_sum() takes 3 positional arguments but 4 were given</span>
0237 </pre></div>
0238
0239 <p>When you use the <code>*</code> operator to unpack a list and pass arguments to a function, it’s exactly as though you’re passing every single argument alone. This means that you can use multiple unpacking operators to get values from several lists and pass them all to a single function.</p>
0240 <p>To test this behavior, consider the following example:</p>
0241 <div class="highlight python"><pre><span></span><span class="c1"># sum_integers_args_3.py</span>
0242 <span class="k">def</span> <span class="nf">my_sum</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">):</span>
0243 <span class="n">result</span> <span class="o">=</span> <span class="mi">0</span>
0244 <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">args</span><span class="p">:</span>
0245 <span class="n">result</span> <span class="o">+=</span> <span class="n">x</span>
0246 <span class="k">return</span> <span class="n">result</span>
0247
0248 <span class="n">list1</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span>
0249 <span class="n">list2</span> <span class="o">=</span> <span class="p">[</span><span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">]</span>
0250 <span class="n">list3</span> <span class="o">=</span> <span class="p">[</span><span class="mi">6</span><span class="p">,</span> <span class="mi">7</span><span class="p">,</span> <span class="mi">8</span><span class="p">,</span> <span class="mi">9</span><span class="p">]</span>
0251
0252 <span class="nb">print</span><span class="p">(</span><span class="n">my_sum</span><span class="p">(</span><span class="o">*</span><span class="n">list1</span><span class="p">,</span> <span class="o">*</span><span class="n">list2</span><span class="p">,</span> <span class="o">*</span><span class="n">list3</span><span class="p">))</span>
0253 </pre></div>
0254
0255 <p>If you run this example, all three lists are unpacked. Each individual item is passed to <code>my_sum()</code>, resulting in the following output:</p>
0256 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python sum_integers_args_3.py
0257 <span class="go">45</span>
0258 </pre></div>
0259
0260 <p>There are other convenient uses of the unpacking operator. For example, say you need to split a list into three different parts. The output should show the first value, the last value, and all the values in between. With the unpacking operator, you can do this in just one line of code:</p>
0261 <div class="highlight python"><pre><span></span><span class="c1"># extract_list_body.py</span>
0262 <span class="n">my_list</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">6</span><span class="p">]</span>
0263
0264 <span class="n">a</span><span class="p">,</span> <span class="o">*</span><span class="n">b</span><span class="p">,</span> <span class="n">c</span> <span class="o">=</span> <span class="n">my_list</span>
0265
0266 <span class="nb">print</span><span class="p">(</span><span class="n">a</span><span class="p">)</span>
0267 <span class="nb">print</span><span class="p">(</span><span class="n">b</span><span class="p">)</span>
0268 <span class="nb">print</span><span class="p">(</span><span class="n">c</span><span class="p">)</span>
0269 </pre></div>
0270
0271 <p>In this example, <code>my_list</code> contains 6 items. The first variable is assigned to <code>a</code>, the last to <code>c</code>, and all other values are packed into a new list <code>b</code>. If you run the <a href="https://realpython.com/run-python-scripts/">script</a>, <code>print()</code> will show you that your three variables have the values you would expect:</p>
0272 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python extract_list_body.py
0273 <span class="go">1</span>
0274 <span class="go">[2, 3, 4, 5]</span>
0275 <span class="go">6</span>
0276 </pre></div>
0277
0278 <p>Another interesting thing you can do with the unpacking operator <code>*</code> is to split the items of any iterable object. This could be very useful if you need to merge two lists, for instance:</p>
0279 <div class="highlight python"><pre><span></span><span class="c1"># merging_lists.py</span>
0280 <span class="n">my_first_list</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span>
0281 <span class="n">my_second_list</span> <span class="o">=</span> <span class="p">[</span><span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">6</span><span class="p">]</span>
0282 <span class="n">my_merged_list</span> <span class="o">=</span> <span class="p">[</span><span class="o">*</span><span class="n">my_first_list</span><span class="p">,</span> <span class="o">*</span><span class="n">my_second_list</span><span class="p">]</span>
0283
0284 <span class="nb">print</span><span class="p">(</span><span class="n">my_merged_list</span><span class="p">)</span>
0285 </pre></div>
0286
0287 <p>The unpacking operator <code>*</code> is prepended to both <code>my_first_list</code> and <code>my_second_list</code>.</p>
0288 <p>If you run this script, you’ll see that the result is a merged list:</p>
0289 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python merging_lists.py
0290 <span class="go">[1, 2, 3, 4, 5, 6]</span>
0291 </pre></div>
0292
0293 <p>You can even merge two different dictionaries by using the unpacking operator <code>**</code>:</p>
0294 <div class="highlight python"><pre><span></span><span class="c1"># merging_dicts.py</span>
0295 <span class="n">my_first_dict</span> <span class="o">=</span> <span class="p">{</span><span class="s2">"A"</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="s2">"B"</span><span class="p">:</span> <span class="mi">2</span><span class="p">}</span>
0296 <span class="n">my_second_dict</span> <span class="o">=</span> <span class="p">{</span><span class="s2">"C"</span><span class="p">:</span> <span class="mi">3</span><span class="p">,</span> <span class="s2">"D"</span><span class="p">:</span> <span class="mi">4</span><span class="p">}</span>
0297 <span class="n">my_merged_dict</span> <span class="o">=</span> <span class="p">{</span><span class="o">**</span><span class="n">my_first_dict</span><span class="p">,</span> <span class="o">**</span><span class="n">my_second_dict</span><span class="p">}</span>
0298
0299 <span class="nb">print</span><span class="p">(</span><span class="n">my_merged_dict</span><span class="p">)</span>
0300 </pre></div>
0301
0302 <p>Here, the iterables to merge are <code>my_first_dict</code> and <code>my_second_dict</code>.</p>
0303 <p>Executing this code outputs a merged dictionary:</p>
0304 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python merging_dicts.py
0305 <span class="go">{'A': 1, 'B': 2, 'C': 3, 'D': 4}</span>
0306 </pre></div>
0307
0308 <p>Remember that the <code>*</code> operator works on <em>any</em> iterable object. It can also be used to unpack a <a href="https://realpython.com/python-strings/">string</a>:</p>
0309 <div class="highlight python"><pre><span></span><span class="c1"># string_to_list.py</span>
0310 <span class="n">a</span> <span class="o">=</span> <span class="p">[</span><span class="o">*</span><span class="s2">"RealPython"</span><span class="p">]</span>
0311 <span class="nb">print</span><span class="p">(</span><span class="n">a</span><span class="p">)</span>
0312 </pre></div>
0313
0314 <p>In Python, strings are iterable objects, so <code>*</code> will unpack it and place all individual values in a list <code>a</code>:</p>
0315 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python string_to_list.py
0316 <span class="go">['R', 'e', 'a', 'l', 'P', 'y', 't', 'h', 'o', 'n']</span>
0317 </pre></div>
0318
0319 <p>The previous example seems great, but when you work with these operators it’s important to keep in mind the seventh rule of <a href="https://www.python.org/dev/peps/pep-0020/"><em>The Zen of Python</em></a> by Tim Peters: <em>Readability counts</em>.</p>
0320 <p>To see why, consider the following example:</p>
0321 <div class="highlight python"><pre><span></span><span class="c1"># mysterious_statement.py</span>
0322 <span class="o">*</span><span class="n">a</span><span class="p">,</span> <span class="o">=</span> <span class="s2">"RealPython"</span>
0323 <span class="nb">print</span><span class="p">(</span><span class="n">a</span><span class="p">)</span>
0324 </pre></div>
0325
0326 <p>There’s the unpacking operator <code>*</code>, followed by a variable, a comma, and an assignment. That’s a lot packed into one line! In fact, this code is no different from the previous example. It just takes the string <code>RealPython</code> and assigns all the items to the new list <code>a</code>, thanks to the unpacking operator <code>*</code>.</p>
0327 <p>The comma after the <code>a</code> does the trick. When you use the unpacking operator with variable assignment, Python requires that your resulting variable is either a list or a tuple. With the trailing comma, you have actually defined a tuple with just one named variable <code>a</code>.</p>
0328 <p>While this is a neat trick, many Pythonistas would not consider this code to be very readable. As such, it’s best to use these kinds of constructions sparingly.</p>
0329 <h2 id="conclusion">Conclusion</h2>
0330 <p>You are now able to use <strong><code>*args</code></strong> and <strong><code>**kwargs</code></strong> to accept a changeable number of arguments in your functions. You have also learned something more about the unpacking operators. </p>
0331 <p>You’ve learned:</p>
0332 <ul>
0333 <li>What <code>*args</code> and <code>**kwargs</code> actually mean</li>
0334 <li>How to use <code>*args</code> and <code>**kwargs</code> in function definitions</li>
0335 <li>How to use a single asterisk (<code>*</code>) to unpack iterables</li>
0336 <li>How to use two asterisks (<code>**</code>) to unpack dictionaries</li>
0337 </ul>
0338 <p>If you still have questions, don’t hesitate to reach out in the comments section below! To learn more about the use of the asterisks in Python, have a look at <a href="https://treyhunner.com/2018/10/asterisks-in-python-what-they-are-and-how-to-use-them/">Trey Hunner’s article on the subject</a>.</p>
0339 <hr />
0340 <p><em>[ Improve Your Python With π Python Tricks π β Get a short & sweet Python Trick delivered to your inbox every couple of days. <a href="https://realpython.com/python-tricks/?utm_source=realpython&utm_medium=rss&utm_campaign=footer">>> Click here to learn more and see examples</a> ]</em></p>#
0341 datePublished: #Wed Sep 4 14:00:00 2019#
0342 dateUpdated: #Wed Sep 4 14:00:00 2019#
0343 # Person begin ####################
0344 name: #Real Python#
0345 # Person end ######################
0346 # Item end ########################
0347 # Item begin ######################
0348 id: #https://realpython.com/courses/lists-tuples-python/#
0349 title: #Lists and Tuples in Python#
0350 link: #https://realpython.com/courses/lists-tuples-python/#
0351 description: #In this course, you'll cover the important characteristics of lists and tuples in Python 3. You'll learn how to define them and how to manipulate them. When you're finished, you'll have a good feel for when and how to use these object types in a Python program.#
0352 content: #<p>In this course, you’ll learn about working with lists and tuples. <strong>Lists</strong> and <strong>tuples</strong> are arguably Python’s most versatile, useful <a href="https://realpython.com/python-data-types/">data types</a>. You’ll find them in virtually every non-trivial Python program.</p>
0353 <p><strong>Here’s what you’ll learn in this tutorial:</strong> You’ll cover the important characteristics of lists and tuples. You’ll learn how to define them and how to manipulate them. When you’re finished, you’ll have a good feel for when and how to use these object types in a Python program.</p>
0354 <div class="alert alert-primary" role="alert">
0355 <p><strong><i class="fa fa-graduation-cap" aria-hidden="true"></i> Take the Quiz:</strong> Test your knowledge with our interactive βPython Lists and Tuplesβ quiz. Upon completion you will receive a score so you can track your learning progress over time:</p><p class="text-center my-2"><a class="btn btn-primary" href="/quizzes/python-lists-tuples/" target="_blank">Take the Quiz Β»</a></p>
0356 </div>
0357 <hr />
0358 <p><em>[ Improve Your Python With π Python Tricks π β Get a short & sweet Python Trick delivered to your inbox every couple of days. <a href="https://realpython.com/python-tricks/?utm_source=realpython&utm_medium=rss&utm_campaign=footer">>> Click here to learn more and see examples</a> ]</em></p>#
0359 datePublished: #Tue Sep 3 14:00:00 2019#
0360 dateUpdated: #Tue Sep 3 14:00:00 2019#
0361 # Person begin ####################
0362 name: #Real Python#
0363 # Person end ######################
0364 # Item end ########################
0365 # Item begin ######################
0366 id: #https://realpython.com/natural-language-processing-spacy-python/#
0367 title: #Natural Language Processing With spaCy in Python#
0368 link: #https://realpython.com/natural-language-processing-spacy-python/#
0369 description: #In this step-by-step tutorial, you'll learn how to use spaCy. This free and open-source library for Natural Language Processing (NLP) in Python has a lot of built-in capabilities and is becoming increasingly popular for processing and analyzing data in NLP.#
0370 content: #<p><strong>spaCy</strong> is a free and open-source library for <strong>Natural Language Processing</strong> (NLP) in Python with a lot of in-built capabilities. It’s becoming increasingly popular for processing and analyzing data in NLP. Unstructured textual data is produced at a large scale, and it’s important to process and derive insights from unstructured data. To do that, you need to represent the data in a format that can be understood by computers. NLP can help you do that.</p>
0371 <p><strong>In this tutorial, you’ll learn:</strong></p>
0372 <ul>
0373 <li>What the foundational terms and concepts in NLP are</li>
0374 <li>How to implement those concepts in spaCy</li>
0375 <li>How to customize and extend built-in functionalities in spaCy</li>
0376 <li>How to perform basic statistical analysis on a text</li>
0377 <li>How to create a pipeline to process unstructured text</li>
0378 <li>How to parse a sentence and extract meaningful insights from it</li>
0379 </ul>
0380 <div class="alert alert-warning" role="alert"><p><strong>Free Bonus:</strong> <a href="" class="alert-link" data-toggle="modal" data-target="#modal-python-tricks-sample" data-focus="false">Click here to get access to a chapter from Python Tricks: The Book</a> that shows you Python's best practices with simple examples you can apply instantly to write more beautiful + Pythonic code.</p></div>
0381
0382 <h2 id="what-are-nlp-and-spacy">What Are NLP and spaCy?</h2>
0383 <p><strong>NLP</strong> is a subfield of <strong>Artificial Intelligence</strong> and is concerned with interactions between computers and human languages. NLP is the process of analyzing, understanding, and deriving meaning from human languages for computers.</p>
0384 <p>NLP helps you extract insights from unstructured text and has several use cases, such as:</p>
0385 <ul>
0386 <li><a href="https://en.wikipedia.org/wiki/Automatic_summarization">Automatic summarization</a></li>
0387 <li><a href="https://en.wikipedia.org/wiki/Named-entity_recognition">Named entity recognition</a></li>
0388 <li><a href="https://en.wikipedia.org/wiki/Question_answering">Question answering systems</a></li>
0389 <li><a href="https://en.wikipedia.org/wiki/Sentiment_analysis">Sentiment analysis</a></li>
0390 </ul>
0391 <p>spaCy is a free, open-source library for NLP in Python. It’s written in <a href="https://cython.org/">Cython</a> and is designed to build information extraction or natural language understanding systems. It’s built for production use and provides a concise and user-friendly API.</p>
0392 <h2 id="installation">Installation</h2>
0393 <p>In this section, you’ll install spaCy and then download data and models for the English language.</p>
0394 <h3 id="how-to-install-spacy">How to Install spaCy</h3>
0395 <p>spaCy can be installed using <strong><code>pip</code></strong>, a Python package manager. You can use a <strong>virtual environment</strong> to avoid depending on system-wide packages. To learn more about virtual environments and <code>pip</code>, check out <a href="https://realpython.com/what-is-pip/">What Is Pip? A Guide for New Pythonistas</a> and <a href="https://realpython.com/python-virtual-environments-a-primer/">Python Virtual Environments: A Primer</a>.</p>
0396 <p>Create a new virtual environment:</p>
0397 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python3 -m venv env
0398 </pre></div>
0399
0400 <p>Activate this virtual environment and install spaCy:</p>
0401 <div class="highlight sh"><pre><span></span><span class="gp">$</span> <span class="nb">source</span> ./env/bin/activate
0402 <span class="gp">$</span> pip install spacy
0403 </pre></div>
0404
0405 <h3 id="how-to-download-models-and-data">How to Download Models and Data</h3>
0406 <p>spaCy has <a href="https://spaCy.io/models">different types</a> of models. The default model for the English language is <code>en_core_web_sm</code>.</p>
0407 <p>Activate the virtual environment created in the previous step and download models and data for the English language:</p>
0408 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python -m spacy download en_core_web_sm
0409 </pre></div>
0410
0411 <p>Verify if the download was successful or not by loading it:</p>
0412 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">spacy</span>
0413 <span class="gp">>>> </span><span class="n">nlp</span> <span class="o">=</span> <span class="n">spacy</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="s1">'en_core_web_sm'</span><span class="p">)</span>
0414 </pre></div>
0415
0416 <p>If the <code>nlp</code> object is created, then it means that spaCy was installed and that models and data were successfully downloaded.</p>
0417 <h2 id="using-spacy">Using spaCy</h2>
0418 <p>In this section, you’ll use spaCy for a given input string and a text file. Load the language model instance in spaCy:</p>
0419 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">spacy</span>
0420 <span class="gp">>>> </span><span class="n">nlp</span> <span class="o">=</span> <span class="n">spacy</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="s1">'en_core_web_sm'</span><span class="p">)</span>
0421 </pre></div>
0422
0423 <p>Here, the <code>nlp</code> object is a language model instance. You can assume that, throughout this tutorial, <code>nlp</code> refers to the language model loaded by <code>en_core_web_sm</code>. Now you can use spaCy to read a string or a text file.</p>
0424 <h3 id="how-to-read-a-string">How to Read a String</h3>
0425 <p>You can use spaCy to create a processed <a href="https://spaCy.io/api/doc">Doc</a> object, which is a container for accessing linguistic annotations, for a given input string:</p>
0426 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">introduction_text</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'This tutorial is about Natural'</span>
0427 <span class="gp">... </span> <span class="s1">' Language Processing in Spacy.'</span><span class="p">)</span>
0428 <span class="gp">>>> </span><span class="n">introduction_doc</span> <span class="o">=</span> <span class="n">nlp</span><span class="p">(</span><span class="n">introduction_text</span><span class="p">)</span>
0429 <span class="gp">>>> </span><span class="c1"># Extract tokens for the given doc</span>
0430 <span class="gp">>>> </span><span class="nb">print</span> <span class="p">([</span><span class="n">token</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">introduction_doc</span><span class="p">])</span>
0431 <span class="go">['This', 'tutorial', 'is', 'about', 'Natural', 'Language',</span>
0432 <span class="go">'Processing', 'in', 'Spacy', '.']</span>
0433 </pre></div>
0434
0435 <p>In the above example, notice how the text is converted to an object that is understood by spaCy. You can use this method to convert any text into a processed <code>Doc</code> object and deduce attributes, which will be covered in the coming sections.</p>
0436 <h3 id="how-to-read-a-text-file">How to Read a Text File</h3>
0437 <p>In this section, you’ll create a processed <a href="https://spaCy.io/api/doc">Doc</a> object for a text file:</p>
0438 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">file_name</span> <span class="o">=</span> <span class="s1">'introduction.txt'</span>
0439 <span class="gp">>>> </span><span class="n">introduction_file_text</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">file_name</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
0440 <span class="gp">>>> </span><span class="n">introduction_file_doc</span> <span class="o">=</span> <span class="n">nlp</span><span class="p">(</span><span class="n">introduction_file_text</span><span class="p">)</span>
0441 <span class="gp">>>> </span><span class="c1"># Extract tokens for the given doc</span>
0442 <span class="gp">>>> </span><span class="nb">print</span> <span class="p">([</span><span class="n">token</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">introduction_file_doc</span><span class="p">])</span>
0443 <span class="go">['This', 'tutorial', 'is', 'about', 'Natural', 'Language',</span>
0444 <span class="go">'Processing', 'in', 'Spacy', '.', '\n']</span>
0445 </pre></div>
0446
0447 <p>This is how you can convert a text file into a processed <code>Doc</code> object.</p>
0448 <div class="alert alert-primary" role="alert">
0449 <p><strong>Note:</strong> </p>
0450 <p>You can assume that:</p>
0451 <ul>
0452 <li>Variable names ending with the suffix <strong><code>_text</code></strong> are <strong><a href="https://realpython.com/python-encodings-guide/">Unicode</a> string objects</strong>.</li>
0453 <li>Variable name ending with the suffix <strong><code>_doc</code></strong> are <strong>spaCy’s language model objects</strong>.</li>
0454 </ul>
0455 </div>
0456 <h2 id="sentence-detection">Sentence Detection</h2>
0457 <p><strong>Sentence Detection</strong> is the process of locating the start and end of sentences in a given text. This allows you to you divide a text into linguistically meaningful units. You’ll use these units when you’re processing your text to perform tasks such as <strong>part of speech tagging</strong> and <strong>entity extraction</strong>.</p>
0458 <p>In spaCy, the <code>sents</code> property is used to extract sentences. Here’s how you would extract the total number of sentences and the sentences for a given input text:</p>
0459 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">about_text</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'Gus Proto is a Python developer currently'</span>
0460 <span class="gp">... </span> <span class="s1">' working for a London-based Fintech'</span>
0461 <span class="gp">... </span> <span class="s1">' company. He is interested in learning'</span>
0462 <span class="gp">... </span> <span class="s1">' Natural Language Processing.'</span><span class="p">)</span>
0463 <span class="gp">>>> </span><span class="n">about_doc</span> <span class="o">=</span> <span class="n">nlp</span><span class="p">(</span><span class="n">about_text</span><span class="p">)</span>
0464 <span class="gp">>>> </span><span class="n">sentences</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">about_doc</span><span class="o">.</span><span class="n">sents</span><span class="p">)</span>
0465 <span class="gp">>>> </span><span class="nb">len</span><span class="p">(</span><span class="n">sentences</span><span class="p">)</span>
0466 <span class="go">2</span>
0467 <span class="gp">>>> </span><span class="k">for</span> <span class="n">sentence</span> <span class="ow">in</span> <span class="n">sentences</span><span class="p">:</span>
0468 <span class="gp">... </span> <span class="nb">print</span> <span class="p">(</span><span class="n">sentence</span><span class="p">)</span>
0469 <span class="gp">...</span>
0470 <span class="go">'Gus Proto is a Python developer currently working for a</span>
0471 <span class="go">London-based Fintech company.'</span>
0472 <span class="go">'He is interested in learning Natural Language Processing.'</span>
0473 </pre></div>
0474
0475 <p>In the above example, spaCy is correctly able to identify sentences in the English language, using a full stop(<code>.</code>) as the sentence delimiter. You can also customize the sentence detection to detect sentences on custom delimiters.</p>
0476 <p>Here’s an example, where an ellipsis(<code>...</code>) is used as the delimiter:</p>
0477 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">def</span> <span class="nf">set_custom_boundaries</span><span class="p">(</span><span class="n">doc</span><span class="p">):</span>
0478 <span class="gp">... </span> <span class="c1"># Adds support to use `...` as the delimiter for sentence detection</span>
0479 <span class="gp">... </span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">doc</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">]:</span>
0480 <span class="gp">... </span> <span class="k">if</span> <span class="n">token</span><span class="o">.</span><span class="n">text</span> <span class="o">==</span> <span class="s1">'...'</span><span class="p">:</span>
0481 <span class="gp">... </span> <span class="n">doc</span><span class="p">[</span><span class="n">token</span><span class="o">.</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">is_sent_start</span> <span class="o">=</span> <span class="kc">True</span>
0482 <span class="gp">... </span> <span class="k">return</span> <span class="n">doc</span>
0483 <span class="gp">...</span>
0484 <span class="gp">>>> </span><span class="n">ellipsis_text</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'Gus, can you, ... never mind, I forgot'</span>
0485 <span class="gp">... </span> <span class="s1">' what I was saying. So, do you think'</span>
0486 <span class="gp">... </span> <span class="s1">' we should ...'</span><span class="p">)</span>
0487 <span class="gp">>>> </span><span class="c1"># Load a new model instance</span>
0488 <span class="gp">>>> </span><span class="n">custom_nlp</span> <span class="o">=</span> <span class="n">spacy</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="s1">'en_core_web_sm'</span><span class="p">)</span>
0489 <span class="gp">>>> </span><span class="n">custom_nlp</span><span class="o">.</span><span class="n">add_pipe</span><span class="p">(</span><span class="n">set_custom_boundaries</span><span class="p">,</span> <span class="n">before</span><span class="o">=</span><span class="s1">'parser'</span><span class="p">)</span>
0490 <span class="gp">>>> </span><span class="n">custom_ellipsis_doc</span> <span class="o">=</span> <span class="n">custom_nlp</span><span class="p">(</span><span class="n">ellipsis_text</span><span class="p">)</span>
0491 <span class="gp">>>> </span><span class="n">custom_ellipsis_sentences</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">custom_ellipsis_doc</span><span class="o">.</span><span class="n">sents</span><span class="p">)</span>
0492 <span class="gp">>>> </span><span class="k">for</span> <span class="n">sentence</span> <span class="ow">in</span> <span class="n">custom_ellipsis_sentences</span><span class="p">:</span>
0493 <span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">sentence</span><span class="p">)</span>
0494 <span class="gp">...</span>
0495 <span class="go">Gus, can you, ...</span>
0496 <span class="go">never mind, I forgot what I was saying.</span>
0497 <span class="go">So, do you think we should ...</span>
0498 <span class="gp">>>> </span><span class="c1"># Sentence Detection with no customization</span>
0499 <span class="gp">>>> </span><span class="n">ellipsis_doc</span> <span class="o">=</span> <span class="n">nlp</span><span class="p">(</span><span class="n">ellipsis_text</span><span class="p">)</span>
0500 <span class="gp">>>> </span><span class="n">ellipsis_sentences</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">ellipsis_doc</span><span class="o">.</span><span class="n">sents</span><span class="p">)</span>
0501 <span class="gp">>>> </span><span class="k">for</span> <span class="n">sentence</span> <span class="ow">in</span> <span class="n">ellipsis_sentences</span><span class="p">:</span>
0502 <span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">sentence</span><span class="p">)</span>
0503 <span class="gp">...</span>
0504 <span class="go">Gus, can you, ... never mind, I forgot what I was saying.</span>
0505 <span class="go">So, do you think we should ...</span>
0506 </pre></div>
0507
0508 <p>Note that <code>custom_ellipsis_sentences</code> contain three sentences, whereas <code>ellipsis_sentences</code> contains two sentences. These sentences are still obtained via the <code>sents</code> attribute, as you saw before.</p>
0509 <h2 id="tokenization-in-spacy">Tokenization in spaCy</h2>
0510 <p><strong>Tokenization</strong> is the next step after sentence detection. It allows you to identify the basic units in your text. These basic units are called <strong>tokens</strong>. Tokenization is useful because it breaks a text into meaningful units. These units are used for further analysis, like part of speech tagging.</p>
0511 <p>In spaCy, you can print tokens by iterating on the <code>Doc</code> object:</p>
0512 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">about_doc</span><span class="p">:</span>
0513 <span class="gp">... </span> <span class="nb">print</span> <span class="p">(</span><span class="n">token</span><span class="p">,</span> <span class="n">token</span><span class="o">.</span><span class="n">idx</span><span class="p">)</span>
0514 <span class="gp">...</span>
0515 <span class="go">Gus 0</span>
0516 <span class="go">Proto 4</span>
0517 <span class="go">is 10</span>
0518 <span class="go">a 13</span>
0519 <span class="go">Python 15</span>
0520 <span class="go">developer 22</span>
0521 <span class="go">currently 32</span>
0522 <span class="go">working 42</span>
0523 <span class="go">for 50</span>
0524 <span class="go">a 54</span>
0525 <span class="go">London 56</span>
0526 <span class="go">- 62</span>
0527 <span class="go">based 63</span>
0528 <span class="go">Fintech 69</span>
0529 <span class="go">company 77</span>
0530 <span class="go">. 84</span>
0531 <span class="go">He 86</span>
0532 <span class="go">is 89</span>
0533 <span class="go">interested 92</span>
0534 <span class="go">in 103</span>
0535 <span class="go">learning 106</span>
0536 <span class="go">Natural 115</span>
0537 <span class="go">Language 123</span>
0538 <span class="go">Processing 132</span>
0539 <span class="go">. 142</span>
0540 </pre></div>
0541
0542 <p>Note how spaCy preserves the <strong>starting index</strong> of the tokens. It’s useful for in-place word replacement. spaCy provides <a href="https://spacy.io/api/token#attributes">various attributes</a> for the <code>Token</code> class:</p>
0543 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">about_doc</span><span class="p">:</span>
0544 <span class="gp">... </span> <span class="nb">print</span> <span class="p">(</span><span class="n">token</span><span class="p">,</span> <span class="n">token</span><span class="o">.</span><span class="n">idx</span><span class="p">,</span> <span class="n">token</span><span class="o">.</span><span class="n">text_with_ws</span><span class="p">,</span>
0545 <span class="gp">... </span> <span class="n">token</span><span class="o">.</span><span class="n">is_alpha</span><span class="p">,</span> <span class="n">token</span><span class="o">.</span><span class="n">is_punct</span><span class="p">,</span> <span class="n">token</span><span class="o">.</span><span class="n">is_space</span><span class="p">,</span>
0546 <span class="gp">... </span> <span class="n">token</span><span class="o">.</span><span class="n">shape_</span><span class="p">,</span> <span class="n">token</span><span class="o">.</span><span class="n">is_stop</span><span class="p">)</span>
0547 <span class="gp">...</span>
0548 <span class="go">Gus 0 Gus True False False Xxx False</span>
0549 <span class="go">Proto 4 Proto True False False Xxxxx False</span>
0550 <span class="go">is 10 is True False False xx True</span>
0551 <span class="go">a 13 a True False False x True</span>
0552 <span class="go">Python 15 Python True False False Xxxxx False</span>
0553 <span class="go">developer 22 developer True False False xxxx False</span>
0554 <span class="go">currently 32 currently True False False xxxx False</span>
0555 <span class="go">working 42 working True False False xxxx False</span>
0556 <span class="go">for 50 for True False False xxx True</span>
0557 <span class="go">a 54 a True False False x True</span>
0558 <span class="go">London 56 London True False False Xxxxx False</span>
0559 <span class="go">- 62 - False True False - False</span>
0560 <span class="go">based 63 based True False False xxxx False</span>
0561 <span class="go">Fintech 69 Fintech True False False Xxxxx False</span>
0562 <span class="go">company 77 company True False False xxxx False</span>
0563 <span class="go">. 84 . False True False . False</span>
0564 <span class="go">He 86 He True False False Xx True</span>
0565 <span class="go">is 89 is True False False xx True</span>
0566 <span class="go">interested 92 interested True False False xxxx False</span>
0567 <span class="go">in 103 in True False False xx True</span>
0568 <span class="go">learning 106 learning True False False xxxx False</span>
0569 <span class="go">Natural 115 Natural True False False Xxxxx False</span>
0570 <span class="go">Language 123 Language True False False Xxxxx False</span>
0571 <span class="go">Processing 132 Processing True False False Xxxxx False</span>
0572 <span class="go">. 142 . False True False . False</span>
0573 </pre></div>
0574
0575 <p>In this example, some of the commonly required attributes are accessed:</p>
0576 <ul>
0577 <li><strong><code>text_with_ws</code></strong> prints token text with trailing space (if present).</li>
0578 <li><strong><code>is_alpha</code></strong> detects if the token consists of alphabetic characters or not.</li>
0579 <li><strong><code>is_punct</code></strong> detects if the token is a punctuation symbol or not.</li>
0580 <li><strong><code>is_space</code></strong> detects if the token is a space or not.</li>
0581 <li><strong><code>shape_</code></strong> prints out the shape of the word.</li>
0582 <li><strong><code>is_stop</code></strong> detects if the token is a stop word or not.</li>
0583 </ul>
0584 <div class="alert alert-primary" role="alert">
0585 <p><strong>Note:</strong> You’ll learn more about <strong>stop words</strong> in the next section.</p>
0586 </div>
0587 <p>You can also customize the tokenization process to detect tokens on custom characters. This is often used for hyphenated words, which are words joined with hyphen. For example, “London-based” is a hyphenated word.</p>
0588 <p>spaCy allows you to customize tokenization by updating the <code>tokenizer</code> property on the <code>nlp</code> object:</p>
0589 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">re</span>
0590 <span class="gp">>>> </span><span class="kn">import</span> <span class="nn">spacy</span>
0591 <span class="gp">>>> </span><span class="kn">from</span> <span class="nn">spacy.tokenizer</span> <span class="k">import</span> <span class="n">Tokenizer</span>
0592 <span class="gp">>>> </span><span class="n">custom_nlp</span> <span class="o">=</span> <span class="n">spacy</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="s1">'en_core_web_sm'</span><span class="p">)</span>
0593 <span class="gp">>>> </span><span class="n">prefix_re</span> <span class="o">=</span> <span class="n">spacy</span><span class="o">.</span><span class="n">util</span><span class="o">.</span><span class="n">compile_prefix_regex</span><span class="p">(</span><span class="n">custom_nlp</span><span class="o">.</span><span class="n">Defaults</span><span class="o">.</span><span class="n">prefixes</span><span class="p">)</span>
0594 <span class="gp">>>> </span><span class="n">suffix_re</span> <span class="o">=</span> <span class="n">spacy</span><span class="o">.</span><span class="n">util</span><span class="o">.</span><span class="n">compile_suffix_regex</span><span class="p">(</span><span class="n">custom_nlp</span><span class="o">.</span><span class="n">Defaults</span><span class="o">.</span><span class="n">suffixes</span><span class="p">)</span>
0595 <span class="gp">>>> </span><span class="n">infix_re</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="sa">r</span><span class="s1">'''[-~]'''</span><span class="p">)</span>
0596 <span class="gp">>>> </span><span class="k">def</span> <span class="nf">customize_tokenizer</span><span class="p">(</span><span class="n">nlp</span><span class="p">):</span>
0597 <span class="gp">... </span> <span class="c1"># Adds support to use `-` as the delimiter for tokenization</span>
0598 <span class="gp">... </span> <span class="k">return</span> <span class="n">Tokenizer</span><span class="p">(</span><span class="n">nlp</span><span class="o">.</span><span class="n">vocab</span><span class="p">,</span> <span class="n">prefix_search</span><span class="o">=</span><span class="n">prefix_re</span><span class="o">.</span><span class="n">search</span><span class="p">,</span>
0599 <span class="gp">... </span> <span class="n">suffix_search</span><span class="o">=</span><span class="n">suffix_re</span><span class="o">.</span><span class="n">search</span><span class="p">,</span>
0600 <span class="gp">... </span> <span class="n">infix_finditer</span><span class="o">=</span><span class="n">infix_re</span><span class="o">.</span><span class="n">finditer</span><span class="p">,</span>
0601 <span class="gp">... </span> <span class="n">token_match</span><span class="o">=</span><span class="kc">None</span>
0602 <span class="gp">... </span> <span class="p">)</span>
0603 <span class="gp">...</span>
0604
0605 <span class="gp">>>> </span><span class="n">custom_nlp</span><span class="o">.</span><span class="n">tokenizer</span> <span class="o">=</span> <span class="n">customize_tokenizer</span><span class="p">(</span><span class="n">custom_nlp</span><span class="p">)</span>
0606 <span class="gp">>>> </span><span class="n">custom_tokenizer_about_doc</span> <span class="o">=</span> <span class="n">custom_nlp</span><span class="p">(</span><span class="n">about_text</span><span class="p">)</span>
0607 <span class="gp">>>> </span><span class="nb">print</span><span class="p">([</span><span class="n">token</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">custom_tokenizer_about_doc</span><span class="p">])</span>
0608 <span class="go">['Gus', 'Proto', 'is', 'a', 'Python', 'developer', 'currently',</span>
0609 <span class="go">'working', 'for', 'a', 'London', '-', 'based', 'Fintech',</span>
0610 <span class="go">'company', '.', 'He', 'is', 'interested', 'in', 'learning',</span>
0611 <span class="go">'Natural', 'Language', 'Processing', '.']</span>
0612 </pre></div>
0613
0614 <p>In order for you to customize, you can pass various parameters to the <code>Tokenizer</code> class:</p>
0615 <ul>
0616 <li><strong><code>nlp.vocab</code></strong> is a storage container for special cases and is used to handle cases like contractions and emoticons.</li>
0617 <li><strong><code>prefix_search</code></strong> is the function that is used to handle preceding punctuation, such as opening parentheses.</li>
0618 <li><strong><code>infix_finditer</code></strong> is the function that is used to handle non-whitespace separators, such as hyphens.</li>
0619 <li><strong><code>suffix_search</code></strong> is the function that is used to handle succeeding punctuation, such as closing parentheses.</li>
0620 <li><strong><code>token_match</code></strong> is an optional boolean function that is used to match strings that should never be split. It overrides the previous rules and is useful for entities like URLs or numbers.</li>
0621 </ul>
0622 <div class="alert alert-primary" role="alert">
0623 <p><strong>Note:</strong> spaCy already detects hyphenated words as individual tokens. The above code is just an example to show how tokenization can be customized. It can be used for any other character.</p>
0624 </div>
0625 <h2 id="stop-words">Stop Words</h2>
0626 <p><strong>Stop words</strong> are the most common words in a language. In the English language, some examples of stop words are <code>the</code>, <code>are</code>, <code>but</code>, and <code>they</code>. Most sentences need to contain stop words in order to be full sentences that make sense.</p>
0627 <p>Generally, stop words are removed because they aren’t significant and distort the word frequency analysis. spaCy has a list of stop words for the English language:</p>
0628 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">spacy</span>
0629 <span class="gp">>>> </span><span class="n">spacy_stopwords</span> <span class="o">=</span> <span class="n">spacy</span><span class="o">.</span><span class="n">lang</span><span class="o">.</span><span class="n">en</span><span class="o">.</span><span class="n">stop_words</span><span class="o">.</span><span class="n">STOP_WORDS</span>
0630 <span class="gp">>>> </span><span class="nb">len</span><span class="p">(</span><span class="n">spacy_stopwords</span><span class="p">)</span>
0631 <span class="go">326</span>
0632 <span class="gp">>>> </span><span class="k">for</span> <span class="n">stop_word</span> <span class="ow">in</span> <span class="nb">list</span><span class="p">(</span><span class="n">spacy_stopwords</span><span class="p">)[:</span><span class="mi">10</span><span class="p">]:</span>
0633 <span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">stop_word</span><span class="p">)</span>
0634 <span class="gp">...</span>
0635 <span class="go">using</span>
0636 <span class="go">becomes</span>
0637 <span class="go">had</span>
0638 <span class="go">itself</span>
0639 <span class="go">once</span>
0640 <span class="go">often</span>
0641 <span class="go">is</span>
0642 <span class="go">herein</span>
0643 <span class="go">who</span>
0644 <span class="go">too</span>
0645 </pre></div>
0646
0647 <p>You can remove stop words from the input text:</p>
0648 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">about_doc</span><span class="p">:</span>
0649 <span class="gp">... </span> <span class="k">if</span> <span class="ow">not</span> <span class="n">token</span><span class="o">.</span><span class="n">is_stop</span><span class="p">:</span>
0650 <span class="gp">... </span> <span class="nb">print</span> <span class="p">(</span><span class="n">token</span><span class="p">)</span>
0651 <span class="gp">...</span>
0652 <span class="go">Gus</span>
0653 <span class="go">Proto</span>
0654 <span class="go">Python</span>
0655 <span class="go">developer</span>
0656 <span class="go">currently</span>
0657 <span class="go">working</span>
0658 <span class="go">London</span>
0659 <span class="go">-</span>
0660 <span class="go">based</span>
0661 <span class="go">Fintech</span>
0662 <span class="go">company</span>
0663 <span class="go">.</span>
0664 <span class="go">interested</span>
0665 <span class="go">learning</span>
0666 <span class="go">Natural</span>
0667 <span class="go">Language</span>
0668 <span class="go">Processing</span>
0669 <span class="go">.</span>
0670 </pre></div>
0671
0672 <p>Stop words like <code>is</code>, <code>a</code>, <code>for</code>, <code>the</code>, and <code>in</code> are not printed in the output above. You can also create a list of tokens not containing stop words:</p>
0673 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">about_no_stopword_doc</span> <span class="o">=</span> <span class="p">[</span><span class="n">token</span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">about_doc</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">token</span><span class="o">.</span><span class="n">is_stop</span><span class="p">]</span>
0674 <span class="gp">>>> </span><span class="nb">print</span> <span class="p">(</span><span class="n">about_no_stopword_doc</span><span class="p">)</span>
0675 <span class="go">[Gus, Proto, Python, developer, currently, working, London,</span>
0676 <span class="go">-, based, Fintech, company, ., interested, learning, Natural,</span>
0677 <span class="go">Language, Processing, .]</span>
0678 </pre></div>
0679
0680 <p><code>about_no_stopword_doc</code> can be joined with spaces to form a sentence with no stop words.</p>
0681 <h2 id="lemmatization">Lemmatization</h2>
0682 <p><strong>Lemmatization</strong> is the process of reducing inflected forms of a word while still ensuring that the reduced form belongs to the language. This reduced form or root word is called a <strong>lemma</strong>.</p>
0683 <p>For example, <em>organizes</em>, <em>organized</em> and <em>organizing</em> are all forms of <em>organize</em>. Here, <em>organize</em> is the lemma. The inflection of a word allows you to express different grammatical categories like tense (<em>organized</em> vs <em>organize</em>), number (<em>trains</em> vs <em>train</em>), and so on. Lemmatization is necessary because it helps you reduce the inflected forms of a word so that they can be analyzed as a single item. It can also help you <strong>normalize</strong> the text.</p>
0684 <p>spaCy has the attribute <code>lemma_</code> on the <code>Token</code> class. This attribute has the lemmatized form of a token:</p>
0685 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">conference_help_text</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'Gus is helping organize a developer'</span>
0686 <span class="gp">... </span> <span class="s1">'conference on Applications of Natural Language'</span>
0687 <span class="gp">... </span> <span class="s1">' Processing. He keeps organizing local Python meetups'</span>
0688 <span class="gp">... </span> <span class="s1">' and several internal talks at his workplace.'</span><span class="p">)</span>
0689 <span class="gp">>>> </span><span class="n">conference_help_doc</span> <span class="o">=</span> <span class="n">nlp</span><span class="p">(</span><span class="n">conference_help_text</span><span class="p">)</span>
0690 <span class="gp">>>> </span><span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">conference_help_doc</span><span class="p">:</span>
0691 <span class="gp">... </span> <span class="nb">print</span> <span class="p">(</span><span class="n">token</span><span class="p">,</span> <span class="n">token</span><span class="o">.</span><span class="n">lemma_</span><span class="p">)</span>
0692 <span class="gp">...</span>
0693 <span class="go">Gus Gus</span>
0694 <span class="go">is be</span>
0695 <span class="go">helping help</span>
0696 <span class="go">organize organize</span>
0697 <span class="go">a a</span>
0698 <span class="go">developer developer</span>
0699 <span class="go">conference conference</span>
0700 <span class="go">on on</span>
0701 <span class="go">Applications Applications</span>
0702 <span class="go">of of</span>
0703 <span class="go">Natural Natural</span>
0704 <span class="go">Language Language</span>
0705 <span class="go">Processing Processing</span>
0706 <span class="go">. .</span>
0707 <span class="go">He -PRON-</span>
0708 <span class="go">keeps keep</span>
0709 <span class="go">organizing organize</span>
0710 <span class="go">local local</span>
0711 <span class="go">Python Python</span>
0712 <span class="go">meetups meetup</span>
0713 <span class="go">and and</span>
0714 <span class="go">several several</span>
0715 <span class="go">internal internal</span>
0716 <span class="go">talks talk</span>
0717 <span class="go">at at</span>
0718 <span class="go">his -PRON-</span>
0719 <span class="go">workplace workplace</span>
0720 <span class="go">. .</span>
0721 </pre></div>
0722
0723 <p>In this example, <code>organizing</code> reduces to its lemma form <code>organize</code>. If you do not lemmatize the text, then <code>organize</code> and <code>organizing</code> will be counted as different tokens, even though they both have a similar meaning. Lemmatization helps you avoid duplicate words that have similar meanings.</p>
0724 <h2 id="word-frequency">Word Frequency</h2>
0725 <p>You can now convert a given text into tokens and perform statistical analysis over it. This analysis can give you various insights about word patterns, such as common words or unique words in the text:</p>
0726 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">collections</span> <span class="k">import</span> <span class="n">Counter</span>
0727 <span class="gp">>>> </span><span class="n">complete_text</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'Gus Proto is a Python developer currently'</span>
0728 <span class="gp">... </span> <span class="s1">'working for a London-based Fintech company. He is'</span>
0729 <span class="gp">... </span> <span class="s1">' interested in learning Natural Language Processing.'</span>
0730 <span class="gp">... </span> <span class="s1">' There is a developer conference happening on 21 July'</span>
0731 <span class="gp">... </span> <span class="s1">' 2019 in London. It is titled "Applications of Natural'</span>
0732 <span class="gp">... </span> <span class="s1">' Language Processing". There is a helpline number '</span>
0733 <span class="gp">... </span> <span class="s1">' available at +1-1234567891. Gus is helping organize it.'</span>
0734 <span class="gp">... </span> <span class="s1">' He keeps organizing local Python meetups and several'</span>
0735 <span class="gp">... </span> <span class="s1">' internal talks at his workplace. Gus is also presenting'</span>
0736 <span class="gp">... </span> <span class="s1">' a talk. The talk will introduce the reader about "Use'</span>
0737 <span class="gp">... </span> <span class="s1">' cases of Natural Language Processing in Fintech".'</span>
0738 <span class="gp">... </span> <span class="s1">' Apart from his work, he is very passionate about music.'</span>
0739 <span class="gp">... </span> <span class="s1">' Gus is learning to play the Piano. He has enrolled '</span>
0740 <span class="gp">... </span> <span class="s1">' himself in the weekend batch of Great Piano Academy.'</span>
0741 <span class="gp">... </span> <span class="s1">' Great Piano Academy is situated in Mayfair or the City'</span>
0742 <span class="gp">... </span> <span class="s1">' of London and has world-class piano instructors.'</span><span class="p">)</span>
0743 <span class="gp">...</span>
0744 <span class="gp">>>> </span><span class="n">complete_doc</span> <span class="o">=</span> <span class="n">nlp</span><span class="p">(</span><span class="n">complete_text</span><span class="p">)</span>
0745 <span class="gp">>>> </span><span class="c1"># Remove stop words and punctuation symbols</span>
0746 <span class="gp">>>> </span><span class="n">words</span> <span class="o">=</span> <span class="p">[</span><span class="n">token</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">complete_doc</span>
0747 <span class="gp">... </span> <span class="k">if</span> <span class="ow">not</span> <span class="n">token</span><span class="o">.</span><span class="n">is_stop</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">token</span><span class="o">.</span><span class="n">is_punct</span><span class="p">]</span>
0748 <span class="gp">>>> </span><span class="n">word_freq</span> <span class="o">=</span> <span class="n">Counter</span><span class="p">(</span><span class="n">words</span><span class="p">)</span>
0749 <span class="gp">>>> </span><span class="c1"># 5 commonly occurring words with their frequencies</span>
0750 <span class="gp">>>> </span><span class="n">common_words</span> <span class="o">=</span> <span class="n">word_freq</span><span class="o">.</span><span class="n">most_common</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>
0751 <span class="gp">>>> </span><span class="nb">print</span> <span class="p">(</span><span class="n">common_words</span><span class="p">)</span>
0752 <span class="go">[('Gus', 4), ('London', 3), ('Natural', 3), ('Language', 3), ('Processing', 3)]</span>
0753 <span class="gp">>>> </span><span class="c1"># Unique words</span>
0754 <span class="gp">>>> </span><span class="n">unique_words</span> <span class="o">=</span> <span class="p">[</span><span class="n">word</span> <span class="k">for</span> <span class="p">(</span><span class="n">word</span><span class="p">,</span> <span class="n">freq</span><span class="p">)</span> <span class="ow">in</span> <span class="n">word_freq</span><span class="o">.</span><span class="n">items</span><span class="p">()</span> <span class="k">if</span> <span class="n">freq</span> <span class="o">==</span> <span class="mi">1</span><span class="p">]</span>
0755 <span class="gp">>>> </span><span class="nb">print</span> <span class="p">(</span><span class="n">unique_words</span><span class="p">)</span>
0756 <span class="go">['Proto', 'currently', 'working', 'based', 'company',</span>
0757 <span class="go">'interested', 'conference', 'happening', '21', 'July',</span>
0758 <span class="go">'2019', 'titled', 'Applications', 'helpline', 'number',</span>
0759 <span class="go">'available', '+1', '1234567891', 'helping', 'organize',</span>
0760 <span class="go">'keeps', 'organizing', 'local', 'meetups', 'internal',</span>
0761 <span class="go">'talks', 'workplace', 'presenting', 'introduce', 'reader',</span>
0762 <span class="go">'Use', 'cases', 'Apart', 'work', 'passionate', 'music', 'play',</span>
0763 <span class="go">'enrolled', 'weekend', 'batch', 'situated', 'Mayfair', 'City',</span>
0764 <span class="go">'world', 'class', 'piano', 'instructors']</span>
0765 </pre></div>
0766
0767 <p>By looking at the common words, you can see that the text as a whole is probably about <code>Gus</code>, <code>London</code>, or <code>Natural Language Processing</code>. This way, you can take any unstructured text and perform statistical analysis to know what it’s about.</p>
0768 <p>Here’s another example of the same text with stop words:</p>
0769 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">words_all</span> <span class="o">=</span> <span class="p">[</span><span class="n">token</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">complete_doc</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">token</span><span class="o">.</span><span class="n">is_punct</span><span class="p">]</span>
0770 <span class="gp">>>> </span><span class="n">word_freq_all</span> <span class="o">=</span> <span class="n">Counter</span><span class="p">(</span><span class="n">words_all</span><span class="p">)</span>
0771 <span class="gp">>>> </span><span class="c1"># 5 commonly occurring words with their frequencies</span>
0772 <span class="gp">>>> </span><span class="n">common_words_all</span> <span class="o">=</span> <span class="n">word_freq_all</span><span class="o">.</span><span class="n">most_common</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>
0773 <span class="gp">>>> </span><span class="nb">print</span> <span class="p">(</span><span class="n">common_words_all</span><span class="p">)</span>
0774 <span class="go">[('is', 10), ('a', 5), ('in', 5), ('Gus', 4), ('of', 4)]</span>
0775 </pre></div>
0776
0777 <p>Four out of five of the most common words are stop words, which don’t tell you much about the text. If you consider stop words while doing word frequency analysis, then you won’t be able to derive meaningful insights from the input text. This is why removing stop words is so important.</p>
0778 <h2 id="part-of-speech-tagging">Part of Speech Tagging</h2>
0779 <p><strong>Part of speech</strong> or <strong>POS</strong> is a grammatical role that explains how a particular word is used in a sentence. There are eight parts of speech:</p>
0780 <ol>
0781 <li>Noun</li>
0782 <li>Pronoun</li>
0783 <li>Adjective</li>
0784 <li>Verb</li>
0785 <li>Adverb</li>
0786 <li>Preposition</li>
0787 <li>Conjunction</li>
0788 <li>Interjection</li>
0789 </ol>
0790 <p><strong>Part of speech tagging</strong> is the process of assigning a <strong>POS tag</strong> to each token depending on its usage in the sentence. POS tags are useful for assigning a syntactic category like <strong>noun</strong> or <strong>verb</strong> to each word.</p>
0791 <p>In spaCy, POS tags are available as an attribute on the <code>Token</code> object:</p>
0792 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">about_doc</span><span class="p">:</span>
0793 <span class="gp">... </span> <span class="nb">print</span> <span class="p">(</span><span class="n">token</span><span class="p">,</span> <span class="n">token</span><span class="o">.</span><span class="n">tag_</span><span class="p">,</span> <span class="n">token</span><span class="o">.</span><span class="n">pos_</span><span class="p">,</span> <span class="n">spacy</span><span class="o">.</span><span class="n">explain</span><span class="p">(</span><span class="n">token</span><span class="o">.</span><span class="n">tag_</span><span class="p">))</span>
0794 <span class="gp">...</span>
0795 <span class="go">Gus NNP PROPN noun, proper singular</span>
0796 <span class="go">Proto NNP PROPN noun, proper singular</span>
0797 <span class="go">is VBZ VERB verb, 3rd person singular present</span>
0798 <span class="go">a DT DET determiner</span>
0799 <span class="go">Python NNP PROPN noun, proper singular</span>
0800 <span class="go">developer NN NOUN noun, singular or mass</span>
0801 <span class="go">currently RB ADV adverb</span>
0802 <span class="go">working VBG VERB verb, gerund or present participle</span>
0803 <span class="go">for IN ADP conjunction, subordinating or preposition</span>
0804 <span class="go">a DT DET determiner</span>
0805 <span class="go">London NNP PROPN noun, proper singular</span>
0806 <span class="go">- HYPH PUNCT punctuation mark, hyphen</span>
0807 <span class="go">based VBN VERB verb, past participle</span>
0808 <span class="go">Fintech NNP PROPN noun, proper singular</span>
0809 <span class="go">company NN NOUN noun, singular or mass</span>
0810 <span class="go">. . PUNCT punctuation mark, sentence closer</span>
0811 <span class="go">He PRP PRON pronoun, personal</span>
0812 <span class="go">is VBZ VERB verb, 3rd person singular present</span>
0813 <span class="go">interested JJ ADJ adjective</span>
0814 <span class="go">in IN ADP conjunction, subordinating or preposition</span>
0815 <span class="go">learning VBG VERB verb, gerund or present participle</span>
0816 <span class="go">Natural NNP PROPN noun, proper singular</span>
0817 <span class="go">Language NNP PROPN noun, proper singular</span>
0818 <span class="go">Processing NNP PROPN noun, proper singular</span>
0819 <span class="go">. . PUNCT punctuation mark, sentence closer</span>
0820 </pre></div>
0821
0822 <p>Here, two attributes of the <code>Token</code> class are accessed:</p>
0823 <ol>
0824 <li><strong><code>tag_</code></strong> lists the fine-grained part of speech.</li>
0825 <li><strong><code>pos_</code></strong> lists the coarse-grained part of speech.</li>
0826 </ol>
0827 <p><code>spacy.explain</code> gives descriptive details about a particular POS tag. spaCy provides a <a href="https://spaCy.io/api/annotation#pos-tagging">complete tag list</a> along with an explanation for each tag.</p>
0828 <p>Using POS tags, you can extract a particular category of words:</p>
0829 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">nouns</span> <span class="o">=</span> <span class="p">[]</span>
0830 <span class="gp">>>> </span><span class="n">adjectives</span> <span class="o">=</span> <span class="p">[]</span>
0831 <span class="gp">>>> </span><span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">about_doc</span><span class="p">:</span>
0832 <span class="gp">... </span> <span class="k">if</span> <span class="n">token</span><span class="o">.</span><span class="n">pos_</span> <span class="o">==</span> <span class="s1">'NOUN'</span><span class="p">:</span>
0833 <span class="gp">... </span> <span class="n">nouns</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">token</span><span class="p">)</span>
0834 <span class="gp">... </span> <span class="k">if</span> <span class="n">token</span><span class="o">.</span><span class="n">pos_</span> <span class="o">==</span> <span class="s1">'ADJ'</span><span class="p">:</span>
0835 <span class="gp">... </span> <span class="n">adjectives</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">token</span><span class="p">)</span>
0836 <span class="gp">...</span>
0837 <span class="gp">>>> </span><span class="n">nouns</span>
0838 <span class="go">[developer, company]</span>
0839 <span class="gp">>>> </span><span class="n">adjectives</span>
0840 <span class="go">[interested]</span>
0841 </pre></div>
0842
0843 <p>You can use this to derive insights, remove the most common nouns, or see which adjectives are used for a particular noun.</p>
0844 <h2 id="visualization-using-displacy">Visualization: Using displaCy</h2>
0845 <p>spaCy comes with a built-in visualizer called <strong>displaCy</strong>. You can use it to visualize a <strong>dependency parse</strong> or <strong>named entities</strong> in a browser or a <a href="https://realpython.com/jupyter-notebook-introduction/">Jupyter notebook</a>.</p>
0846 <p>You can use displaCy to find POS tags for tokens:</p>
0847 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">spacy</span> <span class="k">import</span> <span class="n">displacy</span>
0848 <span class="gp">>>> </span><span class="n">about_interest_text</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'He is interested in learning'</span>
0849 <span class="gp">... </span> <span class="s1">' Natural Language Processing.'</span><span class="p">)</span>
0850 <span class="gp">>>> </span><span class="n">about_interest_doc</span> <span class="o">=</span> <span class="n">nlp</span><span class="p">(</span><span class="n">about_interest_text</span><span class="p">)</span>
0851 <span class="gp">>>> </span><span class="n">displacy</span><span class="o">.</span><span class="n">serve</span><span class="p">(</span><span class="n">about_interest_doc</span><span class="p">,</span> <span class="n">style</span><span class="o">=</span><span class="s1">'dep'</span><span class="p">)</span>
0852 </pre></div>
0853
0854 <p>The above code will spin a simple web server. You can see the visualization by opening <a href="http://127.0.0.1:5000">http://127.0.0.1:5000</a> in your browser:</p>
0855 <figure class="figure mx-auto d-block"><a href="https://files.realpython.com/media/displacy_pos_tags.45059f2bf851.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/displacy_pos_tags.45059f2bf851.png" width="2630" height="600" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/displacy_pos_tags.45059f2bf851.png&w=657&sig=a49d6b5a0e5952aea59c0241f61fb09440bb326b 657w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/displacy_pos_tags.45059f2bf851.png&w=1315&sig=858218e45ae1e23a87ad42204154aeae77c9cc0c 1315w, https://files.realpython.com/media/displacy_pos_tags.45059f2bf851.png 2630w" sizes="75vw" alt="Displacy: Part of Speech Tagging Demo"/></a><figcaption class="figure-caption text-center">displaCy: Part of Speech Tagging Demo</figcaption></figure>
0856
0857 <p>In the image above, each token is assigned a POS tag written just below the token.</p>
0858 <div class="alert alert-primary" role="alert">
0859 <p><strong>Note:</strong> Here’s how you can use displaCy in a Jupyter notebook:</p>
0860 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">displacy</span><span class="o">.</span><span class="n">render</span><span class="p">(</span><span class="n">about_interest_doc</span><span class="p">,</span> <span class="n">style</span><span class="o">=</span><span class="s1">'dep'</span><span class="p">,</span> <span class="n">jupyter</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
0861 </pre></div>
0862
0863 </div>
0864 <h2 id="preprocessing-functions">Preprocessing Functions</h2>
0865 <p>You can create a <strong>preprocessing function</strong> that takes text as input and applies the following operations:</p>
0866 <ul>
0867 <li>Lowercases the text</li>
0868 <li>Lemmatizes each token</li>
0869 <li>Removes punctuation symbols</li>
0870 <li>Removes stop words</li>
0871 </ul>
0872 <p>A preprocessing function converts text to an analyzable format. It’s necessary for most NLP tasks. Here’s an example:</p>
0873 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">def</span> <span class="nf">is_token_allowed</span><span class="p">(</span><span class="n">token</span><span class="p">):</span>
0874 <span class="gp">... </span> <span class="sd">'''</span>
0875 <span class="gp">... </span><span class="sd"> Only allow valid tokens which are not stop words</span>
0876 <span class="gp">... </span><span class="sd"> and punctuation symbols.</span>
0877 <span class="gp">... </span><span class="sd"> '''</span>
0878 <span class="gp">... </span> <span class="k">if</span> <span class="p">(</span><span class="ow">not</span> <span class="n">token</span> <span class="ow">or</span> <span class="ow">not</span> <span class="n">token</span><span class="o">.</span><span class="n">string</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span> <span class="ow">or</span>
0879 <span class="gp">... </span> <span class="n">token</span><span class="o">.</span><span class="n">is_stop</span> <span class="ow">or</span> <span class="n">token</span><span class="o">.</span><span class="n">is_punct</span><span class="p">):</span>
0880 <span class="gp">... </span> <span class="k">return</span> <span class="kc">False</span>
0881 <span class="gp">... </span> <span class="k">return</span> <span class="kc">True</span>
0882 <span class="gp">...</span>
0883 <span class="gp">>>> </span><span class="k">def</span> <span class="nf">preprocess_token</span><span class="p">(</span><span class="n">token</span><span class="p">):</span>
0884 <span class="gp">... </span> <span class="c1"># Reduce token to its lowercase lemma form</span>
0885 <span class="gp">... </span> <span class="k">return</span> <span class="n">token</span><span class="o">.</span><span class="n">lemma_</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span>
0886 <span class="gp">...</span>
0887 <span class="gp">>>> </span><span class="n">complete_filtered_tokens</span> <span class="o">=</span> <span class="p">[</span><span class="n">preprocess_token</span><span class="p">(</span><span class="n">token</span><span class="p">)</span>
0888 <span class="gp">... </span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">complete_doc</span> <span class="k">if</span> <span class="n">is_token_allowed</span><span class="p">(</span><span class="n">token</span><span class="p">)]</span>
0889 <span class="gp">>>> </span><span class="n">complete_filtered_tokens</span>
0890 <span class="go">['gus', 'proto', 'python', 'developer', 'currently', 'work',</span>
0891 <span class="go">'london', 'base', 'fintech', 'company', 'interested', 'learn',</span>
0892 <span class="go">'natural', 'language', 'processing', 'developer', 'conference',</span>
0893 <span class="go">'happen', '21', 'july', '2019', 'london', 'title',</span>
0894 <span class="go">'applications', 'natural', 'language', 'processing', 'helpline',</span>
0895 <span class="go">'number', 'available', '+1', '1234567891', 'gus', 'help',</span>
0896 <span class="go">'organize', 'keep', 'organize', 'local', 'python', 'meetup',</span>
0897 <span class="go">'internal', 'talk', 'workplace', 'gus', 'present', 'talk', 'talk',</span>
0898 <span class="go">'introduce', 'reader', 'use', 'case', 'natural', 'language',</span>
0899 <span class="go">'processing', 'fintech', 'apart', 'work', 'passionate', 'music',</span>
0900 <span class="go">'gus', 'learn', 'play', 'piano', 'enrol', 'weekend', 'batch',</span>
0901 <span class="go">'great', 'piano', 'academy', 'great', 'piano', 'academy',</span>
0902 <span class="go">'situate', 'mayfair', 'city', 'london', 'world', 'class',</span>
0903 <span class="go">'piano', 'instructor']</span>
0904 </pre></div>
0905
0906 <p>Note that the <code>complete_filtered_tokens</code> does not contain any stop word or punctuation symbols and consists of lemmatized lowercase tokens.</p>
0907 <h2 id="rule-based-matching-using-spacy">Rule-Based Matching Using spaCy</h2>
0908 <p><strong>Rule-based matching</strong> is one of the steps in extracting information from unstructured text. It’s used to identify and extract tokens and phrases according to patterns (such as lowercase) and grammatical features (such as part of speech).</p>
0909 <p>Rule-based matching can use <a href="https://en.wikipedia.org/wiki/Regular_expression">regular expressions</a> to extract entities (such as phone numbers) from an unstructured text. It’s different from extracting text using regular expressions only in the sense that regular expressions don’t consider the lexical and grammatical attributes of the text.</p>
0910 <p>With rule-based matching, you can extract a first name and a last name, which are always <strong>proper nouns</strong>:</p>
0911 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">spacy.matcher</span> <span class="k">import</span> <span class="n">Matcher</span>
0912 <span class="gp">>>> </span><span class="n">matcher</span> <span class="o">=</span> <span class="n">Matcher</span><span class="p">(</span><span class="n">nlp</span><span class="o">.</span><span class="n">vocab</span><span class="p">)</span>
0913 <span class="gp">>>> </span><span class="k">def</span> <span class="nf">extract_full_name</span><span class="p">(</span><span class="n">nlp_doc</span><span class="p">):</span>
0914 <span class="gp">... </span> <span class="n">pattern</span> <span class="o">=</span> <span class="p">[{</span><span class="s1">'POS'</span><span class="p">:</span> <span class="s1">'PROPN'</span><span class="p">},</span> <span class="p">{</span><span class="s1">'POS'</span><span class="p">:</span> <span class="s1">'PROPN'</span><span class="p">}]</span>
0915 <span class="gp">... </span> <span class="n">matcher</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="s1">'FULL_NAME'</span><span class="p">,</span> <span class="kc">None</span><span class="p">,</span> <span class="n">pattern</span><span class="p">)</span>
0916 <span class="gp">... </span> <span class="n">matches</span> <span class="o">=</span> <span class="n">matcher</span><span class="p">(</span><span class="n">nlp_doc</span><span class="p">)</span>
0917 <span class="gp">... </span> <span class="k">for</span> <span class="n">match_id</span><span class="p">,</span> <span class="n">start</span><span class="p">,</span> <span class="n">end</span> <span class="ow">in</span> <span class="n">matches</span><span class="p">:</span>
0918 <span class="gp">... </span> <span class="n">span</span> <span class="o">=</span> <span class="n">nlp_doc</span><span class="p">[</span><span class="n">start</span><span class="p">:</span><span class="n">end</span><span class="p">]</span>
0919 <span class="gp">... </span> <span class="k">return</span> <span class="n">span</span><span class="o">.</span><span class="n">text</span>
0920 <span class="gp">...</span>
0921 <span class="gp">>>> </span><span class="n">extract_full_name</span><span class="p">(</span><span class="n">about_doc</span><span class="p">)</span>
0922 <span class="go">'Gus Proto'</span>
0923 </pre></div>
0924
0925 <p>In this example, <code>pattern</code> is a list of objects that defines the combination of tokens to be matched. Both POS tags in it are <code>PROPN</code> (proper noun). So, the <code>pattern</code> consists of two objects in which the POS tags for both tokens should be <code>PROPN</code>. This pattern is then added to <code>Matcher</code> using <code>FULL_NAME</code> and the the <code>match_id</code>. Finally, matches are obtained with their starting and end indexes.</p>
0926 <p>You can also use rule-based matching to extract phone numbers:</p>
0927 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">spacy.matcher</span> <span class="k">import</span> <span class="n">Matcher</span>
0928 <span class="gp">>>> </span><span class="n">matcher</span> <span class="o">=</span> <span class="n">Matcher</span><span class="p">(</span><span class="n">nlp</span><span class="o">.</span><span class="n">vocab</span><span class="p">)</span>
0929 <span class="gp">>>> </span><span class="n">conference_org_text</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'There is a developer conference'</span>
0930 <span class="gp">... </span> <span class="s1">'happening on 21 July 2019 in London. It is titled'</span>
0931 <span class="gp">... </span> <span class="s1">' "Applications of Natural Language Processing".'</span>
0932 <span class="gp">... </span> <span class="s1">' There is a helpline number available'</span>
0933 <span class="gp">... </span> <span class="s1">' at (123) 456-789'</span><span class="p">)</span>
0934 <span class="gp">...</span>
0935 <span class="gp">>>> </span><span class="k">def</span> <span class="nf">extract_phone_number</span><span class="p">(</span><span class="n">nlp_doc</span><span class="p">):</span>
0936 <span class="gp">... </span> <span class="n">pattern</span> <span class="o">=</span> <span class="p">[{</span><span class="s1">'ORTH'</span><span class="p">:</span> <span class="s1">'('</span><span class="p">},</span> <span class="p">{</span><span class="s1">'SHAPE'</span><span class="p">:</span> <span class="s1">'ddd'</span><span class="p">},</span>
0937 <span class="gp">... </span> <span class="p">{</span><span class="s1">'ORTH'</span><span class="p">:</span> <span class="s1">')'</span><span class="p">},</span> <span class="p">{</span><span class="s1">'SHAPE'</span><span class="p">:</span> <span class="s1">'ddd'</span><span class="p">},</span>
0938 <span class="gp">... </span> <span class="p">{</span><span class="s1">'ORTH'</span><span class="p">:</span> <span class="s1">'-'</span><span class="p">,</span> <span class="s1">'OP'</span><span class="p">:</span> <span class="s1">'?'</span><span class="p">},</span>
0939 <span class="gp">... </span> <span class="p">{</span><span class="s1">'SHAPE'</span><span class="p">:</span> <span class="s1">'ddd'</span><span class="p">}]</span>
0940 <span class="gp">... </span> <span class="n">matcher</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="s1">'PHONE_NUMBER'</span><span class="p">,</span> <span class="kc">None</span><span class="p">,</span> <span class="n">pattern</span><span class="p">)</span>
0941 <span class="gp">... </span> <span class="n">matches</span> <span class="o">=</span> <span class="n">matcher</span><span class="p">(</span><span class="n">nlp_doc</span><span class="p">)</span>
0942 <span class="gp">... </span> <span class="k">for</span> <span class="n">match_id</span><span class="p">,</span> <span class="n">start</span><span class="p">,</span> <span class="n">end</span> <span class="ow">in</span> <span class="n">matches</span><span class="p">:</span>
0943 <span class="gp">... </span> <span class="n">span</span> <span class="o">=</span> <span class="n">nlp_doc</span><span class="p">[</span><span class="n">start</span><span class="p">:</span><span class="n">end</span><span class="p">]</span>
0944 <span class="gp">... </span> <span class="k">return</span> <span class="n">span</span><span class="o">.</span><span class="n">text</span>
0945 <span class="gp">...</span>
0946 <span class="gp">>>> </span><span class="n">conference_org_doc</span> <span class="o">=</span> <span class="n">nlp</span><span class="p">(</span><span class="n">conference_org_text</span><span class="p">)</span>
0947 <span class="gp">>>> </span><span class="n">extract_phone_number</span><span class="p">(</span><span class="n">conference_org_doc</span><span class="p">)</span>
0948 <span class="go">'(123) 456-789'</span>
0949 </pre></div>
0950
0951 <p>In this example, only the pattern is updated in order to match phone numbers from the previous example. Here, some attributes of the token are also used:</p>
0952 <ul>
0953 <li><strong><code>ORTH</code></strong> gives the exact text of the token.</li>
0954 <li><strong><code>SHAPE</code></strong> transforms the token string to show orthographic features.</li>
0955 <li><strong><code>OP</code></strong> defines operators. Using <code>?</code> as a value means that the pattern is optional, meaning it can match 0 or 1 times.</li>
0956 </ul>
0957 <div class="alert alert-primary" role="alert">
0958 <p><strong>Note:</strong> For simplicity, phone numbers are assumed to be of a particular format: <code>(123) 456-789</code>. You can change this depending on your use case.</p>
0959 </div>
0960 <p>Rule-based matching helps you identify and extract tokens and phrases according to lexical patterns (such as lowercase) and grammatical features(such as part of speech).</p>
0961 <h2 id="dependency-parsing-using-spacy">Dependency Parsing Using spaCy</h2>
0962 <p><strong>Dependency parsing</strong> is the process of extracting the dependency parse of a sentence to represent its grammatical structure. It defines the dependency relationship between <strong>headwords</strong> and their <strong>dependents</strong>. The head of a sentence has no dependency and is called the <strong>root of the sentence</strong>. The <strong>verb</strong> is usually the head of the sentence. All other words are linked to the headword.</p>
0963 <p>The dependencies can be mapped in a directed graph representation: </p>
0964 <ul>
0965 <li>Words are the nodes.</li>
0966 <li>The grammatical relationships are the edges.</li>
0967 </ul>
0968 <p>Dependency parsing helps you know what role a word plays in the text and how different words relate to each other. It’s also used in <strong>shallow parsing</strong> and named entity recognition.</p>
0969 <p>Here’s how you can use dependency parsing to see the relationships between words:</p>
0970 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">piano_text</span> <span class="o">=</span> <span class="s1">'Gus is learning piano'</span>
0971 <span class="gp">>>> </span><span class="n">piano_doc</span> <span class="o">=</span> <span class="n">nlp</span><span class="p">(</span><span class="n">piano_text</span><span class="p">)</span>
0972 <span class="gp">>>> </span><span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">piano_doc</span><span class="p">:</span>
0973 <span class="gp">... </span> <span class="nb">print</span> <span class="p">(</span><span class="n">token</span><span class="o">.</span><span class="n">text</span><span class="p">,</span> <span class="n">token</span><span class="o">.</span><span class="n">tag_</span><span class="p">,</span> <span class="n">token</span><span class="o">.</span><span class="n">head</span><span class="o">.</span><span class="n">text</span><span class="p">,</span> <span class="n">token</span><span class="o">.</span><span class="n">dep_</span><span class="p">)</span>
0974 <span class="gp">...</span>
0975 <span class="go">Gus NNP learning nsubj</span>
0976 <span class="go">is VBZ learning aux</span>
0977 <span class="go">learning VBG learning ROOT</span>
0978 <span class="go">piano NN learning dobj</span>
0979 </pre></div>
0980
0981 <p>In this example, the sentence contains three relationships:</p>
0982 <ol>
0983 <li><strong><code>nsubj</code></strong> is the subject of the word. Its headword is a verb.</li>
0984 <li><strong><code>aux</code></strong> is an auxiliary word. Its headword is a verb.</li>
0985 <li><strong><code>dobj</code></strong> is the direct object of the verb. Its headword is a verb.</li>
0986 </ol>
0987 <p>There is a detailed <a href="https://nlp.stanford.edu/software/dependencies_manual.pdf">list of relationships</a> with descriptions. You can use displaCy to visualize the dependency tree:</p>
0988 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">displacy</span><span class="o">.</span><span class="n">serve</span><span class="p">(</span><span class="n">piano_doc</span><span class="p">,</span> <span class="n">style</span><span class="o">=</span><span class="s1">'dep'</span><span class="p">)</span>
0989 </pre></div>
0990
0991 <p>This code will produce a visualization that can be accessed by opening <a href="http://127.0.0.1:5000">http://127.0.0.1:5000</a> in your browser:</p>
0992 <figure class="figure mx-auto d-block"><a href="https://files.realpython.com/media/displacy_dependency_parse.de72f9b1d115.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/displacy_dependency_parse.de72f9b1d115.png" width="1278" height="596" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/displacy_dependency_parse.de72f9b1d115.png&w=319&sig=111728c07cf2e1f64b8419cfce8a5f880c244d03 319w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/displacy_dependency_parse.de72f9b1d115.png&w=639&sig=f90a72529d7bc2d2dd944af3c02bbf487b65aaf3 639w, https://files.realpython.com/media/displacy_dependency_parse.de72f9b1d115.png 1278w" sizes="75vw" alt="Displacy: Dependency Parse Demo"/></a><figcaption class="figure-caption text-center">displaCy: Dependency Parse Demo</figcaption></figure>
0993
0994 <p>This image shows you that the subject of the sentence is the proper noun <code>Gus</code> and that it has a <code>learn</code> relationship with <code>piano</code>.</p>
0995 <h2 id="navigating-the-tree-and-subtree">Navigating the Tree and Subtree</h2>
0996 <p>The dependency parse tree has all the properties of a <a href="https://en.wikipedia.org/wiki/Tree_(data_structure)">tree</a>. This tree contains information about sentence structure and grammar and can be traversed in different ways to extract relationships.</p>
0997 <p>spaCy provides attributes like <code>children</code>, <code>lefts</code>, <code>rights</code>, and <code>subtree</code> to navigate the parse tree:</p>
0998 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">one_line_about_text</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'Gus Proto is a Python developer'</span>
0999 <span class="gp">... </span> <span class="s1">' currently working for a London-based Fintech company'</span><span class="p">)</span>
1000 <span class="gp">>>> </span><span class="n">one_line_about_doc</span> <span class="o">=</span> <span class="n">nlp</span><span class="p">(</span><span class="n">one_line_about_text</span><span class="p">)</span>
1001 <span class="gp">>>> </span><span class="c1"># Extract children of `developer`</span>
1002 <span class="gp">>>> </span><span class="nb">print</span><span class="p">([</span><span class="n">token</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">one_line_about_doc</span><span class="p">[</span><span class="mi">5</span><span class="p">]</span><span class="o">.</span><span class="n">children</span><span class="p">])</span>
1003 <span class="go">['a', 'Python', 'working']</span>
1004 <span class="gp">>>> </span><span class="c1"># Extract previous neighboring node of `developer`</span>
1005 <span class="gp">>>> </span><span class="nb">print</span> <span class="p">(</span><span class="n">one_line_about_doc</span><span class="p">[</span><span class="mi">5</span><span class="p">]</span><span class="o">.</span><span class="n">nbor</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">))</span>
1006 <span class="go">Python</span>
1007 <span class="gp">>>> </span><span class="c1"># Extract next neighboring node of `developer`</span>
1008 <span class="gp">>>> </span><span class="nb">print</span> <span class="p">(</span><span class="n">one_line_about_doc</span><span class="p">[</span><span class="mi">5</span><span class="p">]</span><span class="o">.</span><span class="n">nbor</span><span class="p">())</span>
1009 <span class="go">currently</span>
1010 <span class="gp">>>> </span><span class="c1"># Extract all tokens on the left of `developer`</span>
1011 <span class="gp">>>> </span><span class="nb">print</span><span class="p">([</span><span class="n">token</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">one_line_about_doc</span><span class="p">[</span><span class="mi">5</span><span class="p">]</span><span class="o">.</span><span class="n">lefts</span><span class="p">])</span>
1012 <span class="go">['a', 'Python']</span>
1013 <span class="gp">>>> </span><span class="c1"># Extract tokens on the right of `developer`</span>
1014 <span class="gp">>>> </span><span class="nb">print</span><span class="p">([</span><span class="n">token</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">one_line_about_doc</span><span class="p">[</span><span class="mi">5</span><span class="p">]</span><span class="o">.</span><span class="n">rights</span><span class="p">])</span>
1015 <span class="go">['working']</span>
1016 <span class="gp">>>> </span><span class="c1"># Print subtree of `developer`</span>
1017 <span class="gp">>>> </span><span class="nb">print</span> <span class="p">(</span><span class="nb">list</span><span class="p">(</span><span class="n">one_line_about_doc</span><span class="p">[</span><span class="mi">5</span><span class="p">]</span><span class="o">.</span><span class="n">subtree</span><span class="p">))</span>
1018 <span class="go">[a, Python, developer, currently, working, for, a, London, -,</span>
1019 <span class="go">based, Fintech, company]</span>
1020 </pre></div>
1021
1022 <p>You can construct a function that takes a subtree as an argument and returns a string by merging words in it:</p>
1023 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">def</span> <span class="nf">flatten_tree</span><span class="p">(</span><span class="n">tree</span><span class="p">):</span>
1024 <span class="gp">... </span> <span class="k">return</span> <span class="s1">''</span><span class="o">.</span><span class="n">join</span><span class="p">([</span><span class="n">token</span><span class="o">.</span><span class="n">text_with_ws</span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="nb">list</span><span class="p">(</span><span class="n">tree</span><span class="p">)])</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
1025 <span class="gp">...</span>
1026 <span class="gp">>>> </span><span class="c1"># Print flattened subtree of `developer`</span>
1027 <span class="gp">>>> </span><span class="nb">print</span> <span class="p">(</span><span class="n">flatten_tree</span><span class="p">(</span><span class="n">one_line_about_doc</span><span class="p">[</span><span class="mi">5</span><span class="p">]</span><span class="o">.</span><span class="n">subtree</span><span class="p">))</span>
1028 <span class="go">a Python developer currently working for a London-based Fintech company</span>
1029 </pre></div>
1030
1031 <p>You can use this function to print all the tokens in a subtree.</p>
1032 <h2 id="shallow-parsing">Shallow Parsing</h2>
1033 <p><strong>Shallow parsing</strong>, or <strong>chunking</strong>, is the process of extracting phrases from unstructured text. Chunking groups adjacent tokens into phrases on the basis of their POS tags. There are some standard well-known chunks such as noun phrases, verb phrases, and prepositional phrases.</p>
1034 <h3 id="noun-phrase-detection">Noun Phrase Detection</h3>
1035 <p>A noun phrase is a phrase that has a noun as its head. It could also include other kinds of words, such as adjectives, ordinals, determiners. Noun phrases are useful for explaining the context of the sentence. They help you infer <em>what</em> is being talked about in the sentence.</p>
1036 <p>spaCy has the property <code>noun_chunks</code> on <code>Doc</code> object. You can use it to extract noun phrases:</p>
1037 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">conference_text</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'There is a developer conference'</span>
1038 <span class="gp">... </span> <span class="s1">' happening on 21 July 2019 in London.'</span><span class="p">)</span>
1039 <span class="gp">>>> </span><span class="n">conference_doc</span> <span class="o">=</span> <span class="n">nlp</span><span class="p">(</span><span class="n">conference_text</span><span class="p">)</span>
1040 <span class="gp">>>> </span><span class="c1"># Extract Noun Phrases</span>
1041 <span class="gp">>>> </span><span class="k">for</span> <span class="n">chunk</span> <span class="ow">in</span> <span class="n">conference_doc</span><span class="o">.</span><span class="n">noun_chunks</span><span class="p">:</span>
1042 <span class="gp">... </span> <span class="nb">print</span> <span class="p">(</span><span class="n">chunk</span><span class="p">)</span>
1043 <span class="gp">...</span>
1044 <span class="go">a developer conference</span>
1045 <span class="go">21 July</span>
1046 <span class="go">London</span>
1047 </pre></div>
1048
1049 <p>By looking at noun phrases, you can get information about your text. For example, <code>a developer conference</code> indicates that the text mentions a conference, while the date <code>21 July</code> lets you know that conference is scheduled for <code>21 July</code>. You can figure out whether the conference is in the past or the future. <code>London</code> tells you that the conference is in <code>London</code>.</p>
1050 <h3 id="verb-phrase-detection">Verb Phrase Detection</h3>
1051 <p>A <strong>verb phrase</strong> is a syntactic unit composed of at least one verb. This verb can be followed by other chunks, such as noun phrases. Verb phrases are useful for understanding the actions that nouns are involved in. </p>
1052 <p>spaCy has no built-in functionality to extract verb phrases, so you’ll need a library called <a href="https://chartbeat-labs.github.io/textacy/"><code>textacy</code></a>:</p>
1053 <div class="alert alert-primary" role="alert">
1054 <p><strong>Note:</strong> </p>
1055 <p>You can use <code>pip</code> to install <code>textacy</code>:</p>
1056 <div class="highlight sh"><pre><span></span><span class="gp">$</span> pip install textacy
1057 </pre></div>
1058
1059 </div>
1060 <p>Now that you have <code>textacy</code> installed, you can use it to extract verb phrases based on grammar rules:</p>
1061 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">textacy</span>
1062 <span class="gp">>>> </span><span class="n">about_talk_text</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'The talk will introduce reader about Use'</span>
1063 <span class="gp">... </span> <span class="s1">' cases of Natural Language Processing in'</span>
1064 <span class="gp">... </span> <span class="s1">' Fintech'</span><span class="p">)</span>
1065 <span class="gp">>>> </span><span class="n">pattern</span> <span class="o">=</span> <span class="sa">r</span><span class="s1">'(<VERB>?<ADV>*<VERB>+)'</span>
1066 <span class="gp">>>> </span><span class="n">about_talk_doc</span> <span class="o">=</span> <span class="n">textacy</span><span class="o">.</span><span class="n">make_spacy_doc</span><span class="p">(</span><span class="n">about_talk_text</span><span class="p">,</span>
1067 <span class="gp">... </span> <span class="n">lang</span><span class="o">=</span><span class="s1">'en_core_web_sm'</span><span class="p">)</span>
1068 <span class="gp">>>> </span><span class="n">verb_phrases</span> <span class="o">=</span> <span class="n">textacy</span><span class="o">.</span><span class="n">extract</span><span class="o">.</span><span class="n">pos_regex_matches</span><span class="p">(</span><span class="n">about_talk_doc</span><span class="p">,</span> <span class="n">pattern</span><span class="p">)</span>
1069 <span class="gp">>>> </span><span class="c1"># Print all Verb Phrase</span>
1070 <span class="gp">>>> </span><span class="k">for</span> <span class="n">chunk</span> <span class="ow">in</span> <span class="n">verb_phrases</span><span class="p">:</span>
1071 <span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">chunk</span><span class="o">.</span><span class="n">text</span><span class="p">)</span>
1072 <span class="gp">...</span>
1073 <span class="go">will introduce</span>
1074 <span class="gp">>>> </span><span class="c1"># Extract Noun Phrase to explain what nouns are involved</span>
1075 <span class="gp">>>> </span><span class="k">for</span> <span class="n">chunk</span> <span class="ow">in</span> <span class="n">about_talk_doc</span><span class="o">.</span><span class="n">noun_chunks</span><span class="p">:</span>
1076 <span class="gp">... </span> <span class="nb">print</span> <span class="p">(</span><span class="n">chunk</span><span class="p">)</span>
1077 <span class="gp">...</span>
1078 <span class="go">The talk</span>
1079 <span class="go">reader</span>
1080 <span class="go">Use cases</span>
1081 <span class="go">Natural Language Processing</span>
1082 <span class="go">Fintech</span>
1083 </pre></div>
1084
1085 <p>In this example, the verb phrase <code>introduce</code> indicates that something will be introduced. By looking at noun phrases, you can see that there is a <code>talk</code> that will <code>introduce</code> the <code>reader</code> to <code>use cases</code> of <code>Natural Language Processing</code> or <code>Fintech</code>.</p>
1086 <p>The above code extracts all the verb phrases <a href="https://chartbeat-labs.github.io/textacy/api_reference/information_extraction.html?highlight=pos#textacy.extract.pos_regex_matches">using a regular expression pattern</a> of POS tags. You can tweak the pattern for verb phrases depending upon your use case.</p>
1087 <div class="alert alert-primary" role="alert">
1088 <p><strong>Note:</strong> In the previous example, you could have also done dependency parsing to see what the <a href="https://nlp.stanford.edu/software/dependencies_manual.pdf">relationships</a> between the words were.</p>
1089 </div>
1090 <h2 id="named-entity-recognition">Named Entity Recognition</h2>
1091 <p><strong>Named Entity Recognition</strong> (NER) is the process of locating <strong>named entities</strong> in unstructured text and then classifying them into pre-defined categories, such as person names, organizations, locations, monetary values, percentages, time expressions, and so on.</p>
1092 <p>You can use <strong>NER</strong> to know more about the meaning of your text. For example, you could use it to populate tags for a set of documents in order to improve the keyword search. You could also use it to categorize customer support tickets into relevant categories.</p>
1093 <p>spaCy has the property <code>ents</code> on <code>Doc</code> objects. You can use it to extract named entities:</p>
1094 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">piano_class_text</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'Great Piano Academy is situated'</span>
1095 <span class="gp">... </span> <span class="s1">' in Mayfair or the City of London and has'</span>
1096 <span class="gp">... </span> <span class="s1">' world-class piano instructors.'</span><span class="p">)</span>
1097 <span class="gp">>>> </span><span class="n">piano_class_doc</span> <span class="o">=</span> <span class="n">nlp</span><span class="p">(</span><span class="n">piano_class_text</span><span class="p">)</span>
1098 <span class="gp">>>> </span><span class="k">for</span> <span class="n">ent</span> <span class="ow">in</span> <span class="n">piano_class_doc</span><span class="o">.</span><span class="n">ents</span><span class="p">:</span>
1099 <span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">ent</span><span class="o">.</span><span class="n">text</span><span class="p">,</span> <span class="n">ent</span><span class="o">.</span><span class="n">start_char</span><span class="p">,</span> <span class="n">ent</span><span class="o">.</span><span class="n">end_char</span><span class="p">,</span>
1100 <span class="gp">... </span> <span class="n">ent</span><span class="o">.</span><span class="n">label_</span><span class="p">,</span> <span class="n">spacy</span><span class="o">.</span><span class="n">explain</span><span class="p">(</span><span class="n">ent</span><span class="o">.</span><span class="n">label_</span><span class="p">))</span>
1101 <span class="gp">...</span>
1102 <span class="go">Great Piano Academy 0 19 ORG Companies, agencies, institutions, etc.</span>
1103 <span class="go">Mayfair 35 42 GPE Countries, cities, states</span>
1104 <span class="go">the City of London 46 64 GPE Countries, cities, states</span>
1105 </pre></div>
1106
1107 <p>In the above example, <code>ent</code> is a <a href="https://spacy.io/api/span"><code>Span</code></a> object with various attributes:</p>
1108 <ul>
1109 <li><strong><code>text</code></strong> gives the Unicode text representation of the entity.</li>
1110 <li><strong><code>start_char</code></strong> denotes the character offset for the start of the entity.</li>
1111 <li><strong><code>end_char</code></strong> denotes the character offset for the end of the entity.</li>
1112 <li><strong><code>label_</code></strong> gives the label of the entity.</li>
1113 </ul>
1114 <p><code>spacy.explain</code> gives descriptive details about an entity label. The spaCy model has a pre-trained <a href="https://spaCy.io/api/annotation#named-entities">list of entity classes</a>. You can use displaCy to visualize these entities:</p>
1115 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">displacy</span><span class="o">.</span><span class="n">serve</span><span class="p">(</span><span class="n">piano_class_doc</span><span class="p">,</span> <span class="n">style</span><span class="o">=</span><span class="s1">'ent'</span><span class="p">)</span>
1116 </pre></div>
1117
1118 <p>If you open <a href="http://127.0.0.1:5000">http://127.0.0.1:5000</a> in your browser, then you can see the visualization:</p>
1119 <figure class="figure mx-auto d-block"><a href="https://files.realpython.com/media/displacy_ner.1fba6869638f.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/displacy_ner.1fba6869638f.png" width="1930" height="140" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/displacy_ner.1fba6869638f.png&w=482&sig=18b93b0aed61930a6eedd37dbd12fbbce22733d4 482w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/displacy_ner.1fba6869638f.png&w=965&sig=f6b3cfb460053397a23a0eb49ebc22cf05dd15ab 965w, https://files.realpython.com/media/displacy_ner.1fba6869638f.png 1930w" sizes="75vw" alt="Displacy: Named Entity Recognition Demo"/></a><figcaption class="figure-caption text-center">displaCy: Named Entity Recognition Demo</figcaption></figure>
1120
1121 <p>You can use NER to redact people’s names from a text. For example, you might want to do this in order to hide personal information collected in a survey. You can use spaCy to do that:</p>
1122 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">survey_text</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'Out of 5 people surveyed, James Robert,'</span>
1123 <span class="gp">... </span> <span class="s1">' Julie Fuller and Benjamin Brooks like'</span>
1124 <span class="gp">... </span> <span class="s1">' apples. Kelly Cox and Matthew Evans'</span>
1125 <span class="gp">... </span> <span class="s1">' like oranges.'</span><span class="p">)</span>
1126 <span class="gp">...</span>
1127 <span class="gp">>>> </span><span class="k">def</span> <span class="nf">replace_person_names</span><span class="p">(</span><span class="n">token</span><span class="p">):</span>
1128 <span class="gp">... </span> <span class="k">if</span> <span class="n">token</span><span class="o">.</span><span class="n">ent_iob</span> <span class="o">!=</span> <span class="mi">0</span> <span class="ow">and</span> <span class="n">token</span><span class="o">.</span><span class="n">ent_type_</span> <span class="o">==</span> <span class="s1">'PERSON'</span><span class="p">:</span>
1129 <span class="gp">... </span> <span class="k">return</span> <span class="s1">'[REDACTED] '</span>
1130 <span class="gp">... </span> <span class="k">return</span> <span class="n">token</span><span class="o">.</span><span class="n">string</span>
1131 <span class="gp">...</span>
1132 <span class="gp">>>> </span><span class="k">def</span> <span class="nf">redact_names</span><span class="p">(</span><span class="n">nlp_doc</span><span class="p">):</span>
1133 <span class="gp">... </span> <span class="k">for</span> <span class="n">ent</span> <span class="ow">in</span> <span class="n">nlp_doc</span><span class="o">.</span><span class="n">ents</span><span class="p">:</span>
1134 <span class="gp">... </span> <span class="n">ent</span><span class="o">.</span><span class="n">merge</span><span class="p">()</span>
1135 <span class="gp">... </span> <span class="n">tokens</span> <span class="o">=</span> <span class="nb">map</span><span class="p">(</span><span class="n">replace_person_names</span><span class="p">,</span> <span class="n">nlp_doc</span><span class="p">)</span>
1136 <span class="gp">... </span> <span class="k">return</span> <span class="s1">''</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">tokens</span><span class="p">)</span>
1137 <span class="gp">...</span>
1138 <span class="gp">>>> </span><span class="n">survey_doc</span> <span class="o">=</span> <span class="n">nlp</span><span class="p">(</span><span class="n">survey_text</span><span class="p">)</span>
1139 <span class="gp">>>> </span><span class="n">redact_names</span><span class="p">(</span><span class="n">survey_doc</span><span class="p">)</span>
1140 <span class="go">'Out of 5 people surveyed, [REDACTED] , [REDACTED] and'</span>
1141 <span class="go">' [REDACTED] like apples. [REDACTED] and [REDACTED]'</span>
1142 <span class="go">' like oranges.'</span>
1143 </pre></div>
1144
1145 <p>In this example, <code>replace_person_names()</code> uses <code>ent_iob</code>. It gives the IOB code of the named entity tag using <a href="https://en.wikipedia.org/wiki/Inside%E2%80%93outside%E2%80%93beginning_(tagging)">inside-outside-beginning (IOB) tagging</a>. Here, it can assume a value other than zero, because zero means that no entity tag is set.</p>
1146 <h2 id="conclusion">Conclusion</h2>
1147 <p>spaCy is a powerful and advanced library that is gaining huge popularity for NLP applications due to its speed, ease of use, accuracy, and extensibility. Congratulations! You now know:</p>
1148 <ul>
1149 <li>What the foundational terms and concepts in NLP are</li>
1150 <li>How to implement those concepts in spaCy</li>
1151 <li>How to customize and extend built-in functionalities in spaCy</li>
1152 <li>How to perform basic statistical analysis on a text</li>
1153 <li>How to create a pipeline to process unstructured text</li>
1154 <li>How to parse a sentence and extract meaningful insights from it</li>
1155 </ul>
1156 <hr />
1157 <p><em>[ Improve Your Python With π Python Tricks π β Get a short & sweet Python Trick delivered to your inbox every couple of days. <a href="https://realpython.com/python-tricks/?utm_source=realpython&utm_medium=rss&utm_campaign=footer">>> Click here to learn more and see examples</a> ]</em></p>#
1158 datePublished: #Mon Sep 2 14:00:00 2019#
1159 dateUpdated: #Mon Sep 2 14:00:00 2019#
1160 # Person begin ####################
1161 name: #Real Python#
1162 # Person end ######################
1163 # Item end ########################
1164 # Item begin ######################
1165 id: #https://realpython.com/pycharm-guide/#
1166 title: #PyCharm for Productive Python Development (Guide)#
1167 link: #https://realpython.com/pycharm-guide/#
1168 description: #In this step-by-step tutorial, you'll learn how you can use PyCharm to be a more productive Python developer. PyCharm makes debugging and visualization easy so you can focus on business logic and just get the job done.#
1169 content: #<p>As a programmer, you should be focused on the business logic and creating useful applications for your users. In doing that, <a href="https://www.jetbrains.com/pycharm/">PyCharm</a> by <a href="https://www.jetbrains.com/">JetBrains</a> saves you a lot of time by taking care of the routine and by making a number of other tasks such as debugging and visualization easy. </p>
1170 <p><strong>In this article, you’ll learn about:</strong></p>
1171 <ul>
1172 <li>Installing PyCharm</li>
1173 <li>Writing code in PyCharm</li>
1174 <li>Running your code in PyCharm</li>
1175 <li>Debugging and testing your code in PyCharm</li>
1176 <li>Editing an existing project in PyCharm</li>
1177 <li>Searching and navigating in PyCharm</li>
1178 <li>Using Version Control in PyCharm</li>
1179 <li>Using Plugins and External Tools in PyCharm</li>
1180 <li>Using PyCharm Professional features, such as Django support and Scientific mode</li>
1181 </ul>
1182 <p>This article assumes that you’re familiar with Python development and already have some form of Python installed on your system. Python 3.6 will be used for this tutorial. Screenshots and demos provided are for macOS. Because PyCharm runs on all major platforms, you may see slightly different UI elements and may need to modify certain commands.</p>
1183 <div class="alert alert-primary" role="alert">
1184 <p><strong>Note</strong>: </p>
1185 <p>PyCharm comes in three editions: </p>
1186 <ol>
1187 <li><a href="https://www.jetbrains.com/pycharm-edu/">PyCharm Edu</a> is free and for educational purposes. </li>
1188 <li><a href="https://www.jetbrains.com/pycharm">PyCharm Community</a> is free as well and intended for pure Python development. </li>
1189 <li><a href="https://www.jetbrains.com/pycharm">PyCharm Professional</a> is paid, has everything the Community edition has and also is very well suited for Web and Scientific development with support for such frameworks as Django and Flask, Database and SQL, and scientific tools such as Jupyter.</li>
1190 </ol>
1191 <p>For more details on their differences, check out the <a href="https://www.jetbrains.com/pycharm/features/editions_comparison_matrix.html">PyCharm Editions Comparison Matrix</a> by JetBrains. The company also has <a href="https://www.jetbrains.com/pycharm/buy/#edition=discounts">special offers</a> for students, teachers, open source projects, and other cases.</p>
1192 </div>
1193 <div class="alert alert-warning" role="alert"><p><strong>Clone Repo:</strong> <a href="https://realpython.com/optins/view/alcazar-web-framework/" class="alert-link" data-toggle="modal" data-target="#modal-alcazar-web-framework" data-focus="false">Click here to clone the repo you'll use</a> to explore the project-focused features of PyCharm in this tutorial.</p></div>
1194
1195 <h2 id="installing-pycharm">Installing PyCharm</h2>
1196 <p>This article will use PyCharm Community Edition 2019.1 as it’s free and available on every major platform. Only the section about the professional features will use PyCharm Professional Edition 2019.1. </p>
1197 <p>The recommended way of installing PyCharm is with the <a href="https://www.jetbrains.com/toolbox/app/">JetBrains Toolbox App</a>. With its help, you’ll be able to install different JetBrains products or several versions of the same product, update, rollback, and easily remove any tool when necessary. You’ll also be able to quickly open any project in the right IDE and version.</p>
1198 <p>To install the Toolbox App, refer to the <a href="https://www.jetbrains.com/help/pycharm/installation-guide.html#toolbox">documentation</a> by JetBrains. It will automatically give you the right instructions depending on your OS. In case it didn’t recognize your OS correctly, you can always find it from the drop down list on the top right section: </p>
1199 <p><a href="https://files.realpython.com/media/pycharm-jetbrains-os-list.231740335aaa.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/pycharm-jetbrains-os-list.231740335aaa.png" width="1010" height="679" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-jetbrains-os-list.231740335aaa.png&w=252&sig=e331b2eb15a3c8b9396327dedc700bd2bcbbc9e3 252w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-jetbrains-os-list.231740335aaa.png&w=505&sig=4a0a0527b968050fb042a0565f5d6970d72ee1f9 505w, https://files.realpython.com/media/pycharm-jetbrains-os-list.231740335aaa.png 1010w" sizes="75vw" alt="List of OSes in the JetBrains website"/></a></p>
1200 <p>After installing, launch the app and accept the user agreement. Under the <em>Tools</em> tab, you’ll see a list of available products. Find PyCharm Community there and click <em>Install</em>:</p>
1201 <p><a href="https://files.realpython.com/media/pycharm-toolbox-installed-pycharm.cdcf1b52bc02.png" target="_blank"><img class="img-fluid mx-auto d-block border w-33" src="https://files.realpython.com/media/pycharm-toolbox-installed-pycharm.cdcf1b52bc02.png" width="337" height="537" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-toolbox-installed-pycharm.cdcf1b52bc02.png&w=84&sig=5f1e571c6c7bed958efddaec87d6ac5168713217 84w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-toolbox-installed-pycharm.cdcf1b52bc02.png&w=168&sig=0a76111939657ae01eaa8203b24c0c2e4fff5ee6 168w, https://files.realpython.com/media/pycharm-toolbox-installed-pycharm.cdcf1b52bc02.png 337w" sizes="75vw" alt="PyCharm installed with the Toolbox app"/></a></p>
1202 <p>VoilΓ ! You have PyCharm available on your machine. If you don’t want to use the Toolbox app, then you can also do a <a href="https://www.jetbrains.com/help/pycharm/installation-guide.html#standalone">stand-alone installation of PyCharm</a>.</p>
1203 <p>Launch PyCharm, and you’ll see the import settings popup:</p>
1204 <p><a href="https://files.realpython.com/media/pycharm-import-settings-popup.4e360260c697.png" target="_blank"><img class="img-fluid mx-auto d-block w-50" src="https://files.realpython.com/media/pycharm-import-settings-popup.4e360260c697.png" width="416" height="156" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-import-settings-popup.4e360260c697.png&w=104&sig=4920753cc035f162c505253937453e1aa7cc4d26 104w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-import-settings-popup.4e360260c697.png&w=208&sig=b65b1226bcc172a811d7cd1e00cd600408fba092 208w, https://files.realpython.com/media/pycharm-import-settings-popup.4e360260c697.png 416w" sizes="75vw" alt="PyCharm Import Settings Popup"/></a></p>
1205 <p>PyCharm will automatically detect that this is a fresh install and choose <em>Do not import settings</em> for you. Click <em>OK</em>, and PyCharm will ask you to select a keymap scheme. Leave the default and click <em>Next: UI Themes</em> on the bottom right:</p>
1206 <p><a href="https://files.realpython.com/media/pycharm-keymap-scheme.c8115fda9bdd.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/pycharm-keymap-scheme.c8115fda9bdd.png" width="805" height="666" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-keymap-scheme.c8115fda9bdd.png&w=201&sig=644595a94c07780a552f76abbfc5fe526b3c9459 201w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-keymap-scheme.c8115fda9bdd.png&w=402&sig=64c348872c1519bc4148e3dbbac2550ed6c0fa30 402w, https://files.realpython.com/media/pycharm-keymap-scheme.c8115fda9bdd.png 805w" sizes="75vw" alt="PyCharm Keymap Scheme"/></a></p>
1207 <p>PyCharm will then ask you to choose a dark theme called Darcula or a light theme. Choose whichever you prefer and click <em>Next: Launcher Script</em>: </p>
1208 <p><a href="https://files.realpython.com/media/pycharm-set-ui-theme.c48aac8e3fe0.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/pycharm-set-ui-theme.c48aac8e3fe0.png" width="803" height="666" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-set-ui-theme.c48aac8e3fe0.png&w=200&sig=6998b85afd9e2ca1503624ba55b904f4051f1ffe 200w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-set-ui-theme.c48aac8e3fe0.png&w=401&sig=a6480f0b0073f0680068828fc2f0f5c8ee55cbdb 401w, https://files.realpython.com/media/pycharm-set-ui-theme.c48aac8e3fe0.png 803w" sizes="75vw" alt="PyCharm Set UI Theme Page"/></a></p>
1209 <p>I’ll be using the dark theme Darcula throughout this tutorial. You can find and install other themes as <a href="#using-plugins-and-external-tools-in-pycharm">plugins</a>, or you can also <a href="https://blog.codota.com/5-best-intellij-themes/">import them</a>.</p>
1210 <p>On the next page, leave the defaults and click <em>Next: Featured plugins</em>. There, PyCharm will show you a list of plugins you may want to install because most users like to use them. Click <em>Start using PyCharm</em>, and now you are ready to write some code!</p>
1211 <h2 id="writing-code-in-pycharm">Writing Code in PyCharm</h2>
1212 <p>In PyCharm, you do everything in the context of a <strong>project</strong>. Thus, the first thing you need to do is create one.</p>
1213 <p>After installing and opening PyCharm, you are on the welcome screen. Click <em>Create New Project</em>, and you’ll see the <em>New Project</em> popup:</p>
1214 <p><a href="https://files.realpython.com/media/pycharm-new-project.cc35f3aa1056.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/pycharm-new-project.cc35f3aa1056.png" width="664" height="480" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-new-project.cc35f3aa1056.png&w=166&sig=6423b68127eae8ca93165323df4884844265f5e3 166w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-new-project.cc35f3aa1056.png&w=332&sig=50d660be9a42904b1161bd76df1ab9ddd77e2132 332w, https://files.realpython.com/media/pycharm-new-project.cc35f3aa1056.png 664w" sizes="75vw" alt="New Project in PyCharm"/></a></p>
1215 <p>Specify the project location and expand the <em>Project Interpreter</em> drop down. Here, you have options to create a new project interpreter or reuse an existing one. Choose <em>New environment using</em>. Right next to it, you have a drop down list to select one of <em>Virtualenv</em>, <em>Pipenv</em>, or <em>Conda</em>, which are the tools that help to keep dependencies required by different projects separate by creating isolated Python environments for them. </p>
1216 <p>You are free to select whichever you like, but <em>Virtualenv</em> is used for this tutorial. If you choose to, you can specify the environment location and choose the base interpreter from the list, which is a list of Python interpreters (such as Python2.7 and Python3.6) installed on your system. Usually, the defaults are fine. Then you have to select boxes to inherit global site-packages to your new environment and make it available to all other projects. Leave them unselected. </p>
1217 <p>Click <em>Create</em> on the bottom right and you will see the new project created:</p>
1218 <p><a href="https://files.realpython.com/media/pycharm-project-created.99dffd1d4e9a.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/pycharm-project-created.99dffd1d4e9a.png" width="1174" height="734" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-project-created.99dffd1d4e9a.png&w=293&sig=d6394f174acab8ee63eb6ce0360d0174857f7afb 293w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-project-created.99dffd1d4e9a.png&w=587&sig=2fd8ea44cc0ad15f015aab687018ec7ad8861a53 587w, https://files.realpython.com/media/pycharm-project-created.99dffd1d4e9a.png 1174w" sizes="75vw" alt="Project created in PyCharm"/></a></p>
1219 <p>You will also see a small <em>Tip of the Day</em> popup where PyCharm gives you one trick to learn at each startup. Go ahead and close this popup.</p>
1220 <p>It is now time to start a new Python program. Type <span class="keys"><kbd class="key-command">Cmd</kbd><span>+</span><kbd class="key-n">N</kbd></span> if you are on Mac or <span class="keys"><kbd class="key-alt">Alt</kbd><span>+</span><kbd class="key-insert">Ins</kbd></span> if you are on Windows or Linux. Then, choose <em>Python File</em>. You can also select <em>File β New</em> from the menu. Name the new file <code>guess_game.py</code> and click <em>OK</em>. You will see a PyCharm window similar to the following:</p>
1221 <p><a href="https://files.realpython.com/media/pycharm-new-file.7ea9902d73ea.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/pycharm-new-file.7ea9902d73ea.png" width="1172" height="734" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-new-file.7ea9902d73ea.png&w=293&sig=b1ee432e97d642aea67818cc7280971247196a62 293w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-new-file.7ea9902d73ea.png&w=586&sig=3563b00140a3edd3f1df0e522d5069f6efcb62d3 586w, https://files.realpython.com/media/pycharm-new-file.7ea9902d73ea.png 1172w" sizes="75vw" alt="PyCharm New File"/></a></p>
1222 <p>For our test code, let’s quickly code up a simple guessing game in which the program chooses a number that the user has to guess. For every guess, the program will tell if the user’s guess was smaller or bigger than the secret number. The game ends when the user guesses the number. Here’s the code for the game:</p>
1223 <div class="highlight python"><pre><span></span><span class="lineno"> 1 </span><span class="kn">from</span> <span class="nn">random</span> <span class="k">import</span> <span class="n">randint</span>
1224 <span class="lineno"> 2 </span>
1225 <span class="lineno"> 3 </span><span class="k">def</span> <span class="nf">play</span><span class="p">():</span>
1226 <span class="lineno"> 4 </span> <span class="n">random_int</span> <span class="o">=</span> <span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">100</span><span class="p">)</span>
1227 <span class="lineno"> 5 </span>
1228 <span class="lineno"> 6 </span> <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
1229 <span class="lineno"> 7 </span> <span class="n">user_guess</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="nb">input</span><span class="p">(</span><span class="s2">"What number did we guess (0-100)?"</span><span class="p">))</span>
1230 <span class="lineno"> 8 </span>
1231 <span class="lineno"> 9 </span> <span class="k">if</span> <span class="n">user_guess</span> <span class="o">==</span> <span class="n">randint</span><span class="p">:</span>
1232 <span class="lineno">10 </span> <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s2">"You found the number (</span><span class="si">{random_int}</span><span class="s2">). Congrats!"</span><span class="p">)</span>
1233 <span class="lineno">11 </span> <span class="k">break</span>
1234 <span class="lineno">12 </span>
1235 <span class="lineno">13 </span> <span class="k">if</span> <span class="n">user_guess</span> <span class="o"><</span> <span class="n">random_int</span><span class="p">:</span>
1236 <span class="lineno">14 </span> <span class="nb">print</span><span class="p">(</span><span class="s2">"Your number is less than the number we guessed."</span><span class="p">)</span>
1237 <span class="lineno">15 </span> <span class="k">continue</span>
1238 <span class="lineno">16 </span>
1239 <span class="lineno">17 </span> <span class="k">if</span> <span class="n">user_guess</span> <span class="o">></span> <span class="n">random_int</span><span class="p">:</span>
1240 <span class="lineno">18 </span> <span class="nb">print</span><span class="p">(</span><span class="s2">"Your number is more than the number we guessed."</span><span class="p">)</span>
1241 <span class="lineno">19 </span> <span class="k">continue</span>
1242 <span class="lineno">20 </span>
1243 <span class="lineno">21 </span>
1244 <span class="lineno">22 </span><span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
1245 <span class="lineno">23 </span> <span class="n">play</span><span class="p">()</span>
1246 </pre></div>
1247
1248 <p>Type this code directly rather than copying and pasting. You’ll see something like this:</p>
1249 <p><a href="https://files.realpython.com/media/typing-guess-game.fcaedeb8ece2.gif" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/typing-guess-game.fcaedeb8ece2.gif" width="528" height="480" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/typing-guess-game.fcaedeb8ece2.gif&w=132&sig=7e5eb20fb9ae97b1cea80380f9ad00f35dd76707 132w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/typing-guess-game.fcaedeb8ece2.gif&w=264&sig=eb242bca301203a38741a986d7847cf0f3ef4cff 264w, https://files.realpython.com/media/typing-guess-game.fcaedeb8ece2.gif 528w" sizes="75vw" alt="Typing Guessing Game"/></a></p>
1250 <p>As you can see, PyCharm provides <a href="https://www.jetbrains.com/pycharm/features/coding_assistance.html">Intelligent Coding Assistance</a> with code completion, code inspections, on-the-fly error highlighting, and quick-fix suggestions. In particular, note how when you typed <code>main</code> and then hit tab, PyCharm auto-completed the whole <code>main</code> clause for you. </p>
1251 <p>Also note how, if you forget to type <code>if</code> before the condition, append <code>.if</code>, and then hit <span class="keys"><kbd class="key-tab">Tab</kbd></span>, PyCharm fixes the <code>if</code> clause for you. The same is true with <code>True.while</code>. That’s <a href="https://www.jetbrains.com/help/pycharm/settings-postfix-completion.html">PyCharm’s Postfix completions</a> working for you to help reduce backward caret jumps.</p>
1252 <h2 id="running-code-in-pycharm">Running Code in PyCharm</h2>
1253 <p>Now that you’ve coded up the game, it’s time for you to run it.</p>
1254 <p>You have three ways of running this program:</p>
1255 <ol>
1256 <li>Use the shortcut <span class="keys"><kbd class="key-control">Ctrl</kbd><span>+</span><kbd class="key-shift">Shift</kbd><span>+</span><kbd class="key-r">R</kbd></span> on Mac or <span class="keys"><kbd class="key-control">Ctrl</kbd><span>+</span><kbd class="key-shift">Shift</kbd><span>+</span><kbd class="key-f10">F10</kbd></span> on Windows or Linux.</li>
1257 <li>Right-click the background and choose <em>Run ‘guess_game’</em> from the menu.</li>
1258 <li>Since this program has the <code>__main__</code> clause, you can click on the little green arrow to the left of the <code>__main__</code> clause and choose <em>Run ‘guess_game’</em> from there.</li>
1259 </ol>
1260 <p>Use any one of the options above to run the program, and you’ll see the Run Tool pane appear at the bottom of the window, with your code output showing:</p>
1261 <p><a href="https://files.realpython.com/media/pycharm-running-script.33fb830f45b4.gif" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/pycharm-running-script.33fb830f45b4.gif" width="1068" height="720" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-running-script.33fb830f45b4.gif&w=267&sig=44be962297881f8ae66557c19905a55202ee14de 267w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-running-script.33fb830f45b4.gif&w=534&sig=000552999cea7a9ce79eeec2f9db7361e9585d77 534w, https://files.realpython.com/media/pycharm-running-script.33fb830f45b4.gif 1068w" sizes="75vw" alt="Running a script in PyCharm"/></a></p>
1262 <p>Play the game for a little bit to see if you can find the number guessed. Pro tip: start with 50. </p>
1263 <h2 id="debugging-in-pycharm">Debugging in PyCharm</h2>
1264 <p>Did you find the number? If so, you may have seen something weird after you found the number. Instead of printing the congratulations message and exiting, the program seems to start over. That’s a bug right there. To discover why the program starts over, you’ll now debug the program.</p>
1265 <p>First, place a breakpoint by clicking on the blank space to the left of line number 8:</p>
1266 <p><a href="https://files.realpython.com/media/pycharm-debug-breakpoint.55cf93c49859.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/pycharm-debug-breakpoint.55cf93c49859.png" width="1042" height="710" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-debug-breakpoint.55cf93c49859.png&w=260&sig=e714eeae34fad6c0e5889bee0f236f9c30e100a0 260w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-debug-breakpoint.55cf93c49859.png&w=521&sig=7c4692a273d49a627785a715fd0a08d4c8120649 521w, https://files.realpython.com/media/pycharm-debug-breakpoint.55cf93c49859.png 1042w" sizes="75vw" alt="Debug breakpoint in PyCharm"/></a></p>
1267 <p>This will be the point where the program will be suspended, and you can start exploring what went wrong from there on. Next, choose one of the following three ways to start debugging:</p>
1268 <ol>
1269 <li>Press <span class="keys"><kbd class="key-control">Ctrl</kbd><span>+</span><kbd class="key-shift">Shift</kbd><span>+</span><kbd class="key-d">D</kbd></span> on Mac or <span class="keys"><kbd class="key-shift">Shift</kbd><span>+</span><kbd class="key-alt">Alt</kbd><span>+</span><kbd class="key-f9">F9</kbd></span> on Windows or Linux.</li>
1270 <li>Right-click the background and choose <em>Debug ‘guess_game’</em>.</li>
1271 <li>Click on the little green arrow to the left of the <code>__main__</code> clause and choose <em>Debug ‘guess_game</em> from there.</li>
1272 </ol>
1273 <p>Afterwards, you’ll see a <em>Debug</em> window open at the bottom:</p>
1274 <p><a href="https://files.realpython.com/media/pycharm-debugging-start.04246b743469.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/pycharm-debugging-start.04246b743469.png" width="1043" height="711" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-debugging-start.04246b743469.png&w=260&sig=cea78f8df9a7f183330e3610c90a2abeab879923 260w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-debugging-start.04246b743469.png&w=521&sig=6ae0fe4515cf7f0d52cbcda986eb9c52d0a602ee 521w, https://files.realpython.com/media/pycharm-debugging-start.04246b743469.png 1043w" sizes="75vw" alt="Start of debugging in PyCharm"/></a></p>
1275 <p>Follow the steps below to debug the program:</p>
1276 <ol>
1277 <li>
1278 <p>Notice that the current line is highlighted in blue.</p>
1279 </li>
1280 <li>
1281 <p>See that <code>random_int</code> and its value are listed in the Debug window. Make a note of this number. (In the picture, the number is 85.)</p>
1282 </li>
1283 <li>
1284 <p>Hit <span class="keys"><kbd class="key-f8">F8</kbd></span> to execute the current line and step <em>over</em> to the next one. You can also use <span class="keys"><kbd class="key-f7">F7</kbd></span> to step <em>into</em> the function in the current line, if necessary. As you continue executing the statements, the changes in the variables will be automatically reflected in the Debugger window.</p>
1285 </li>
1286 <li>
1287 <p>Notice that there is the Console tab right next to the Debugger tab that opened. This Console tab and the Debugger tab are mutually exclusive. In the Console tab, you will be interacting with your program, and in the Debugger tab you will do the debugging actions.</p>
1288 </li>
1289 <li>
1290 <p>Switch to the Console tab to enter your guess.</p>
1291 </li>
1292 <li>
1293 <p>Type the number shown, and then hit <span class="keys"><kbd class="key-enter">Enter</kbd></span>.</p>
1294 </li>
1295 <li>
1296 <p>Switch back to the Debugger tab.</p>
1297 </li>
1298 <li>
1299 <p>Hit <span class="keys"><kbd class="key-f8">F8</kbd></span> again to evaluate the <code>if</code> statement. Notice that you are now on line 14. But wait a minute! Why didn’t it go to the line 11? The reason is that the <code>if</code> statement on line 10 evaluated to <code>False</code>. But why did it evaluate to <code>False</code> when you entered the number that was chosen?</p>
1300 </li>
1301 <li>
1302 <p>Look carefully at line 10 and notice that we are comparing <code>user_guess</code> with the wrong thing. Instead of comparing it with <code>random_int</code>, we are comparing it with <code>randint</code>, the function that was imported from the <code>random</code> package.</p>
1303 </li>
1304 <li>
1305 <p>Change it to <code>random_int</code>, restart the debugging, and follow the same steps again. You will see that, this time, it will go to line 11, and line 10 will evaluate to <code>True</code>:</p>
1306 </li>
1307 </ol>
1308 <p><a href="https://files.realpython.com/media/pycharm-debugging-scripts.bb5a077da438.gif" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/pycharm-debugging-scripts.bb5a077da438.gif" width="1092" height="720" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-debugging-scripts.bb5a077da438.gif&w=273&sig=fc5de269fbc13ea5c1d8be4ca7f525e04a4bb68c 273w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-debugging-scripts.bb5a077da438.gif&w=546&sig=4719f69fd6e78ed3e68f16a9355d00d0edd85a26 546w, https://files.realpython.com/media/pycharm-debugging-scripts.bb5a077da438.gif 1092w" sizes="75vw" alt="Debugging Script in PyCharm"/></a></p>
1309 <p>Congratulations! You fixed the bug.</p>
1310 <h2 id="testing-in-pycharm">Testing in PyCharm</h2>
1311 <p>No application is reliable without unit tests. PyCharm helps you write and run them very quickly and comfortably. By default, <a href="https://docs.python.org/3/library/unittest.html"><code>unittest</code></a> is used as the test runner, but PyCharm also supports other testing frameworks such as <a href="http://www.pytest.org/en/latest/"><code>pytest</code></a>, <a href="https://nose.readthedocs.io/en/latest/"><code>nose</code></a>, <a href="https://docs.python.org/3/library/doctest.html"><code>doctest</code></a>, <a href="https://www.jetbrains.com/help/pycharm/tox-support.html"><code>tox</code></a>, and <a href="https://twistedmatrix.com/trac/wiki/TwistedTrial"><code>trial</code></a>. You can, for example, enable <code>pytest</code> for your project like this:</p>
1312 <ol>
1313 <li>Open the <em>Settings/Preferences β Tools β Python Integrated Tools</em> settings dialog.</li>
1314 <li>Select <code>pytest</code> in the Default test runner field.</li>
1315 <li>Click <em>OK</em> to save the settings. </li>
1316 </ol>
1317 <p>For this example, we’ll be using the default test runner <code>unittest</code>. </p>
1318 <p>In the same project, create a file called <code>calculator.py</code> and put the following <code>Calculator</code> class in it:</p>
1319 <div class="highlight python"><pre><span></span><span class="lineno"> 1 </span><span class="k">class</span> <span class="nc">Calculator</span><span class="p">:</span>
1320 <span class="lineno"> 2 </span> <span class="k">def</span> <span class="nf">add</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">):</span>
1321 <span class="lineno"> 3 </span> <span class="k">return</span> <span class="n">a</span> <span class="o">+</span> <span class="n">b</span>
1322 <span class="lineno"> 4 </span>
1323 <span class="lineno"> 5 </span> <span class="k">def</span> <span class="nf">multiply</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">):</span>
1324 <span class="lineno"> 6 </span> <span class="k">return</span> <span class="n">a</span> <span class="o">*</span> <span class="n">b</span>
1325 </pre></div>
1326
1327 <p>PyCharm makes it very easy to create tests for your existing code. With the <code>calculator.py</code> file open, execute any one of the following that you like:</p>
1328 <ul>
1329 <li>Press <span class="keys"><kbd class="key-shift">Shift</kbd><span>+</span><kbd class="key-command">Cmd</kbd><span>+</span><kbd class="key-t">T</kbd></span> on Mac or <span class="keys"><kbd class="key-control">Ctrl</kbd><span>+</span><kbd class="key-shift">Shift</kbd><span>+</span><kbd class="key-t">T</kbd></span> on Windows or Linux.</li>
1330 <li>Right-click in the background of the class and then choose <em>Go To</em> and <em>Test</em>.</li>
1331 <li>On the main menu, choose <em>Navigate β Test</em>.</li>
1332 </ul>
1333 <p>Choose <em>Create New Test…</em>, and you will see the following window:</p>
1334 <p><a href="https://files.realpython.com/media/pycharm-create-tests.9a6cea78f9c6.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/pycharm-create-tests.9a6cea78f9c6.png" width="500" height="402" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-create-tests.9a6cea78f9c6.png&w=125&sig=0c50b83f35578fd8004dce9e7d55fcd3b09a1967 125w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-create-tests.9a6cea78f9c6.png&w=250&sig=f6740142e71d2d2f6196ffbe341ae09bc9a64453 250w, https://files.realpython.com/media/pycharm-create-tests.9a6cea78f9c6.png 500w" sizes="75vw" alt="Create tests in PyCharm"/></a></p>
1335 <p>Leave the defaults of <em>Target directory</em>, <em>Test file name</em>, and <em>Test class name</em>. Select both of the methods and click <em>OK</em>. Voila! PyCharm automatically created a file called <code>test_calculator.py</code> and created the following stub tests for you in it:</p>
1336 <div class="highlight python"><pre><span></span><span class="lineno"> 1 </span><span class="kn">from</span> <span class="nn">unittest</span> <span class="k">import</span> <span class="n">TestCase</span>
1337 <span class="lineno"> 2 </span>
1338 <span class="lineno"> 3 </span><span class="k">class</span> <span class="nc">TestCalculator</span><span class="p">(</span><span class="n">TestCase</span><span class="p">):</span>
1339 <span class="lineno"> 4 </span> <span class="k">def</span> <span class="nf">test_add</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
1340 <span class="lineno"> 5 </span> <span class="bp">self</span><span class="o">.</span><span class="n">fail</span><span class="p">()</span>
1341 <span class="lineno"> 6 </span>
1342 <span class="lineno"> 7 </span> <span class="k">def</span> <span class="nf">test_multiply</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
1343 <span class="lineno"> 8 </span> <span class="bp">self</span><span class="o">.</span><span class="n">fail</span><span class="p">()</span>
1344 </pre></div>
1345
1346 <p>Run the tests using one of the methods below:</p>
1347 <ul>
1348 <li>Press <span class="keys"><kbd class="key-control">Ctrl</kbd><span>+</span><kbd class="key-r">R</kbd></span> on Mac or <span class="keys"><kbd class="key-shift">Shift</kbd><span>+</span><kbd class="key-f10">F10</kbd></span> on Windows or Linux.</li>
1349 <li>Right-click the background and choose <em>Run ‘Unittests for test_calculator.py’</em>.</li>
1350 <li>Click on the little green arrow to the left of the test class name and choose <em>Run ‘Unittests for test_calculator.py’</em>.</li>
1351 </ul>
1352 <p>You’ll see the tests window open on the bottom with all the tests failing:</p>
1353 <p><a href="https://files.realpython.com/media/pycharm-failed-tests.810aa9c365cb.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/pycharm-failed-tests.810aa9c365cb.png" width="972" height="645" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-failed-tests.810aa9c365cb.png&w=243&sig=cb7ef285c20ed83b9771a91cc38d77342e4d3745 243w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-failed-tests.810aa9c365cb.png&w=486&sig=aaec55471da2d168048783b9c4e573f5ce876de4 486w, https://files.realpython.com/media/pycharm-failed-tests.810aa9c365cb.png 972w" sizes="75vw" alt="Failed tests in PyCharm"/></a></p>
1354 <p>Notice that you have the hierarchy of the test results on the left and the output of the terminal on the right. </p>
1355 <p>Now, implement <code>test_add</code> by changing the code to the following:</p>
1356 <div class="highlight python"><pre><span></span><span class="lineno"> 1 </span><span class="kn">from</span> <span class="nn">unittest</span> <span class="k">import</span> <span class="n">TestCase</span>
1357 <span class="lineno"> 2 </span>
1358 <span class="lineno"> 3 </span><span class="kn">from</span> <span class="nn">calculator</span> <span class="k">import</span> <span class="n">Calculator</span>
1359 <span class="lineno"> 4 </span>
1360 <span class="lineno"> 5 </span><span class="k">class</span> <span class="nc">TestCalculator</span><span class="p">(</span><span class="n">TestCase</span><span class="p">):</span>
1361 <span class="lineno"> 6 </span> <span class="k">def</span> <span class="nf">test_add</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
1362 <span class="lineno"> 7 </span> <span class="bp">self</span><span class="o">.</span><span class="n">calculator</span> <span class="o">=</span> <span class="n">Calculator</span><span class="p">()</span>
1363 <span class="lineno"> 8 </span> <span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">calculator</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">),</span> <span class="mi">7</span><span class="p">)</span>
1364 <span class="lineno"> 9 </span>
1365 <span class="lineno">10 </span> <span class="k">def</span> <span class="nf">test_multiply</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
1366 <span class="lineno">11 </span> <span class="bp">self</span><span class="o">.</span><span class="n">fail</span><span class="p">()</span>
1367 </pre></div>
1368
1369 <p>Run the tests again, and you’ll see that one test passed and the other failed. Explore the options to show passed tests, to show ignored tests, to sort tests alphabetically, and to sort tests by duration:</p>
1370 <p><a href="https://files.realpython.com/media/pycharm-running-tests.6077562207ba.gif" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/pycharm-running-tests.6077562207ba.gif" width="1092" height="720" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-running-tests.6077562207ba.gif&w=273&sig=e2238425e1cfb9a9a298741244f3021f3984dbf8 273w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-running-tests.6077562207ba.gif&w=546&sig=85203afd5ff2189fdfadad0960da514b03063953 546w, https://files.realpython.com/media/pycharm-running-tests.6077562207ba.gif 1092w" sizes="75vw" alt="Running tests in PyCharm"/></a></p>
1371 <p>Note that the <code>sleep(0.1)</code> method that you see in the GIF above is intentionally used to make one of the tests slower so that sorting by duration works. </p>
1372 <h2 id="editing-an-existing-project-in-pycharm">Editing an Existing Project in PyCharm</h2>
1373 <p>These single file projects are great for examples, but you’ll often work on much larger projects over a longer period of time. In this section, you’ll take a look at how PyCharm works with a larger project. </p>
1374 <p>To explore the project-focused features of PyCharm, you’ll use the Alcazar web framework that was built for learning purposes. To continue following along, clone the repo locally:</p>
1375 <div class="alert alert-warning" role="alert"><p><strong>Clone Repo:</strong> <a href="https://realpython.com/optins/view/alcazar-web-framework/" class="alert-link" data-toggle="modal" data-target="#modal-alcazar-web-framework" data-focus="false">Click here to clone the repo you'll use</a> to explore the project-focused features of PyCharm in this tutorial.</p></div>
1376
1377 <p>Once you have a project locally, open it in PyCharm using one of the following methods:</p>
1378 <ul>
1379 <li>Click <em>File β Open</em> on the main menu.</li>
1380 <li>Click <em>Open</em> on the <a href="https://www.jetbrains.com/help/pycharm/welcome-screen.html">Welcome Screen</a> if you are there.</li>
1381 </ul>
1382 <p>After either of these steps, find the folder containing the project on your computer and open it.</p>
1383 <p>If this project contains a <a href="https://realpython.com/python-virtual-environments-a-primer/">virtual environment</a>, then PyCharm will automatically use this virtual environment and make it the project interpreter.</p>
1384 <p>If you need to configure a different <code>virtualenv</code>, then open <em>Preferences</em> on Mac by pressing <span class="keys"><kbd class="key-command">Cmd</kbd><span>+</span><kbd class="key-comma">,</kbd></span> or <em>Settings</em> on Windows or Linux by pressing <span class="keys"><kbd class="key-control">Ctrl</kbd><span>+</span><kbd class="key-alt">Alt</kbd><span>+</span><kbd class="key-s">S</kbd></span> and find the <em>Project: ProjectName</em> section. Open the drop-down and choose <em>Project Interpreter</em>:</p>
1385 <p><a href="https://files.realpython.com/media/pycharm-project-interpreter.57282306555a.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/pycharm-project-interpreter.57282306555a.png" width="1083" height="723" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-project-interpreter.57282306555a.png&w=270&sig=286643bc473f648bbcce27338c980eb023746ac2 270w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-project-interpreter.57282306555a.png&w=541&sig=6d33c5adf0380e8c5b01ae9430436983580b4b49 541w, https://files.realpython.com/media/pycharm-project-interpreter.57282306555a.png 1083w" sizes="75vw" alt="Project interpreter in PyCharm"/></a></p>
1386 <p>Choose the <code>virtualenv</code> from the drop-down list. If it’s not there, then click on the settings button to the right of the drop-down list and then choose <em>Add…</em>. The rest of the steps should be the same as when we were <a href="#writing-code-in-pycharm">creating a new project</a>.</p>
1387 <h2 id="searching-and-navigating-in-pycharm">Searching and Navigating in PyCharm</h2>
1388 <p>In a big project where it’s difficult for a single person to remember where everything is located, it’s very important to be able to quickly navigate and find what you looking for. PyCharm has you covered here as well. Use the project you opened in the section above to practice these shortcuts: </p>
1389 <ul>
1390 <li><strong>Searching for a fragment in the current file:</strong> Press <span class="keys"><kbd class="key-command">Cmd</kbd><span>+</span><kbd class="key-f">F</kbd></span> on Mac or <span class="keys"><kbd class="key-control">Ctrl</kbd><span>+</span><kbd class="key-f">F</kbd></span> on Windows or Linux.</li>
1391 <li><strong>Searching for a fragment in the entire project:</strong> Press <span class="keys"><kbd class="key-command">Cmd</kbd><span>+</span><kbd class="key-shift">Shift</kbd><span>+</span><kbd class="key-f">F</kbd></span> on Mac or <span class="keys"><kbd class="key-control">Ctrl</kbd><span>+</span><kbd class="key-shift">Shift</kbd><span>+</span><kbd class="key-f">F</kbd></span> on Windows or Linux.</li>
1392 <li><strong>Searching for a class:</strong> Press <span class="keys"><kbd class="key-command">Cmd</kbd><span>+</span><kbd class="key-o">O</kbd></span> on Mac or <span class="keys"><kbd class="key-control">Ctrl</kbd><span>+</span><kbd class="key-n">N</kbd></span> on Windows or Linux.</li>
1393 <li><strong>Searching for a file:</strong> Press <span class="keys"><kbd class="key-command">Cmd</kbd><span>+</span><kbd class="key-shift">Shift</kbd><span>+</span><kbd class="key-o">O</kbd></span> on Mac or <span class="keys"><kbd class="key-control">Ctrl</kbd><span>+</span><kbd class="key-shift">Shift</kbd><span>+</span><kbd class="key-n">N</kbd></span> on Windows or Linux.</li>
1394 <li><strong>Searching all if you don’t know whether it’s a file, class, or a code fragment that you are looking for:</strong> Press <span class="keys"><kbd class="key-shift">Shift</kbd></span> twice.</li>
1395 </ul>
1396 <p>As for the navigation, the following shortcuts may save you a lot of time:</p>
1397 <ul>
1398 <li><strong>Going to the declaration of a variable:</strong> Press <span class="keys"><kbd class="key-command">Cmd</kbd></span> on Mac or <span class="keys"><kbd class="key-control">Ctrl</kbd></span> on Windows or Linux, and click on the variable.</li>
1399 <li><strong>Finding usages of a class, a method, or any symbol:</strong> Press <span class="keys"><kbd class="key-alt">Alt</kbd><span>+</span><kbd class="key-f7">F7</kbd></span>.</li>
1400 <li><strong>Seeing your recent changes:</strong> Press <span class="keys"><kbd class="key-shift">Shift</kbd><span>+</span><kbd class="key-alt">Alt</kbd><span>+</span><kbd class="key-c">C</kbd></span> or go to <em>View β Recent Changes</em> on the main menu.</li>
1401 <li><strong>Seeing your recent files:</strong> Press <span class="keys"><kbd class="key-command">Cmd</kbd><span>+</span><kbd class="key-e">E</kbd></span> on Mac or <span class="keys"><kbd class="key-control">Ctrl</kbd><span>+</span><kbd class="key-e">E</kbd></span> on Windows or Linux, or go to <em>View β Recent Files</em> on the main menu.</li>
1402 <li><strong>Going backward and forward through your history of navigation after you jumped around:</strong> Press <span class="keys"><kbd class="key-command">Cmd</kbd><span>+</span><kbd class="key-bracket-left">[</kbd></span> / <span class="keys"><kbd class="key-command">Cmd</kbd><span>+</span><kbd class="key-bracket-right">]</kbd></span> on Mac or <span class="keys"><kbd class="key-control">Ctrl</kbd><span>+</span><kbd class="key-alt">Alt</kbd><span>+</span><kbd class="key-arrow-left">Left</kbd></span> / <span class="keys"><kbd class="key-control">Ctrl</kbd><span>+</span><kbd class="key-alt">Alt</kbd><span>+</span><kbd class="key-arrow-right">Right</kbd></span> on Windows or Linux.</li>
1403 </ul>
1404 <p>For more details, see the <a href="https://www.jetbrains.com/help/pycharm/tutorial-exploring-navigation-and-search.html">official documentation</a>. </p>
1405 <h2 id="using-version-control-in-pycharm">Using Version Control in PyCharm</h2>
1406 <p>Version control systems such as <a href="https://git-scm.com/">Git</a> and <a href="https://www.mercurial-scm.org/">Mercurial</a> are some of the most important tools in the modern software development world. So, it is essential for an IDE to support them. PyCharm does that very well by integrating with a lot of popular VC systems such as Git (and <a href="https://github.com/">Github</a>), Mercurial, <a href="https://www.perforce.com/solutions/version-control">Perforce</a> and, <a href="https://subversion.apache.org/">Subversion</a>.</p>
1407 <div class="alert alert-primary" role="alert">
1408 <p><strong>Note</strong>: <a href="https://realpython.com/python-git-github-intro/">Git</a> is used for the following examples.</p>
1409 </div>
1410 <h3 id="configuring-vcs">Configuring VCS</h3>
1411 <p>To enable VCS integration. Go to <em>VCS β VCS Operations Popup…</em> from the menu on the top or press <span class="keys"><kbd class="key-control">Ctrl</kbd><span>+</span><kbd class="key-v">V</kbd></span> on Mac or <span class="keys"><kbd class="key-alt">Alt</kbd><span>+</span><kbd class="key-grave">`</kbd></span> on Windows or Linux. Choose <em>Enable Version Control Integration…</em>. You’ll see the following window open:</p>
1412 <p><a href="https://files.realpython.com/media/pycharm-enable-vc-integration.b30ec94c1246.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/pycharm-enable-vc-integration.b30ec94c1246.png" width="715" height="147" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-enable-vc-integration.b30ec94c1246.png&w=178&sig=7ef55e4ed6068c86d831adaefc1af11f4c083763 178w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-enable-vc-integration.b30ec94c1246.png&w=357&sig=c701d809b49e9837477c14e8dfe34dc4cfa66c33 357w, https://files.realpython.com/media/pycharm-enable-vc-integration.b30ec94c1246.png 715w" sizes="75vw" alt="Enable Version Control Integration in PyCharm"/></a></p>
1413 <p>Choose <em>Git</em> from the drop down list, click <em>OK</em>, and you have VCS enabled for your project. Note that if you opened an existing project that has version control enabled, then PyCharm will see that and automatically enable it.</p>
1414 <p>Now, if you go to the <em>VCS Operations Popup…</em>, you’ll see a different popup with the options to do <code>git add</code>, <code>git stash</code>, <code>git branch</code>, <code>git commit</code>, <code>git push</code> and more:</p>
1415 <p><a href="https://files.realpython.com/media/pycharm-vcs-operations.70dbafcb983a.png" target="_blank"><img class="img-fluid mx-auto d-block border w-50" src="https://files.realpython.com/media/pycharm-vcs-operations.70dbafcb983a.png" width="392" height="379" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-vcs-operations.70dbafcb983a.png&w=98&sig=f285b015c957936448441c4ec8b03cf8627cdffc 98w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-vcs-operations.70dbafcb983a.png&w=196&sig=a16c449a21dde2a2b0f2f7533c92e71574cd3d60 196w, https://files.realpython.com/media/pycharm-vcs-operations.70dbafcb983a.png 392w" sizes="75vw" alt="VCS operations in PyCharm"/></a></p>
1416 <p>If you can’t find what you need, you can most probably find it by going to <em>VCS</em> from the top menu and choosing <em>Git</em>, where you can even create and view pull requests.</p>
1417 <h3 id="committing-and-conflict-resolution">Committing and Conflict Resolution</h3>
1418 <p>These are two features of VCS integration in PyCharm that I personally use and enjoy a lot! Let’s say you have finished your work and want to commit it. Go to <em>VCS β VCS Operations Popup… β Commit…</em> or press <span class="keys"><kbd class="key-command">Cmd</kbd><span>+</span><kbd class="key-k">K</kbd></span> on Mac or <span class="keys"><kbd class="key-control">Ctrl</kbd><span>+</span><kbd class="key-k">K</kbd></span> on Windows or Linux. You’ll see the following window open:</p>
1419 <p><a href="https://files.realpython.com/media/pycharm-commit-window.a4ceff16c2d3.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/pycharm-commit-window.a4ceff16c2d3.png" width="929" height="682" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-commit-window.a4ceff16c2d3.png&w=232&sig=935dabf7a28cf757a5c87165e3da494540c3e4a6 232w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-commit-window.a4ceff16c2d3.png&w=464&sig=7a05fe8a5cbfb1c3524c24515f9a8d7fa3211591 464w, https://files.realpython.com/media/pycharm-commit-window.a4ceff16c2d3.png 929w" sizes="75vw" alt="Commit window in PyCharm"/></a></p>
1420 <p>In this window, you can do the following:</p>
1421 <ol>
1422 <li>Choose which files to commit</li>
1423 <li>Write your commit message</li>
1424 <li>Do all kinds of checks and cleanup <a href="https://www.jetbrains.com/help/idea/commit-changes-dialog.html#before_commit">before commit</a></li>
1425 <li>See the difference of changes</li>
1426 <li>Commit and push at once by pressing the arrow to the right of the <em>Commit</em> button on the right bottom and choosing <em>Commit and Push…</em></li>
1427 </ol>
1428 <p>It can feel magical and fast, especially if you’re used to doing everything manually on the command line.</p>
1429 <p>When you work in a team, <strong>merge conflicts</strong> do happen. When somebody commits changes to a file that you’re working on, but their changes overlap with yours because both of you changed the same lines, then VCS will not be able to figure out if it should choose your changes or those of your teammate. So you’ll get these unfortunate arrows and symbols:</p>
1430 <p><a href="https://files.realpython.com/media/pycharm-conflicts.74b23b9ec798.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/pycharm-conflicts.74b23b9ec798.png" width="996" height="691" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-conflicts.74b23b9ec798.png&w=249&sig=d02221be0ce12dbc9a8ea7514e047bb608b16c08 249w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-conflicts.74b23b9ec798.png&w=498&sig=1286c7aa0f62795e01c6a2f0607096b8c0d67b20 498w, https://files.realpython.com/media/pycharm-conflicts.74b23b9ec798.png 996w" sizes="75vw" alt="Conflicts in PyCharm"/></a></p>
1431 <p>This looks strange, and it’s difficult to figure out which changes should be deleted and which ones should stay. PyCharm to the rescue! It has a much nicer and cleaner way of resolving conflicts. Go to <em>VCS</em> in the top menu, choose <em>Git</em> and then <em>Resolve conflicts…</em>. Choose the file whose conflicts you want to resolve and click on <em>Merge</em>. You will see the following window open:</p>
1432 <p><a href="https://files.realpython.com/media/pycharm-conflict-resolving-window.eea8f79a12b2.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/pycharm-conflict-resolving-window.eea8f79a12b2.png" width="1174" height="709" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-conflict-resolving-window.eea8f79a12b2.png&w=293&sig=ef195e8acbb5ec9fa55fca46a43486995c2efca7 293w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-conflict-resolving-window.eea8f79a12b2.png&w=587&sig=067731ba579736c92c747eb128b3fb02b0d68417 587w, https://files.realpython.com/media/pycharm-conflict-resolving-window.eea8f79a12b2.png 1174w" sizes="75vw" alt="Conflict resolving windown in PyCharm"/></a></p>
1433 <p>On the left column, you will see your changes. On the right one, the changes made by your teammate. Finally, in the middle column, you will see the result. The conflicting lines are highlighted, and you can see a little <em>X</em> and <em>>></em>/<em><<</em> right beside those lines. Press the arrows to accept the changes and the <em>X</em> to decline. After you resolve all those conflicts, click the <em>Apply</em> button: </p>
1434 <p><a href="https://files.realpython.com/media/pycharm-resolving-conflicts.d3128ce78c45.gif" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/pycharm-resolving-conflicts.d3128ce78c45.gif" width="1200" height="720" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-resolving-conflicts.d3128ce78c45.gif&w=300&sig=099dcca659431f9d2a1315b1fea5d7cbe246425c 300w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-resolving-conflicts.d3128ce78c45.gif&w=600&sig=6b98751300d8750c93d80cf7b65e96fb57d81714 600w, https://files.realpython.com/media/pycharm-resolving-conflicts.d3128ce78c45.gif 1200w" sizes="75vw" alt="Resolving Conflicts in PyCharm"/></a></p>
1435 <p>In the GIF above, for the first conflicting line, the author declined his own changes and accepted those of his teammates. Conversely, the author accepted his own changes and declined his teammates’ for the second conflicting line.</p>
1436 <p>There’s a lot more that you can do with the VCS integration in PyCharm. For more details, see <a href="https://www.jetbrains.com/help/pycharm/version-control-integration.html">this documentation</a>.</p>
1437 <h2 id="using-plugins-and-external-tools-in-pycharm">Using Plugins and External Tools in PyCharm</h2>
1438 <p>You can find almost everything you need for development in PyCharm. If you can’t, there is most probably a <a href="https://plugins.jetbrains.com/">plugin</a> that adds that functionality you need to PyCharm. For example, they can:</p>
1439 <ul>
1440 <li>Add support for various languages and frameworks </li>
1441 <li>Boost your productivity with shortcut hints, file watchers, and so on </li>
1442 <li>Help you learn a new programming language with coding exercises</li>
1443 </ul>
1444 <p>For instance, <a href="https://plugins.jetbrains.com/plugin/164-ideavim">IdeaVim</a> adds Vim emulation to PyCharm. If you like Vim, this can be a pretty good combination. </p>
1445 <p><a href="https://plugins.jetbrains.com/plugin/8006-material-theme-ui">Material Theme UI</a> changes the appearance of PyCharm to a Material Design look and feel: </p>
1446 <p><a href="https://files.realpython.com/media/pycharm-material-theme.178175815adc.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/pycharm-material-theme.178175815adc.png" width="1110" height="743" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-material-theme.178175815adc.png&w=277&sig=60ec4c8b5f6a89af345a230518e21ee8a33d174b 277w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-material-theme.178175815adc.png&w=555&sig=fdac8603145ed77eb443abcbbe8ebdb9811bdff4 555w, https://files.realpython.com/media/pycharm-material-theme.178175815adc.png 1110w" sizes="75vw" alt="Material Theme in PyCharm"/></a></p>
1447 <p><a href="https://plugins.jetbrains.com/plugin/9442-vue-js">Vue.js</a> adds support for <a href="https://vuejs.org/">Vue.js</a> projects. <a href="https://plugins.jetbrains.com/plugin/7793-markdown">Markdown</a> provides the capability to edit Markdown files within the IDE and see the rendered HTML in a live preview. You can find and install all of the available plugins by going to the <em>Preferences β Plugins</em> on Mac or <em>Settings β Plugins</em> on Windows or Linux, under the <em>Marketplace</em> tab:</p>
1448 <p><a href="https://files.realpython.com/media/pycharm-plugin-marketplace.7d1cecfdc8b3.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/pycharm-plugin-marketplace.7d1cecfdc8b3.png" width="1047" height="687" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-plugin-marketplace.7d1cecfdc8b3.png&w=261&sig=8d4a9ba35b5eb27b5604f86108b79f28e40f3cc9 261w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-plugin-marketplace.7d1cecfdc8b3.png&w=523&sig=ec5e20ae96bf9ee37a1011bcc2203e7d1599b4a4 523w, https://files.realpython.com/media/pycharm-plugin-marketplace.7d1cecfdc8b3.png 1047w" sizes="75vw" alt="Plugin Marketplace in PyCharm"/></a></p>
1449 <p>If you can’t find what you need, you can even <a href="http://www.jetbrains.org/intellij/sdk/docs/basics.html">develop your own plugin</a>.</p>
1450 <p>If you can’t find the right plugin and don’t want to develop your own because there’s already a package in PyPI, then you can add it to PyCharm as an external tool. Take <a href="http://flake8.pycqa.org/en/latest/"><code>Flake8</code></a>, the code analyzer, as an example. </p>
1451 <p>First, install <code>flake8</code> in your virtualenv with <code>pip install flake8</code> in the Terminal app of your choice. You can also use the one integrated into PyCharm:</p>
1452 <p><a href="https://files.realpython.com/media/pycharm-terminal.bb20cae6697e.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/pycharm-terminal.bb20cae6697e.png" width="972" height="646" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-terminal.bb20cae6697e.png&w=243&sig=860217f31e60a4bb574e169ee05b6788cacaa388 243w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-terminal.bb20cae6697e.png&w=486&sig=26a30f328e42cc19e7e652c44767093480dbf352 486w, https://files.realpython.com/media/pycharm-terminal.bb20cae6697e.png 972w" sizes="75vw" alt="Terminal in PyCharm"/></a></p>
1453 <p>Then, go to <em>Preferences β Tools</em> on Mac or <em>Settings β Tools</em> on Windows/Linux, and then choose <em>External Tools</em>. Then click on the little <em>+</em> button at the bottom (1). In the new popup window, insert the details as shown below and click <em>OK</em> for both windows:</p>
1454 <p><a href="https://files.realpython.com/media/pycharm-flake8-tool.3963506224b4.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/pycharm-flake8-tool.3963506224b4.png" width="1082" height="720" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-flake8-tool.3963506224b4.png&w=270&sig=152f2ccf0a75a950b6a5cd6b5087507b66288595 270w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-flake8-tool.3963506224b4.png&w=541&sig=ee7221ee226076dfeaedfd5d2756263e351d8d14 541w, https://files.realpython.com/media/pycharm-flake8-tool.3963506224b4.png 1082w" sizes="75vw" alt="Flake8 tool in PyCharm"/></a></p>
1455 <p>Here, <em>Program</em> (2) refers to the Flake8 executable that can be found in the folder <em>/bin</em> of your virtual environment. <em>Arguments</em> (3) refers to which file you want to analyze with the help of Flake8. <em>Working directory</em> is the directory of your project.</p>
1456 <p>You could hardcode the absolute paths for everything here, but that would mean that you couldn’t use this external tool in other projects. You would be able to use it only inside one project for one file. </p>
1457 <p>So you need to use something called <em>Macros</em>. Macros are basically variables in the format of <code>$name$</code> that change according to your context. For example, <code>$FileName$</code> is <code>first.py</code> when you’re editing <code>first.py</code>, and it is <code>second.py</code> when you’re editing <code>second.py</code>. You can see their list and insert any of them by clicking on the <em>Insert Macro…</em> buttons. Because you used macros here, the values will change according to the project you’re currently working on, and Flake8 will continue to do its job properly. </p>
1458 <p>In order to use it, create a file <code>example.py</code> and put the following code in it:</p>
1459 <div class="highlight python"><pre><span></span><span class="lineno"> 1 </span><span class="n">CONSTANT_VAR</span> <span class="o">=</span> <span class="mi">1</span>
1460 <span class="lineno"> 2 </span>
1461 <span class="lineno"> 3 </span>
1462 <span class="lineno"> 4 </span>
1463 <span class="lineno"> 5 </span><span class="k">def</span> <span class="nf">add</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">):</span>
1464 <span class="lineno"> 6 </span> <span class="n">c</span> <span class="o">=</span> <span class="s2">"hello"</span>
1465 <span class="lineno"> 7 </span> <span class="k">return</span> <span class="n">a</span> <span class="o">+</span> <span class="n">b</span>
1466 </pre></div>
1467
1468 <p>It deliberately breaks some of the Flake8 rules. Right-click the background of this file. Choose <em>External Tools</em> and then <em>Flake8</em>. VoilΓ ! The output of the Flake8 analysis will appear at the bottom: </p>
1469 <p><a href="https://files.realpython.com/media/pycharm-flake8-output.5b78e911e6d3.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/pycharm-flake8-output.5b78e911e6d3.png" width="997" height="634" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-flake8-output.5b78e911e6d3.png&w=249&sig=8fecbaaf9d4e2daa2bbe443be4b6dee2634f2a46 249w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-flake8-output.5b78e911e6d3.png&w=498&sig=4e41c6dbac28fbe23c18716f49c3cadf63767500 498w, https://files.realpython.com/media/pycharm-flake8-output.5b78e911e6d3.png 997w" sizes="75vw" alt="Flake8 Output in PyCharm"/></a></p>
1470 <p>In order to make it even better, you can add a shortcut for it. Go to <em>Preferences</em> on Mac or to <em>Settings</em> on Windows or Linux. Then, go to <em>Keymap β External Tools β External Tools</em>. Double-click <em>Flake8</em> and choose <em>Add Keyboard Shortcut</em>. You’ll see this window:</p>
1471 <p><a href="https://files.realpython.com/media/pycharm-add-shortcut.8c66b2bd12c0.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/pycharm-add-shortcut.8c66b2bd12c0.png" width="1084" height="724" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-add-shortcut.8c66b2bd12c0.png&w=271&sig=e95cc32634af125588e6881ec6992dace79ec667 271w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-add-shortcut.8c66b2bd12c0.png&w=542&sig=836cbeec5256df8e8048aa6a0c3e7d89b2bac444 542w, https://files.realpython.com/media/pycharm-add-shortcut.8c66b2bd12c0.png 1084w" sizes="75vw" alt="Add shortcut in PyCharm"/></a></p>
1472 <p>In the image above, the shortcut is <span class="keys"><kbd class="key-control">Ctrl</kbd><span>+</span><kbd class="key-alt">Alt</kbd><span>+</span><kbd class="key-a">A</kbd></span> for this tool. Add your preferred shortcut in the textbox and click <em>OK</em> for both windows. Now you can now use that shortcut to analyze the file you’re currently working on with Flake8.</p>
1473 <h2 id="pycharm-professional-features">PyCharm Professional Features</h2>
1474 <p>PyCharm Professional is a paid version of PyCharm with more out-of-the-box features and integrations. In this section, you’ll mainly be presented with overviews of its main features and links to the official documentation, where each feature is discussed in detail. Remember that none of the following features is available in the Community edition. </p>
1475 <h3 id="django-support">Django Support</h3>
1476 <p>PyCharm has extensive support for <a href="https://www.djangoproject.com/">Django</a>, one of the most popular and beloved <a href="https://realpython.com/learning-paths/become-python-web-developer/">Python web frameworks</a>. To make sure that it’s enabled, do the following:</p>
1477 <ol>
1478 <li>Open <em>Preferences</em> on Mac or <em>Settings</em> on Windows or Linux.</li>
1479 <li>Choose <em>Languages and Frameworks</em>.</li>
1480 <li>Choose <em>Django</em>.</li>
1481 <li>Check the checkbox <em>Enable Django support</em>.</li>
1482 <li>Apply changes.</li>
1483 </ol>
1484 <p>Now that you’ve enabled Django support, your Django development journey will be a lot easier in PyCharm:</p>
1485 <ul>
1486 <li>When creating a project, you’ll have a dedicated Django project type. This means that, when you choose this type, you’ll have all the necessary files and settings. This is the equivalent of using <code>django-admin startproject mysite</code>.</li>
1487 <li>You can run <code>manage.py</code> commands directly inside PyCharm. </li>
1488 <li>Django templates are supported, including:<ul>
1489 <li>Syntax and error highlighting</li>
1490 <li>Code completion</li>
1491 <li>Navigation</li>
1492 <li>Completion for block names</li>
1493 <li>Completion for custom tags and filters</li>
1494 <li>Quick documentation for tags and filters</li>
1495 <li>Capability to debug them</li>
1496 </ul>
1497 </li>
1498 <li>Code completion in all other Django parts such as views, URLs and models, and code insight support for Django ORM.</li>
1499 <li>Model dependency diagrams for Django models.</li>
1500 </ul>
1501 <p>For more details on Django support, see the <a href="https://www.jetbrains.com/help/pycharm/django-support7.html">official documentation</a>.</p>
1502 <h3 id="database-support">Database Support</h3>
1503 <p>Modern database development is a complex task with many supporting systems and workflows. That’s why JetBrains, the company behind PyCharm, developed a standalone IDE called <a href="https://www.jetbrains.com/datagrip/">DataGrip</a> for that. It’s a separate product from PyCharm with a separate license. </p>
1504 <p>Luckily, PyCharm supports all the features that are available in DataGrip through a plugin called <em>Database tools and SQL</em>, which is enabled by default. With the help of it, you can query, create and manage databases whether they’re working locally, on a server, or in the cloud. The plugin supports MySQL, PostgreSQL, Microsoft SQL Server, SQLite, MariaDB, Oracle, Apache Cassandra, and others. For more information on what you can do with this plugin, check out <a href="https://www.jetbrains.com/help/pycharm/relational-databases.html">the comprehensive documentation on the database support</a>.</p>
1505 <h3 id="thread-concurrency-visualization">Thread Concurrency Visualization</h3>
1506 <p><a href="https://channels.readthedocs.io/en/latest/"><code>Django Channels</code></a>, <a href="https://realpython.com/async-io-python/"><code>asyncio</code></a>, and the recent frameworks like <a href="https://www.starlette.io/"><code>Starlette</code></a> are examples of a growing trend in asynchronous Python programming. While it’s true that asynchronous programs do bring a lot of benefits to the table, it’s also notoriously hard to write and debug them. In such cases, <em>Thread Concurrency Visualization</em> can be just what the doctor ordered because it helps you take full control over your multi-threaded applications and optimize them.</p>
1507 <p>Check out <a href="https://www.jetbrains.com/help/pycharm/thread-concurrency-visualization.html">the comprehensive documentation of this feature</a> for more details.</p>
1508 <h3 id="profiler">Profiler</h3>
1509 <p>Speaking of optimization, profiling is another technique that you can use to optimize your code. With its help, you can see which parts of your code are taking most of the execution time. A profiler runs in the following order of priority: </p>
1510 <ol>
1511 <li><a href="https://vmprof.readthedocs.io/en/latest/"><code>vmprof</code></a> </li>
1512 <li><a href="https://github.com/sumerc/yappi"><code>yappi</code></a></li>
1513 <li><a href="https://docs.python.org/3/library/profile.html"><code>cProfile</code></a></li>
1514 </ol>
1515 <p>If you don’t have <code>vmprof</code> or <code>yappi</code> installed, then it’ll fall back to the standard <code>cProfile</code>. It’s <a href="https://www.jetbrains.com/help/pycharm/profiler.html">well-documented</a>, so I won’t rehash it here. </p>
1516 <h3 id="scientific-mode">Scientific Mode</h3>
1517 <p>Python is not only a language for general and web programming. It also emerged as the best tool for data science and machine learning over these last years thanks to libraries and tools like <a href="http://www.numpy.org/">NumPy</a>, <a href="https://www.scipy.org/">SciPy</a>, <a href="https://scikit-learn.org/">scikit-learn</a>, <a href="https://matplotlib.org/">Matplotlib</a>, <a href="https://jupyter.org/">Jupyter</a>, and more. With such powerful libraries available, you need a powerful IDE to support all the functions such as graphing and analyzing those libraries have. PyCharm provides everything you need as <a href="https://www.jetbrains.com/help/pycharm/matplotlib-support.html">thoroughly documented here</a>. </p>
1518 <h3 id="remote-development">Remote Development</h3>
1519 <p>One common cause of bugs in many applications is that development and production environments differ. Although, in most cases, it’s not possible to provide an exact copy of the production environment for development, pursuing it is a worthy goal.</p>
1520 <p>With PyCharm, you can debug your application using an interpreter that is located on the other computer, such as a Linux VM. As a result, you can have the same interpreter as your production environment to fix and avoid many bugs resulting from the difference between development and production environments. Make sure to check out the <a href="https://www.jetbrains.com/help/pycharm/remote-debugging-with-product.html">official documentation</a> to learn more.</p>
1521 <h2 id="conclusion">Conclusion</h2>
1522 <p>PyCharm is one of best, if not the best, full-featured, dedicated, and versatile IDEs for Python development. It offers a ton of benefits, saving you a lot of time by helping you with routine tasks. Now you know how to be productive with it!</p>
1523 <p>In this article, you learned about a lot, including:</p>
1524 <ul>
1525 <li>Installing PyCharm</li>
1526 <li>Writing code in PyCharm</li>
1527 <li>Running your code in PyCharm</li>
1528 <li>Debugging and testing your code in PyCharm</li>
1529 <li>Editing an existing project in PyCharm</li>
1530 <li>Searching and navigating in PyCharm</li>
1531 <li>Using Version Control in PyCharm</li>
1532 <li>Using Plugins and External Tools in PyCharm</li>
1533 <li>Using PyCharm Professional features, such as Django support and Scientific mode</li>
1534 </ul>
1535 <p>If there’s anything you’d like to ask or share, please reach out in the comments below. There’s also a lot more information at the <a href="https://www.jetbrains.com/pycharm/documentation/">PyCharm website</a> for you to explore.</p>
1536 <div class="alert alert-warning" role="alert"><p><strong>Clone Repo:</strong> <a href="https://realpython.com/optins/view/alcazar-web-framework/" class="alert-link" data-toggle="modal" data-target="#modal-alcazar-web-framework" data-focus="false">Click here to clone the repo you'll use</a> to explore the project-focused features of PyCharm in this tutorial.</p></div>
1537 <hr />
1538 <p><em>[ Improve Your Python With π Python Tricks π β Get a short & sweet Python Trick delivered to your inbox every couple of days. <a href="https://realpython.com/python-tricks/?utm_source=realpython&utm_medium=rss&utm_campaign=footer">>> Click here to learn more and see examples</a> ]</em></p>#
1539 datePublished: #Wed Aug 28 14:00:00 2019#
1540 dateUpdated: #Wed Aug 28 14:00:00 2019#
1541 # Person begin ####################
1542 name: #Real Python#
1543 # Person end ######################
1544 # Item end ########################
1545 # Item begin ######################
1546 id: #https://realpython.com/courses/python-lambda-functions/#
1547 title: #How to Use Python Lambda Functions#
1548 link: #https://realpython.com/courses/python-lambda-functions/#
1549 description: #In this step-by-step course, you'll learn about Python lambda functions. You'll see how they compare with regular functions and how you can use them in accordance with best practices.#
1550 content: #<p>Python and other languages like Java, C#, and even C++ have had lambda functions added to their syntax, whereas languages like LISP or the ML family of languages, Haskell, OCaml, and F#, use lambdas as a core concept. Python lambdas are little, anonymous functions, subject to a more restrictive but more concise syntax than regular Python functions.</p>
1551 <p><strong>By the end of this course, you’ll know:</strong></p>
1552 <ul>
1553 <li>How Python lambdas came to be </li>
1554 <li>How lambdas compare with regular function objects</li>
1555 <li>How to write lambda functions</li>
1556 <li>Which functions in the Python standard library leverage lambdas</li>
1557 <li>When to use or avoid Python lambda functions</li>
1558 </ul>
1559 <p>This course is mainly for intermediate to experienced Python programmers, but it is accessible to any curious minds with interest in programming. All the examples included in this tutorial have been tested with Python 3.7.</p>
1560 <hr />
1561 <p><em>[ Improve Your Python With π Python Tricks π β Get a short & sweet Python Trick delivered to your inbox every couple of days. <a href="https://realpython.com/python-tricks/?utm_source=realpython&utm_medium=rss&utm_campaign=footer">>> Click here to learn more and see examples</a> ]</em></p>#
1562 datePublished: #Tue Aug 27 14:00:00 2019#
1563 dateUpdated: #Tue Aug 27 14:00:00 2019#
1564 # Person begin ####################
1565 name: #Real Python#
1566 # Person end ######################
1567 # Item end ########################
1568 # Item begin ######################
1569 id: #https://realpython.com/openpyxl-excel-spreadsheets-python/#
1570 title: #A Guide to Excel Spreadsheets in Python With openpyxl#
1571 link: #https://realpython.com/openpyxl-excel-spreadsheets-python/#
1572 description: #In this step-by-step tutorial, you'll learn how to handle spreadsheets in Python using the openpyxl package. You'll learn how to manipulate Excel spreadsheets, extract information from spreadsheets, create simple or more complex spreadsheets, including adding styles, charts, and so on.#
1573 content: #<p>Excel spreadsheets are one of those things you might have to deal with at some point. Either it’s because your boss loves them or because marketing needs them, you might have to learn how to work with spreadsheets, and that’s when knowing <code>openpyxl</code> comes in handy!</p>
1574 <p>Spreadsheets are a very intuitive and user-friendly way to manipulate large datasets without any prior technical background. That’s why they’re still so commonly used today.</p>
1575 <p><strong>In this article, you’ll learn how to use openpyxl to:</strong></p>
1576 <ul>
1577 <li>Manipulate Excel spreadsheets with confidence</li>
1578 <li>Extract information from spreadsheets</li>
1579 <li>Create simple or more complex spreadsheets, including adding styles, charts, and so on</li>
1580 </ul>
1581 <p>This article is written for intermediate developers who have a pretty good knowledge of Python data structures, such as <a href="https://realpython.com/python-dicts/">dicts</a> and <a href="https://realpython.com/python-lists-tuples/">lists</a>, but also feel comfortable around <a href="https://realpython.com/python3-object-oriented-programming/">OOP</a> and more intermediate level topics.</p>
1582 <div class="alert alert-warning" role="alert"><p><strong>Download Dataset:</strong> <a href="https://realpython.com/optins/view/openpyxl-sample-dataset/" class="alert-link" data-toggle="modal" data-target="#modal-openpyxl-sample-dataset" data-focus="false">Click here to download the dataset for the openpyxl exercise you'll be following in this tutorial.</a></p></div>
1583
1584 <h2 id="before-you-begin">Before You Begin</h2>
1585 <p>If you ever get asked to extract some data from a database or log file into an Excel spreadsheet, or if you often have to convert an Excel spreadsheet into some more usable programmatic form, then this tutorial is perfect for you. Let’s jump into the <code>openpyxl</code> caravan!</p>
1586 <h3 id="practical-use-cases">Practical Use Cases</h3>
1587 <p>First things first, when would you need to use a package like <code>openpyxl</code> in a real-world scenario? You’ll see a few examples below, but really, there are hundreds of possible scenarios where this knowledge could come in handy.</p>
1588 <h4 id="importing-new-products-into-a-database">Importing New Products Into a Database</h4>
1589 <p>You are responsible for tech in an online store company, and your boss doesn’t want to pay for a cool and expensive CMS system.</p>
1590 <p>Every time they want to add new products to the online store, they come to you with an Excel spreadsheet with a few hundred rows and, for each of them, you have the product name, description, price, and so forth.</p>
1591 <p>Now, to import the data, you’ll have to iterate over each spreadsheet row and add each product to the online store.</p>
1592 <h4 id="exporting-database-data-into-a-spreadsheet">Exporting Database Data Into a Spreadsheet</h4>
1593 <p>Say you have a Database table where you record all your users’ information, including name, phone number, email address, and so forth.</p>
1594 <p>Now, the Marketing team wants to contact all users to give them some discounted offer or promotion. However, they don’t have access to the Database, or they don’t know how to use SQL to extract that information easily.</p>
1595 <p>What can you do to help? Well, you can make a quick script using <code>openpyxl</code> that iterates over every single User record and puts all the essential information into an Excel spreadsheet.</p>
1596 <p>That’s gonna earn you an extra slice of cake at your company’s next birthday party!</p>
1597 <h4 id="appending-information-to-an-existing-spreadsheet">Appending Information to an Existing Spreadsheet</h4>
1598 <p>You may also have to open a spreadsheet, read the information in it and, according to some business logic, append more data to it.</p>
1599 <p>For example, using the online store scenario again, say you get an Excel spreadsheet with a list of users and you need to append to each row the total amount they’ve spent in your store.</p>
1600 <p>This data is in the Database and, in order to do this, you have to read the spreadsheet, iterate through each row, fetch the total amount spent from the Database and then write back to the spreadsheet.</p>
1601 <p>Not a problem for <code>openpyxl</code>!</p>
1602 <h3 id="learning-some-basic-excel-terminology">Learning Some Basic Excel Terminology</h3>
1603 <p>Here’s a quick list of basic terms you’ll see when you’re working with Excel spreadsheets:</p>
1604 <div class="table-responsive">
1605 <table class="table table-hover">
1606 <thead>
1607 <tr>
1608 <th>Term</th>
1609 <th>Explanation</th>
1610 </tr>
1611 </thead>
1612 <tbody>
1613 <tr>
1614 <td>Spreadsheet or Workbook</td>
1615 <td>A <strong>Spreadsheet</strong> is the main file you are creating or working with.</td>
1616 </tr>
1617 <tr>
1618 <td>Worksheet or Sheet</td>
1619 <td>A <strong>Sheet</strong> is used to split different kinds of content within the same spreadsheet. A <strong>Spreadsheet</strong> can have one or more <strong>Sheets</strong>.</td>
1620 </tr>
1621 <tr>
1622 <td>Column</td>
1623 <td>A <strong>Column</strong> is a vertical line, and it’s represented by an uppercase letter: <em>A</em>.</td>
1624 </tr>
1625 <tr>
1626 <td>Row</td>
1627 <td>A <strong>Row</strong> is a horizontal line, and it’s represented by a number: <em>1</em>.</td>
1628 </tr>
1629 <tr>
1630 <td>Cell</td>
1631 <td>A <strong>Cell</strong> is a combination of <strong>Column</strong> and <strong>Row</strong>, represented by both an uppercase letter and a number: <em>A1</em>.</td>
1632 </tr>
1633 </tbody>
1634 </table>
1635 </div>
1636 <h3 id="getting-started-with-openpyxl">Getting Started With openpyxl</h3>
1637 <p>Now that you’re aware of the benefits of a tool like <code>openpyxl</code>, let’s get down to it and start by installing the package. For this tutorial, you should use Python 3.7 and openpyxl 2.6.2. To install the package, you can do the following:</p>
1638 <div class="highlight sh"><pre><span></span><span class="gp">$</span> pip install openpyxl
1639 </pre></div>
1640
1641 <p>After you install the package, you should be able to create a super simple spreadsheet with the following code:</p>
1642 <div class="highlight python"><pre><span></span><span class="kn">from</span> <span class="nn">openpyxl</span> <span class="k">import</span> <span class="n">Workbook</span>
1643
1644 <span class="n">workbook</span> <span class="o">=</span> <span class="n">Workbook</span><span class="p">()</span>
1645 <span class="n">sheet</span> <span class="o">=</span> <span class="n">workbook</span><span class="o">.</span><span class="n">active</span>
1646
1647 <span class="n">sheet</span><span class="p">[</span><span class="s2">"A1"</span><span class="p">]</span> <span class="o">=</span> <span class="s2">"hello"</span>
1648 <span class="n">sheet</span><span class="p">[</span><span class="s2">"B1"</span><span class="p">]</span> <span class="o">=</span> <span class="s2">"world!"</span>
1649
1650 <span class="n">workbook</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="n">filename</span><span class="o">=</span><span class="s2">"hello_world.xlsx"</span><span class="p">)</span>
1651 </pre></div>
1652
1653 <p>The code above should create a file called <code>hello_world.xlsx</code> in the folder you are using to run the code. If you open that file with Excel you should see something like this:</p>
1654 <p><a href="https://files.realpython.com/media/Screenshot_2019-06-24_16.54.45.e646867e4dbb.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/Screenshot_2019-06-24_16.54.45.e646867e4dbb.png" width="2160" height="1352" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_16.54.45.e646867e4dbb.png&w=540&sig=4c3acdcf35f528b6ed0cf6e299c2575781934414 540w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_16.54.45.e646867e4dbb.png&w=1080&sig=328d4ff12cec767d684f5b7666380d9f23a2a548 1080w, https://files.realpython.com/media/Screenshot_2019-06-24_16.54.45.e646867e4dbb.png 2160w" sizes="75vw" alt="A Simple Hello World Spreadsheet"/></a></p>
1655 <p><em>Woohoo</em>, your first spreadsheet created!</p>
1656 <h2 id="reading-excel-spreadsheets-with-openpyxl">Reading Excel Spreadsheets With openpyxl</h2>
1657 <p>Let’s start with the most essential thing one can do with a spreadsheet: read it.</p>
1658 <p>You’ll go from a straightforward approach to reading a spreadsheet to more complex examples where you read the data and convert it into more useful Python structures.</p>
1659 <h3 id="dataset-for-this-tutorial">Dataset for This Tutorial</h3>
1660 <p>Before you dive deep into some code examples, you should <strong>download this sample dataset</strong> and store it somewhere as <code>sample.xlsx</code>:</p>
1661 <div class="alert alert-warning" role="alert"><p><strong>Download Dataset:</strong> <a href="https://realpython.com/optins/view/openpyxl-sample-dataset/" class="alert-link" data-toggle="modal" data-target="#modal-openpyxl-sample-dataset" data-focus="false">Click here to download the dataset for the openpyxl exercise you'll be following in this tutorial.</a></p></div>
1662
1663 <p>This is one of the datasets you’ll be using throughout this tutorial, and it’s a spreadsheet with a sample of real data from Amazon’s online product reviews. This dataset is only a tiny fraction of what Amazon <a href="https://registry.opendata.aws/amazon-reviews/">provides</a>, but for testing purposes, it’s more than enough.</p>
1664 <h3 id="a-simple-approach-to-reading-an-excel-spreadsheet">A Simple Approach to Reading an Excel Spreadsheet</h3>
1665 <p>Finally, let’s start reading some spreadsheets! To begin with, open our sample spreadsheet:</p>
1666 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">openpyxl</span> <span class="k">import</span> <span class="n">load_workbook</span>
1667 <span class="gp">>>> </span><span class="n">workbook</span> <span class="o">=</span> <span class="n">load_workbook</span><span class="p">(</span><span class="n">filename</span><span class="o">=</span><span class="s2">"sample.xlsx"</span><span class="p">)</span>
1668 <span class="gp">>>> </span><span class="n">workbook</span><span class="o">.</span><span class="n">sheetnames</span>
1669 <span class="go">['Sheet 1']</span>
1670
1671 <span class="gp">>>> </span><span class="n">sheet</span> <span class="o">=</span> <span class="n">workbook</span><span class="o">.</span><span class="n">active</span>
1672 <span class="gp">>>> </span><span class="n">sheet</span>
1673 <span class="go"><Worksheet "Sheet 1"></span>
1674
1675 <span class="gp">>>> </span><span class="n">sheet</span><span class="o">.</span><span class="n">title</span>
1676 <span class="go">'Sheet 1'</span>
1677 </pre></div>
1678
1679 <p>In the code above, you first open the spreadsheet <code>sample.xlsx</code> using <code>load_workbook()</code>, and then you can use <code>workbook.sheetnames</code> to see all the sheets you have available to work with. After that, <code>workbook.active</code> selects the first available sheet and, in this case, you can see that it selects <strong>Sheet 1</strong> automatically. Using these methods is the default way of opening a spreadsheet, and you’ll see it many times during this tutorial.</p>
1680 <p>Now, after opening a spreadsheet, you can easily retrieve data from it like this:</p>
1681 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">sheet</span><span class="p">[</span><span class="s2">"A1"</span><span class="p">]</span>
1682 <span class="go"><Cell 'Sheet 1'.A1></span>
1683
1684 <span class="gp">>>> </span><span class="n">sheet</span><span class="p">[</span><span class="s2">"A1"</span><span class="p">]</span><span class="o">.</span><span class="n">value</span>
1685 <span class="go">'marketplace'</span>
1686
1687 <span class="gp">>>> </span><span class="n">sheet</span><span class="p">[</span><span class="s2">"F10"</span><span class="p">]</span><span class="o">.</span><span class="n">value</span>
1688 <span class="go">"G-Shock Men's Grey Sport Watch"</span>
1689 </pre></div>
1690
1691 <p>To return the actual value of a cell, you need to do <code>.value</code>. Otherwise, you’ll get the main <code>Cell</code> object. You can also use the method <code>.cell()</code> to retrieve a cell using index notation. Remember to add <code>.value</code> to get the actual value and not a <code>Cell</code> object:</p>
1692 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">sheet</span><span class="o">.</span><span class="n">cell</span><span class="p">(</span><span class="n">row</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">column</span><span class="o">=</span><span class="mi">6</span><span class="p">)</span>
1693 <span class="go"><Cell 'Sheet 1'.F10></span>
1694
1695 <span class="gp">>>> </span><span class="n">sheet</span><span class="o">.</span><span class="n">cell</span><span class="p">(</span><span class="n">row</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">column</span><span class="o">=</span><span class="mi">6</span><span class="p">)</span><span class="o">.</span><span class="n">value</span>
1696 <span class="go">"G-Shock Men's Grey Sport Watch"</span>
1697 </pre></div>
1698
1699 <p>You can see that the results returned are the same, no matter which way you decide to go with. However, in this tutorial, you’ll be mostly using the first approach: <code>["A1"]</code>.</p>
1700 <div class="alert alert-primary" role="alert">
1701 <p><strong>Note:</strong> Even though in Python you’re used to a zero-indexed notation, with spreadsheets you’ll always use a one-indexed notation where the first row or column always has index <code>1</code>.</p>
1702 </div>
1703 <p>The above shows you the quickest way to open a spreadsheet. However, you can pass additional parameters to change the way a spreadsheet is loaded.</p>
1704 <h4 id="additional-reading-options">Additional Reading Options</h4>
1705 <p>There are a few arguments you can pass to <code>load_workbook()</code> that change the way a spreadsheet is loaded. The most important ones are the following two Booleans:</p>
1706 <ol>
1707 <li><strong>read_only</strong> loads a spreadsheet in read-only mode allowing you to open very large Excel files.</li>
1708 <li><strong>data_only</strong> ignores loading formulas and instead loads only the resulting values.</li>
1709 </ol>
1710 <h3 id="importing-data-from-a-spreadsheet">Importing Data From a Spreadsheet</h3>
1711 <p>Now that you’ve learned the basics about loading a spreadsheet, it’s about time you get to the fun part: <strong>the iteration and actual usage of the values within the spreadsheet</strong>.</p>
1712 <p>This section is where you’ll learn all the different ways you can iterate through the data, but also how to convert that data into something usable and, more importantly, how to do it in a Pythonic way.</p>
1713 <h4 id="iterating-through-the-data">Iterating Through the Data</h4>
1714 <p>There are a few different ways you can iterate through the data depending on your needs.</p>
1715 <p>You can slice the data with a combination of columns and rows:</p>
1716 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">sheet</span><span class="p">[</span><span class="s2">"A1:C2"</span><span class="p">]</span>
1717 <span class="go">((<Cell 'Sheet 1'.A1>, <Cell 'Sheet 1'.B1>, <Cell 'Sheet 1'.C1>),</span>
1718 <span class="go"> (<Cell 'Sheet 1'.A2>, <Cell 'Sheet 1'.B2>, <Cell 'Sheet 1'.C2>))</span>
1719 </pre></div>
1720
1721 <p>You can get ranges of rows or columns:</p>
1722 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="c1"># Get all cells from column A</span>
1723 <span class="gp">>>> </span><span class="n">sheet</span><span class="p">[</span><span class="s2">"A"</span><span class="p">]</span>
1724 <span class="go">(<Cell 'Sheet 1'.A1>,</span>
1725 <span class="go"> <Cell 'Sheet 1'.A2>,</span>
1726 <span class="go"> ...</span>
1727 <span class="go"> <Cell 'Sheet 1'.A99>,</span>
1728 <span class="go"> <Cell 'Sheet 1'.A100>)</span>
1729
1730 <span class="gp">>>> </span><span class="c1"># Get all cells for a range of columns</span>
1731 <span class="gp">>>> </span><span class="n">sheet</span><span class="p">[</span><span class="s2">"A:B"</span><span class="p">]</span>
1732 <span class="go">((<Cell 'Sheet 1'.A1>,</span>
1733 <span class="go"> <Cell 'Sheet 1'.A2>,</span>
1734 <span class="go"> ...</span>
1735 <span class="go"> <Cell 'Sheet 1'.A99>,</span>
1736 <span class="go"> <Cell 'Sheet 1'.A100>),</span>
1737 <span class="go"> (<Cell 'Sheet 1'.B1>,</span>
1738 <span class="go"> <Cell 'Sheet 1'.B2>,</span>
1739 <span class="go"> ...</span>
1740 <span class="go"> <Cell 'Sheet 1'.B99>,</span>
1741 <span class="go"> <Cell 'Sheet 1'.B100>))</span>
1742
1743 <span class="gp">>>> </span><span class="c1"># Get all cells from row 5</span>
1744 <span class="gp">>>> </span><span class="n">sheet</span><span class="p">[</span><span class="mi">5</span><span class="p">]</span>
1745 <span class="go">(<Cell 'Sheet 1'.A5>,</span>
1746 <span class="go"> <Cell 'Sheet 1'.B5>,</span>
1747 <span class="go"> ...</span>
1748 <span class="go"> <Cell 'Sheet 1'.N5>,</span>
1749 <span class="go"> <Cell 'Sheet 1'.O5>)</span>
1750
1751 <span class="gp">>>> </span><span class="c1"># Get all cells for a range of rows</span>
1752 <span class="gp">>>> </span><span class="n">sheet</span><span class="p">[</span><span class="mi">5</span><span class="p">:</span><span class="mi">6</span><span class="p">]</span>
1753 <span class="go">((<Cell 'Sheet 1'.A5>,</span>
1754 <span class="go"> <Cell 'Sheet 1'.B5>,</span>
1755 <span class="go"> ...</span>
1756 <span class="go"> <Cell 'Sheet 1'.N5>,</span>
1757 <span class="go"> <Cell 'Sheet 1'.O5>),</span>
1758 <span class="go"> (<Cell 'Sheet 1'.A6>,</span>
1759 <span class="go"> <Cell 'Sheet 1'.B6>,</span>
1760 <span class="go"> ...</span>
1761 <span class="go"> <Cell 'Sheet 1'.N6>,</span>
1762 <span class="go"> <Cell 'Sheet 1'.O6>))</span>
1763 </pre></div>
1764
1765 <p>You’ll notice that all of the above examples return a <code>tuple</code>. If you want to refresh your memory on how to handle <code>tuples</code> in Python, check out the article on <a href="https://realpython.com/python-lists-tuples/#python-tuples">Lists and Tuples in Python</a>.</p>
1766 <p>There are also multiple ways of using normal Python <a href="https://realpython.com/introduction-to-python-generators/">generators</a> to go through the data. The main methods you can use to achieve this are:</p>
1767 <ul>
1768 <li><code>.iter_rows()</code></li>
1769 <li><code>.iter_cols()</code></li>
1770 </ul>
1771 <p>Both methods can receive the following arguments:</p>
1772 <ul>
1773 <li><code>min_row</code></li>
1774 <li><code>max_row</code></li>
1775 <li><code>min_col</code></li>
1776 <li><code>max_col</code></li>
1777 </ul>
1778 <p>These arguments are used to set boundaries for the iteration:</p>
1779 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">sheet</span><span class="o">.</span><span class="n">iter_rows</span><span class="p">(</span><span class="n">min_row</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
1780 <span class="gp">... </span> <span class="n">max_row</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span>
1781 <span class="gp">... </span> <span class="n">min_col</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
1782 <span class="gp">... </span> <span class="n">max_col</span><span class="o">=</span><span class="mi">3</span><span class="p">):</span>
1783 <span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">row</span><span class="p">)</span>
1784 <span class="go">(<Cell 'Sheet 1'.A1>, <Cell 'Sheet 1'.B1>, <Cell 'Sheet 1'.C1>)</span>
1785 <span class="go">(<Cell 'Sheet 1'.A2>, <Cell 'Sheet 1'.B2>, <Cell 'Sheet 1'.C2>)</span>
1786
1787
1788 <span class="gp">>>> </span><span class="k">for</span> <span class="n">column</span> <span class="ow">in</span> <span class="n">sheet</span><span class="o">.</span><span class="n">iter_cols</span><span class="p">(</span><span class="n">min_row</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
1789 <span class="gp">... </span> <span class="n">max_row</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span>
1790 <span class="gp">... </span> <span class="n">min_col</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
1791 <span class="gp">... </span> <span class="n">max_col</span><span class="o">=</span><span class="mi">3</span><span class="p">):</span>
1792 <span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">column</span><span class="p">)</span>
1793 <span class="go">(<Cell 'Sheet 1'.A1>, <Cell 'Sheet 1'.A2>)</span>
1794 <span class="go">(<Cell 'Sheet 1'.B1>, <Cell 'Sheet 1'.B2>)</span>
1795 <span class="go">(<Cell 'Sheet 1'.C1>, <Cell 'Sheet 1'.C2>)</span>
1796 </pre></div>
1797
1798 <p>You’ll notice that in the first example, when iterating through the rows using <code>.iter_rows()</code>, you get one <code>tuple</code> element per row selected. While when using <code>.iter_cols()</code> and iterating through columns, you’ll get one <code>tuple</code> per column instead.</p>
1799 <p>One additional argument you can pass to both methods is the Boolean <code>values_only</code>. When it’s set to <code>True</code>, the values of the cell are returned, instead of the <code>Cell</code> object:</p>
1800 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">for</span> <span class="n">value</span> <span class="ow">in</span> <span class="n">sheet</span><span class="o">.</span><span class="n">iter_rows</span><span class="p">(</span><span class="n">min_row</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
1801 <span class="gp">... </span> <span class="n">max_row</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span>
1802 <span class="gp">... </span> <span class="n">min_col</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
1803 <span class="gp">... </span> <span class="n">max_col</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span>
1804 <span class="gp">... </span> <span class="n">values_only</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
1805 <span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
1806 <span class="go">('marketplace', 'customer_id', 'review_id')</span>
1807 <span class="go">('US', 3653882, 'R3O9SGZBVQBV76')</span>
1808 </pre></div>
1809
1810 <p>If you want to iterate through the whole dataset, then you can also use the attributes <code>.rows</code> or <code>.columns</code> directly, which are shortcuts to using <code>.iter_rows()</code> and <code>.iter_cols()</code> without any arguments:</p>
1811 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">sheet</span><span class="o">.</span><span class="n">rows</span><span class="p">:</span>
1812 <span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">row</span><span class="p">)</span>
1813 <span class="go">(<Cell 'Sheet 1'.A1>, <Cell 'Sheet 1'.B1>, <Cell 'Sheet 1'.C1></span>
1814 <span class="gp">...</span>
1815 <span class="go"><Cell 'Sheet 1'.M100>, <Cell 'Sheet 1'.N100>, <Cell 'Sheet 1'.O100>)</span>
1816 </pre></div>
1817
1818 <p>These shortcuts are very useful when you’re iterating through the whole dataset.</p>
1819 <h4 id="manipulate-data-using-pythons-default-data-structures">Manipulate Data Using Python’s Default Data Structures</h4>
1820 <p>Now that you know the basics of iterating through the data in a workbook, let’s look at smart ways of converting that data into Python structures.</p>
1821 <p>As you saw earlier, the result from all iterations comes in the form of <code>tuples</code>. However, since a <code>tuple</code> is nothing more than an immutable <code>list</code>, you can easily access its data and transform it into other structures.</p>
1822 <p>For example, say you want to extract product information from the <code>sample.xlsx</code> spreadsheet and into a dictionary where each key is a product ID.</p>
1823 <p>A straightforward way to do this is to iterate over all the rows, pick the columns you know are related to product information, and then store that in a dictionary. Let’s code this out!</p>
1824 <p>First of all, have a look at the headers and see what information you care most about:</p>
1825 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">for</span> <span class="n">value</span> <span class="ow">in</span> <span class="n">sheet</span><span class="o">.</span><span class="n">iter_rows</span><span class="p">(</span><span class="n">min_row</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
1826 <span class="gp">... </span> <span class="n">max_row</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
1827 <span class="gp">... </span> <span class="n">values_only</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
1828 <span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
1829 <span class="go">('marketplace', 'customer_id', 'review_id', 'product_id', ...)</span>
1830 </pre></div>
1831
1832 <p>This code returns a list of all the column names you have in the spreadsheet. To start, grab the columns with names:</p>
1833 <ul>
1834 <li><code>product_id</code></li>
1835 <li><code>product_parent</code></li>
1836 <li><code>product_title</code></li>
1837 <li><code>product_category</code></li>
1838 </ul>
1839 <p>Lucky for you, the columns you need are all next to each other so you can use the <code>min_column</code> and <code>max_column</code> to easily get the data you want:</p>
1840 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">for</span> <span class="n">value</span> <span class="ow">in</span> <span class="n">sheet</span><span class="o">.</span><span class="n">iter_rows</span><span class="p">(</span><span class="n">min_row</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span>
1841 <span class="gp">... </span> <span class="n">min_col</span><span class="o">=</span><span class="mi">4</span><span class="p">,</span>
1842 <span class="gp">... </span> <span class="n">max_col</span><span class="o">=</span><span class="mi">7</span><span class="p">,</span>
1843 <span class="gp">... </span> <span class="n">values_only</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
1844 <span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
1845 <span class="go">('B00FALQ1ZC', 937001370, 'Invicta Women\'s 15150 "Angel" 18k Yellow...)</span>
1846 <span class="go">('B00D3RGO20', 484010722, "Kenneth Cole New York Women's KC4944...)</span>
1847 <span class="gp">...</span>
1848 </pre></div>
1849
1850 <p>Nice! Now that you know how to get all the important product information you need, let’s put that data into a dictionary:</p>
1851 <div class="highlight python"><pre><span></span><span class="kn">import</span> <span class="nn">json</span>
1852 <span class="kn">from</span> <span class="nn">openpyxl</span> <span class="k">import</span> <span class="n">load_workbook</span>
1853
1854 <span class="n">workbook</span> <span class="o">=</span> <span class="n">load_workbook</span><span class="p">(</span><span class="n">filename</span><span class="o">=</span><span class="s2">"sample.xlsx"</span><span class="p">)</span>
1855 <span class="n">sheet</span> <span class="o">=</span> <span class="n">workbook</span><span class="o">.</span><span class="n">active</span>
1856
1857 <span class="n">products</span> <span class="o">=</span> <span class="p">{}</span>
1858
1859 <span class="c1"># Using the values_only because you want to return the cells' values</span>
1860 <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">sheet</span><span class="o">.</span><span class="n">iter_rows</span><span class="p">(</span><span class="n">min_row</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span>
1861 <span class="n">min_col</span><span class="o">=</span><span class="mi">4</span><span class="p">,</span>
1862 <span class="n">max_col</span><span class="o">=</span><span class="mi">7</span><span class="p">,</span>
1863 <span class="n">values_only</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
1864 <span class="n">product_id</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
1865 <span class="n">product</span> <span class="o">=</span> <span class="p">{</span>
1866 <span class="s2">"parent"</span><span class="p">:</span> <span class="n">row</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span>
1867 <span class="s2">"title"</span><span class="p">:</span> <span class="n">row</span><span class="p">[</span><span class="mi">2</span><span class="p">],</span>
1868 <span class="s2">"category"</span><span class="p">:</span> <span class="n">row</span><span class="p">[</span><span class="mi">3</span><span class="p">]</span>
1869 <span class="p">}</span>
1870 <span class="n">products</span><span class="p">[</span><span class="n">product_id</span><span class="p">]</span> <span class="o">=</span> <span class="n">product</span>
1871
1872 <span class="c1"># Using json here to be able to format the output for displaying later</span>
1873 <span class="nb">print</span><span class="p">(</span><span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">products</span><span class="p">))</span>
1874 </pre></div>
1875
1876 <p>The code above returns a JSON similar to this:</p>
1877 <div class="highlight json"><pre><span></span><span class="p">{</span>
1878 <span class="nt">"B00FALQ1ZC"</span><span class="p">:</span> <span class="p">{</span>
1879 <span class="nt">"parent"</span><span class="p">:</span> <span class="mi">937001370</span><span class="p">,</span>
1880 <span class="nt">"title"</span><span class="p">:</span> <span class="s2">"Invicta Women's 15150 ..."</span><span class="p">,</span>
1881 <span class="nt">"category"</span><span class="p">:</span> <span class="s2">"Watches"</span>
1882 <span class="p">},</span>
1883 <span class="nt">"B00D3RGO20"</span><span class="p">:</span> <span class="p">{</span>
1884 <span class="nt">"parent"</span><span class="p">:</span> <span class="mi">484010722</span><span class="p">,</span>
1885 <span class="nt">"title"</span><span class="p">:</span> <span class="s2">"Kenneth Cole New York ..."</span><span class="p">,</span>
1886 <span class="nt">"category"</span><span class="p">:</span> <span class="s2">"Watches"</span>
1887 <span class="p">}</span>
1888 <span class="p">}</span>
1889 </pre></div>
1890
1891 <p>Here you can see that the output is trimmed to 2 products only, but if you run the script as it is, then you should get 98 products.</p>
1892 <h4 id="convert-data-into-python-classes">Convert Data Into Python Classes</h4>
1893 <p>To finalize the reading section of this tutorial, let’s dive into Python classes and see how you could improve on the example above and better structure the data.</p>
1894 <p>For this, you’ll be using the new Python <a href="https://realpython.com/python-data-classes/">Data Classes</a> that are available from Python 3.7. If you’re using an older version of Python, then you can use the default <a href="https://realpython.com/python3-object-oriented-programming/#classes-in-python">Classes</a> instead.</p>
1895 <p>So, first things first, let’s look at the data you have and decide what you want to store and how you want to store it.</p>
1896 <p>As you saw right at the start, this data comes from Amazon, and it’s a list of product reviews. You can check the <a href="https://s3.amazonaws.com/amazon-reviews-pds/tsv/index.txt">list of all the columns and their meaning</a> on Amazon.</p>
1897 <p>There are two significant elements you can extract from the data available:</p>
1898 <ol>
1899 <li>Products</li>
1900 <li>Reviews</li>
1901 </ol>
1902 <p>A <strong>Product</strong> has:</p>
1903 <ul>
1904 <li>ID</li>
1905 <li>Title</li>
1906 <li>Parent</li>
1907 <li>Category</li>
1908 </ul>
1909 <p>The <strong>Review</strong> has a few more fields:</p>
1910 <ul>
1911 <li>ID</li>
1912 <li>Customer ID</li>
1913 <li>Stars</li>
1914 <li>Headline</li>
1915 <li>Body</li>
1916 <li>Date</li>
1917 </ul>
1918 <p>You can ignore a few of the review fields to make things a bit simpler.</p>
1919 <p>So, a straightforward implementation of these two classes could be written in a separate file <code>classes.py</code>:</p>
1920 <div class="highlight python"><pre><span></span><span class="kn">import</span> <span class="nn">datetime</span>
1921 <span class="kn">from</span> <span class="nn">dataclasses</span> <span class="k">import</span> <span class="n">dataclass</span>
1922
1923 <span class="nd">@dataclass</span>
1924 <span class="k">class</span> <span class="nc">Product</span><span class="p">:</span>
1925 <span class="nb">id</span><span class="p">:</span> <span class="nb">str</span>
1926 <span class="n">parent</span><span class="p">:</span> <span class="nb">str</span>
1927 <span class="n">title</span><span class="p">:</span> <span class="nb">str</span>
1928 <span class="n">category</span><span class="p">:</span> <span class="nb">str</span>
1929
1930 <span class="nd">@dataclass</span>
1931 <span class="k">class</span> <span class="nc">Review</span><span class="p">:</span>
1932 <span class="nb">id</span><span class="p">:</span> <span class="nb">str</span>
1933 <span class="n">customer_id</span><span class="p">:</span> <span class="nb">str</span>
1934 <span class="n">stars</span><span class="p">:</span> <span class="nb">int</span>
1935 <span class="n">headline</span><span class="p">:</span> <span class="nb">str</span>
1936 <span class="n">body</span><span class="p">:</span> <span class="nb">str</span>
1937 <span class="n">date</span><span class="p">:</span> <span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span>
1938 </pre></div>
1939
1940 <p>After defining your data classes, you need to convert the data from the spreadsheet into these new structures.</p>
1941 <p>Before doing the conversion, it’s worth looking at our header again and creating a mapping between columns and the fields you need:</p>
1942 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">for</span> <span class="n">value</span> <span class="ow">in</span> <span class="n">sheet</span><span class="o">.</span><span class="n">iter_rows</span><span class="p">(</span><span class="n">min_row</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
1943 <span class="gp">... </span> <span class="n">max_row</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
1944 <span class="gp">... </span> <span class="n">values_only</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
1945 <span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
1946 <span class="go">('marketplace', 'customer_id', 'review_id', 'product_id', ...)</span>
1947
1948 <span class="gp">>>> </span><span class="c1"># Or an alternative</span>
1949 <span class="gp">>>> </span><span class="k">for</span> <span class="n">cell</span> <span class="ow">in</span> <span class="n">sheet</span><span class="p">[</span><span class="mi">1</span><span class="p">]:</span>
1950 <span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">cell</span><span class="o">.</span><span class="n">value</span><span class="p">)</span>
1951 <span class="go">marketplace</span>
1952 <span class="go">customer_id</span>
1953 <span class="go">review_id</span>
1954 <span class="go">product_id</span>
1955 <span class="go">product_parent</span>
1956 <span class="gp">...</span>
1957 </pre></div>
1958
1959 <p>Let’s create a file <code>mapping.py</code> where you have a list of all the field names and their column location (zero-indexed) on the spreadsheet:</p>
1960 <div class="highlight python"><pre><span></span><span class="c1"># Product fields</span>
1961 <span class="n">PRODUCT_ID</span> <span class="o">=</span> <span class="mi">3</span>
1962 <span class="n">PRODUCT_PARENT</span> <span class="o">=</span> <span class="mi">4</span>
1963 <span class="n">PRODUCT_TITLE</span> <span class="o">=</span> <span class="mi">5</span>
1964 <span class="n">PRODUCT_CATEGORY</span> <span class="o">=</span> <span class="mi">6</span>
1965
1966 <span class="c1"># Review fields</span>
1967 <span class="n">REVIEW_ID</span> <span class="o">=</span> <span class="mi">2</span>
1968 <span class="n">REVIEW_CUSTOMER</span> <span class="o">=</span> <span class="mi">1</span>
1969 <span class="n">REVIEW_STARS</span> <span class="o">=</span> <span class="mi">7</span>
1970 <span class="n">REVIEW_HEADLINE</span> <span class="o">=</span> <span class="mi">12</span>
1971 <span class="n">REVIEW_BODY</span> <span class="o">=</span> <span class="mi">13</span>
1972 <span class="n">REVIEW_DATE</span> <span class="o">=</span> <span class="mi">14</span>
1973 </pre></div>
1974
1975 <p>You don’t necessarily have to do the mapping above. It’s more for readability when parsing the row data, so you don’t end up with a lot of magic numbers lying around.</p>
1976 <p>Finally, let’s look at the code needed to parse the spreadsheet data into a list of product and review objects:</p>
1977 <div class="highlight python"><pre><span></span><span class="kn">from</span> <span class="nn">datetime</span> <span class="k">import</span> <span class="n">datetime</span>
1978 <span class="kn">from</span> <span class="nn">openpyxl</span> <span class="k">import</span> <span class="n">load_workbook</span>
1979 <span class="kn">from</span> <span class="nn">classes</span> <span class="k">import</span> <span class="n">Product</span><span class="p">,</span> <span class="n">Review</span>
1980 <span class="kn">from</span> <span class="nn">mapping</span> <span class="k">import</span> <span class="n">PRODUCT_ID</span><span class="p">,</span> <span class="n">PRODUCT_PARENT</span><span class="p">,</span> <span class="n">PRODUCT_TITLE</span><span class="p">,</span> \
1981 <span class="n">PRODUCT_CATEGORY</span><span class="p">,</span> <span class="n">REVIEW_DATE</span><span class="p">,</span> <span class="n">REVIEW_ID</span><span class="p">,</span> <span class="n">REVIEW_CUSTOMER</span><span class="p">,</span> \
1982 <span class="n">REVIEW_STARS</span><span class="p">,</span> <span class="n">REVIEW_HEADLINE</span><span class="p">,</span> <span class="n">REVIEW_BODY</span>
1983
1984 <span class="c1"># Using the read_only method since you're not gonna be editing the spreadsheet</span>
1985 <span class="n">workbook</span> <span class="o">=</span> <span class="n">load_workbook</span><span class="p">(</span><span class="n">filename</span><span class="o">=</span><span class="s2">"sample.xlsx"</span><span class="p">,</span> <span class="n">read_only</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
1986 <span class="n">sheet</span> <span class="o">=</span> <span class="n">workbook</span><span class="o">.</span><span class="n">active</span>
1987
1988 <span class="n">products</span> <span class="o">=</span> <span class="p">[]</span>
1989 <span class="n">reviews</span> <span class="o">=</span> <span class="p">[]</span>
1990
1991 <span class="c1"># Using the values_only because you just want to return the cell value</span>
1992 <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">sheet</span><span class="o">.</span><span class="n">iter_rows</span><span class="p">(</span><span class="n">min_row</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">values_only</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
1993 <span class="n">product</span> <span class="o">=</span> <span class="n">Product</span><span class="p">(</span><span class="nb">id</span><span class="o">=</span><span class="n">row</span><span class="p">[</span><span class="n">PRODUCT_ID</span><span class="p">],</span>
1994 <span class="n">parent</span><span class="o">=</span><span class="n">row</span><span class="p">[</span><span class="n">PRODUCT_PARENT</span><span class="p">],</span>
1995 <span class="n">title</span><span class="o">=</span><span class="n">row</span><span class="p">[</span><span class="n">PRODUCT_TITLE</span><span class="p">],</span>
1996 <span class="n">category</span><span class="o">=</span><span class="n">row</span><span class="p">[</span><span class="n">PRODUCT_CATEGORY</span><span class="p">])</span>
1997 <span class="n">products</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">product</span><span class="p">)</span>
1998
1999 <span class="c1"># You need to parse the date from the spreadsheet into a datetime format</span>
2000 <span class="n">spread_date</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="n">REVIEW_DATE</span><span class="p">]</span>
2001 <span class="n">parsed_date</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">strptime</span><span class="p">(</span><span class="n">spread_date</span><span class="p">,</span> <span class="s2">"%Y-%m-</span><span class="si">%d</span><span class="s2">"</span><span class="p">)</span>
2002
2003 <span class="n">review</span> <span class="o">=</span> <span class="n">Review</span><span class="p">(</span><span class="nb">id</span><span class="o">=</span><span class="n">row</span><span class="p">[</span><span class="n">REVIEW_ID</span><span class="p">],</span>
2004 <span class="n">customer_id</span><span class="o">=</span><span class="n">row</span><span class="p">[</span><span class="n">REVIEW_CUSTOMER</span><span class="p">],</span>
2005 <span class="n">stars</span><span class="o">=</span><span class="n">row</span><span class="p">[</span><span class="n">REVIEW_STARS</span><span class="p">],</span>
2006 <span class="n">headline</span><span class="o">=</span><span class="n">row</span><span class="p">[</span><span class="n">REVIEW_HEADLINE</span><span class="p">],</span>
2007 <span class="n">body</span><span class="o">=</span><span class="n">row</span><span class="p">[</span><span class="n">REVIEW_BODY</span><span class="p">],</span>
2008 <span class="n">date</span><span class="o">=</span><span class="n">parsed_date</span><span class="p">)</span>
2009 <span class="n">reviews</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">review</span><span class="p">)</span>
2010
2011 <span class="nb">print</span><span class="p">(</span><span class="n">products</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
2012 <span class="nb">print</span><span class="p">(</span><span class="n">reviews</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
2013 </pre></div>
2014
2015 <p>After you run the code above, you should get some output like this:</p>
2016 <div class="highlight python"><pre><span></span><span class="n">Product</span><span class="p">(</span><span class="nb">id</span><span class="o">=</span><span class="s1">'B00FALQ1ZC'</span><span class="p">,</span> <span class="n">parent</span><span class="o">=</span><span class="mi">937001370</span><span class="p">,</span> <span class="o">...</span><span class="p">)</span>
2017 <span class="n">Review</span><span class="p">(</span><span class="nb">id</span><span class="o">=</span><span class="s1">'R3O9SGZBVQBV76'</span><span class="p">,</span> <span class="n">customer_id</span><span class="o">=</span><span class="mi">3653882</span><span class="p">,</span> <span class="o">...</span><span class="p">)</span>
2018 </pre></div>
2019
2020 <p>That’s it! Now you should have the data in a very simple and digestible class format, and you can start thinking of storing this in a <a href="https://realpython.com/tutorials/databases/">Database</a> or any other type of data storage you like.</p>
2021 <p>Using this kind of OOP strategy to parse spreadsheets makes handling the data much simpler later on.</p>
2022 <h3 id="appending-new-data">Appending New Data</h3>
2023 <p>Before you start creating very complex spreadsheets, have a quick look at an example of how to append data to an existing spreadsheet.</p>
2024 <p>Go back to the first example spreadsheet you created (<code>hello_world.xlsx</code>) and try opening it and appending some data to it, like this:</p>
2025 <div class="highlight python"><pre><span></span><span class="kn">from</span> <span class="nn">openpyxl</span> <span class="k">import</span> <span class="n">load_workbook</span>
2026
2027 <span class="c1"># Start by opening the spreadsheet and selecting the main sheet</span>
2028 <span class="n">workbook</span> <span class="o">=</span> <span class="n">load_workbook</span><span class="p">(</span><span class="n">filename</span><span class="o">=</span><span class="s2">"hello_world.xlsx"</span><span class="p">)</span>
2029 <span class="n">sheet</span> <span class="o">=</span> <span class="n">workbook</span><span class="o">.</span><span class="n">active</span>
2030
2031 <span class="c1"># Write what you want into a specific cell</span>
2032 <span class="n">sheet</span><span class="p">[</span><span class="s2">"C1"</span><span class="p">]</span> <span class="o">=</span> <span class="s2">"writing ;)"</span>
2033
2034 <span class="c1"># Save the spreadsheet</span>
2035 <span class="n">workbook</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="n">filename</span><span class="o">=</span><span class="s2">"hello_world_append.xlsx"</span>
2036 </pre></div>
2037
2038 <p><em>Et voilΓ </em>, if you open the new <code>hello_world_append.xlsx</code> spreadsheet, you’ll see the following change:</p>
2039 <p><a href="https://files.realpython.com/media/Screenshot_2019-06-24_17.44.22.e4f18e5abc42.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/Screenshot_2019-06-24_17.44.22.e4f18e5abc42.png" width="2160" height="1352" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_17.44.22.e4f18e5abc42.png&w=540&sig=098886279b90048004feb6dcdbe1c66ac3e231ce 540w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_17.44.22.e4f18e5abc42.png&w=1080&sig=8619e04c109779499f96dcd8aee01c4cf1ed52eb 1080w, https://files.realpython.com/media/Screenshot_2019-06-24_17.44.22.e4f18e5abc42.png 2160w" sizes="75vw" alt="Appending Data to a Spreadsheet"/></a></p>
2040 <p>Notice the additional <em>writing ;)</em> on cell <code>C1</code>.</p>
2041 <h2 id="writing-excel-spreadsheets-with-openpyxl">Writing Excel Spreadsheets With openpyxl</h2>
2042 <p>There are a lot of different things you can write to a spreadsheet, from simple text or number values to complex formulas, charts, or even images.</p>
2043 <p>Let’s start creating some spreadsheets!</p>
2044 <h3 id="creating-a-simple-spreadsheet">Creating a Simple Spreadsheet</h3>
2045 <p>Previously, you saw a very quick example of how to write “Hello world!” into a spreadsheet, so you can start with that:</p>
2046 <div class="highlight python"><pre><span></span><span class="lineno"> 1 </span><span class="kn">from</span> <span class="nn">openpyxl</span> <span class="k">import</span> <span class="n">Workbook</span>
2047 <span class="lineno"> 2 </span>
2048 <span class="lineno"> 3 </span><span class="n">filename</span> <span class="o">=</span> <span class="s2">"hello_world.xlsx"</span>
2049 <span class="lineno"> 4 </span>
2050 <span class="lineno"> 5 </span><span class="hll"><span class="n">workbook</span> <span class="o">=</span> <span class="n">Workbook</span><span class="p">()</span>
2051 </span><span class="lineno"> 6 </span><span class="n">sheet</span> <span class="o">=</span> <span class="n">workbook</span><span class="o">.</span><span class="n">active</span>
2052 <span class="lineno"> 7 </span>
2053 <span class="lineno"> 8 </span><span class="hll"><span class="n">sheet</span><span class="p">[</span><span class="s2">"A1"</span><span class="p">]</span> <span class="o">=</span> <span class="s2">"hello"</span>
2054 </span><span class="lineno"> 9 </span><span class="hll"><span class="n">sheet</span><span class="p">[</span><span class="s2">"B1"</span><span class="p">]</span> <span class="o">=</span> <span class="s2">"world!"</span>
2055 </span><span class="lineno">10 </span>
2056 <span class="lineno">11 </span><span class="hll"><span class="n">workbook</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="n">filename</span><span class="o">=</span><span class="n">filename</span><span class="p">)</span>
2057 </span></pre></div>
2058
2059 <p>The highlighted lines in the code above are the most important ones for writing. In the code, you can see that:</p>
2060 <ul>
2061 <li><strong>Line 5</strong> shows you how to create a new empty workbook.</li>
2062 <li><strong>Lines 8 and 9</strong> show you how to add data to specific cells.</li>
2063 <li><strong>Line 11</strong> shows you how to save the spreadsheet when you’re done.</li>
2064 </ul>
2065 <p>Even though these lines above can be straightforward, it’s still good to know them well for when things get a bit more complicated.</p>
2066 <div class="alert alert-primary" role="alert">
2067 <p><strong>Note:</strong> You’ll be using the <code>hello_world.xlsx</code> spreadsheet for some of the upcoming examples, so keep it handy.</p>
2068 </div>
2069 <p>One thing you can do to help with coming code examples is add the following method to your Python file or console:</p>
2070 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">def</span> <span class="nf">print_rows</span><span class="p">():</span>
2071 <span class="gp">... </span> <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">sheet</span><span class="o">.</span><span class="n">iter_rows</span><span class="p">(</span><span class="n">values_only</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
2072 <span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">row</span><span class="p">)</span>
2073 </pre></div>
2074
2075 <p>It makes it easier to print all of your spreadsheet values by just calling <code>print_rows()</code>.</p>
2076 <h3 id="basic-spreadsheet-operations">Basic Spreadsheet Operations</h3>
2077 <p>Before you get into the more advanced topics, it’s good for you to know how to manage the most simple elements of a spreadsheet.</p>
2078 <h4 id="adding-and-updating-cell-values">Adding and Updating Cell Values</h4>
2079 <p>You already learned how to add values to a spreadsheet like this:</p>
2080 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">sheet</span><span class="p">[</span><span class="s2">"A1"</span><span class="p">]</span> <span class="o">=</span> <span class="s2">"value"</span>
2081 </pre></div>
2082
2083 <p>There’s another way you can do this, by first selecting a cell and then changing its value:</p>
2084 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">cell</span> <span class="o">=</span> <span class="n">sheet</span><span class="p">[</span><span class="s2">"A1"</span><span class="p">]</span>
2085 <span class="gp">>>> </span><span class="n">cell</span>
2086 <span class="go"><Cell 'Sheet'.A1></span>
2087
2088 <span class="gp">>>> </span><span class="n">cell</span><span class="o">.</span><span class="n">value</span>
2089 <span class="go">'hello'</span>
2090
2091 <span class="gp">>>> </span><span class="n">cell</span><span class="o">.</span><span class="n">value</span> <span class="o">=</span> <span class="s2">"hey"</span>
2092 <span class="gp">>>> </span><span class="n">cell</span><span class="o">.</span><span class="n">value</span>
2093 <span class="go">'hey'</span>
2094 </pre></div>
2095
2096 <p>The new value is only stored into the spreadsheet once you call <code>workbook.save()</code>.</p>
2097 <p>The <code>openpyxl</code> creates a cell when adding a value, if that cell didn’t exist before:</p>
2098 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="c1"># Before, our spreadsheet has only 1 row</span>
2099 <span class="gp">>>> </span><span class="n">print_rows</span><span class="p">()</span>
2100 <span class="go">('hello', 'world!')</span>
2101
2102 <span class="gp">>>> </span><span class="c1"># Try adding a value to row 10</span>
2103 <span class="gp">>>> </span><span class="n">sheet</span><span class="p">[</span><span class="s2">"B10"</span><span class="p">]</span> <span class="o">=</span> <span class="s2">"test"</span>
2104 <span class="gp">>>> </span><span class="n">print_rows</span><span class="p">()</span>
2105 <span class="go">('hello', 'world!')</span>
2106 <span class="go">(None, None)</span>
2107 <span class="go">(None, None)</span>
2108 <span class="go">(None, None)</span>
2109 <span class="go">(None, None)</span>
2110 <span class="go">(None, None)</span>
2111 <span class="go">(None, None)</span>
2112 <span class="go">(None, None)</span>
2113 <span class="go">(None, None)</span>
2114 <span class="go">(None, 'test')</span>
2115 </pre></div>
2116
2117 <p>As you can see, when trying to add a value to cell <code>B10</code>, you end up with a tuple with 10 rows, just so you can have that <em>test</em> value.</p>
2118 <h4 id="managing-rows-and-columns">Managing Rows and Columns</h4>
2119 <p>One of the most common things you have to do when manipulating spreadsheets is adding or removing rows and columns. The <code>openpyxl</code> package allows you to do that in a very straightforward way by using the methods:</p>
2120 <ul>
2121 <li><code>.insert_rows()</code></li>
2122 <li><code>.delete_rows()</code></li>
2123 <li><code>.insert_cols()</code></li>
2124 <li><code>.delete_cols()</code></li>
2125 </ul>
2126 <p>Every single one of those methods can receive two arguments:</p>
2127 <ol>
2128 <li><code>idx</code></li>
2129 <li><code>amount</code></li>
2130 </ol>
2131 <p>Using our basic <code>hello_world.xlsx</code> example again, let’s see how these methods work:</p>
2132 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">print_rows</span><span class="p">()</span>
2133 <span class="go">('hello', 'world!')</span>
2134
2135 <span class="gp">>>> </span><span class="c1"># Insert a column before the existing column 1 ("A")</span>
2136 <span class="gp">>>> </span><span class="n">sheet</span><span class="o">.</span><span class="n">insert_cols</span><span class="p">(</span><span class="n">idx</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
2137 <span class="gp">>>> </span><span class="n">print_rows</span><span class="p">()</span>
2138 <span class="go">(None, 'hello', 'world!')</span>
2139
2140 <span class="gp">>>> </span><span class="c1"># Insert 5 columns between column 2 ("B") and 3 ("C")</span>
2141 <span class="gp">>>> </span><span class="n">sheet</span><span class="o">.</span><span class="n">insert_cols</span><span class="p">(</span><span class="n">idx</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span> <span class="n">amount</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span>
2142 <span class="gp">>>> </span><span class="n">print_rows</span><span class="p">()</span>
2143 <span class="go">(None, 'hello', None, None, None, None, None, 'world!')</span>
2144
2145 <span class="gp">>>> </span><span class="c1"># Delete the created columns</span>
2146 <span class="gp">>>> </span><span class="n">sheet</span><span class="o">.</span><span class="n">delete_cols</span><span class="p">(</span><span class="n">idx</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span> <span class="n">amount</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span>
2147 <span class="gp">>>> </span><span class="n">sheet</span><span class="o">.</span><span class="n">delete_cols</span><span class="p">(</span><span class="n">idx</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
2148 <span class="gp">>>> </span><span class="n">print_rows</span><span class="p">()</span>
2149 <span class="go">('hello', 'world!')</span>
2150
2151 <span class="gp">>>> </span><span class="c1"># Insert a new row in the beginning</span>
2152 <span class="gp">>>> </span><span class="n">sheet</span><span class="o">.</span><span class="n">insert_rows</span><span class="p">(</span><span class="n">idx</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
2153 <span class="gp">>>> </span><span class="n">print_rows</span><span class="p">()</span>
2154 <span class="go">(None, None)</span>
2155 <span class="go">('hello', 'world!')</span>
2156
2157 <span class="gp">>>> </span><span class="c1"># Insert 3 new rows in the beginning</span>
2158 <span class="gp">>>> </span><span class="n">sheet</span><span class="o">.</span><span class="n">insert_rows</span><span class="p">(</span><span class="n">idx</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">amount</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span>
2159 <span class="gp">>>> </span><span class="n">print_rows</span><span class="p">()</span>
2160 <span class="go">(None, None)</span>
2161 <span class="go">(None, None)</span>
2162 <span class="go">(None, None)</span>
2163 <span class="go">(None, None)</span>
2164 <span class="go">('hello', 'world!')</span>
2165
2166 <span class="gp">>>> </span><span class="c1"># Delete the first 4 rows</span>
2167 <span class="gp">>>> </span><span class="n">sheet</span><span class="o">.</span><span class="n">delete_rows</span><span class="p">(</span><span class="n">idx</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">amount</span><span class="o">=</span><span class="mi">4</span><span class="p">)</span>
2168 <span class="gp">>>> </span><span class="n">print_rows</span><span class="p">()</span>
2169 <span class="go">('hello', 'world!')</span>
2170 </pre></div>
2171
2172 <p>The only thing you need to remember is that when inserting new data (rows or columns), the insertion happens <strong>before</strong> the <code>idx</code> parameter.</p>
2173 <p>So, if you do <code>insert_rows(1)</code>, it inserts a new row <strong>before</strong> the existing first row.</p>
2174 <p>It’s the same for columns: when you call <code>insert_cols(2)</code>, it inserts a new column right <strong>before</strong> the already existing second column (<code>B</code>).</p>
2175 <p>However, when deleting rows or columns, <code>.delete_...</code> deletes data <strong>starting from</strong> the index passed as an argument.</p>
2176 <p>For example, when doing <code>delete_rows(2)</code> it deletes row <code>2</code>, and when doing <code>delete_cols(3)</code> it deletes the third column (<code>C</code>).</p>
2177 <h4 id="managing-sheets">Managing Sheets</h4>
2178 <p>Sheet management is also one of those things you might need to know, even though it might be something that you don’t use that often.</p>
2179 <p>If you look back at the code examples from this tutorial, you’ll notice the following recurring piece of code:</p>
2180 <div class="highlight python"><pre><span></span><span class="n">sheet</span> <span class="o">=</span> <span class="n">workbook</span><span class="o">.</span><span class="n">active</span>
2181 </pre></div>
2182
2183 <p>This is the way to select the default sheet from a spreadsheet. However, if you’re opening a spreadsheet with multiple sheets, then you can always select a specific one like this:</p>
2184 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="c1"># Let's say you have two sheets: "Products" and "Company Sales"</span>
2185 <span class="gp">>>> </span><span class="n">workbook</span><span class="o">.</span><span class="n">sheetnames</span>
2186 <span class="go">['Products', 'Company Sales']</span>
2187
2188 <span class="gp">>>> </span><span class="c1"># You can select a sheet using its title</span>
2189 <span class="gp">>>> </span><span class="n">products_sheet</span> <span class="o">=</span> <span class="n">workbook</span><span class="p">[</span><span class="s2">"Products"</span><span class="p">]</span>
2190 <span class="gp">>>> </span><span class="n">sales_sheet</span> <span class="o">=</span> <span class="n">workbook</span><span class="p">[</span><span class="s2">"Company Sales"</span><span class="p">]</span>
2191 </pre></div>
2192
2193 <p>You can also change a sheet title very easily:</p>
2194 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">workbook</span><span class="o">.</span><span class="n">sheetnames</span>
2195 <span class="go">['Products', 'Company Sales']</span>
2196
2197 <span class="gp">>>> </span><span class="n">products_sheet</span> <span class="o">=</span> <span class="n">workbook</span><span class="p">[</span><span class="s2">"Products"</span><span class="p">]</span>
2198 <span class="gp">>>> </span><span class="n">products_sheet</span><span class="o">.</span><span class="n">title</span> <span class="o">=</span> <span class="s2">"New Products"</span>
2199
2200 <span class="gp">>>> </span><span class="n">workbook</span><span class="o">.</span><span class="n">sheetnames</span>
2201 <span class="go">['New Products', 'Company Sales']</span>
2202 </pre></div>
2203
2204 <p>If you want to create or delete sheets, then you can also do that with <code>.create_sheet()</code> and <code>.remove()</code>:</p>
2205 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">workbook</span><span class="o">.</span><span class="n">sheetnames</span>
2206 <span class="go">['Products', 'Company Sales']</span>
2207
2208 <span class="gp">>>> </span><span class="n">operations_sheet</span> <span class="o">=</span> <span class="n">workbook</span><span class="o">.</span><span class="n">create_sheet</span><span class="p">(</span><span class="s2">"Operations"</span><span class="p">)</span>
2209 <span class="gp">>>> </span><span class="n">workbook</span><span class="o">.</span><span class="n">sheetnames</span>
2210 <span class="go">['Products', 'Company Sales', 'Operations']</span>
2211
2212 <span class="gp">>>> </span><span class="c1"># You can also define the position to create the sheet at</span>
2213 <span class="gp">>>> </span><span class="n">hr_sheet</span> <span class="o">=</span> <span class="n">workbook</span><span class="o">.</span><span class="n">create_sheet</span><span class="p">(</span><span class="s2">"HR"</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span>
2214 <span class="gp">>>> </span><span class="n">workbook</span><span class="o">.</span><span class="n">sheetnames</span>
2215 <span class="go">['HR', 'Products', 'Company Sales', 'Operations']</span>
2216
2217 <span class="gp">>>> </span><span class="c1"># To remove them, just pass the sheet as an argument to the .remove()</span>
2218 <span class="gp">>>> </span><span class="n">workbook</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="n">operations_sheet</span><span class="p">)</span>
2219 <span class="gp">>>> </span><span class="n">workbook</span><span class="o">.</span><span class="n">sheetnames</span>
2220 <span class="go">['HR', 'Products', 'Company Sales']</span>
2221
2222 <span class="gp">>>> </span><span class="n">workbook</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="n">hr_sheet</span><span class="p">)</span>
2223 <span class="gp">>>> </span><span class="n">workbook</span><span class="o">.</span><span class="n">sheetnames</span>
2224 <span class="go">['Products', 'Company Sales']</span>
2225 </pre></div>
2226
2227 <p>One other thing you can do is make duplicates of a sheet using <code>copy_worksheet()</code>:</p>
2228 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">workbook</span><span class="o">.</span><span class="n">sheetnames</span>
2229 <span class="go">['Products', 'Company Sales']</span>
2230
2231 <span class="gp">>>> </span><span class="n">products_sheet</span> <span class="o">=</span> <span class="n">workbook</span><span class="p">[</span><span class="s2">"Products"</span><span class="p">]</span>
2232 <span class="gp">>>> </span><span class="n">workbook</span><span class="o">.</span><span class="n">copy_worksheet</span><span class="p">(</span><span class="n">products_sheet</span><span class="p">)</span>
2233 <span class="go"><Worksheet "Products Copy"></span>
2234
2235 <span class="gp">>>> </span><span class="n">workbook</span><span class="o">.</span><span class="n">sheetnames</span>
2236 <span class="go">['Products', 'Company Sales', 'Products Copy']</span>
2237 </pre></div>
2238
2239 <p>If you open your spreadsheet after saving the above code, you’ll notice that the sheet <em>Products Copy</em> is a duplicate of the sheet <em>Products</em>.</p>
2240 <h4 id="freezing-rows-and-columns">Freezing Rows and Columns</h4>
2241 <p>Something that you might want to do when working with big spreadsheets is to freeze a few rows or columns, so they remain visible when you scroll right or down.</p>
2242 <p>Freezing data allows you to keep an eye on important rows or columns, regardless of where you scroll in the spreadsheet.</p>
2243 <p>Again, <code>openpyxl</code> also has a way to accomplish this by using the worksheet <code>freeze_panes</code> attribute. For this example, go back to our <code>sample.xlsx</code> spreadsheet and try doing the following:</p>
2244 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">workbook</span> <span class="o">=</span> <span class="n">load_workbook</span><span class="p">(</span><span class="n">filename</span><span class="o">=</span><span class="s2">"sample.xlsx"</span><span class="p">)</span>
2245 <span class="gp">>>> </span><span class="n">sheet</span> <span class="o">=</span> <span class="n">workbook</span><span class="o">.</span><span class="n">active</span>
2246 <span class="gp">>>> </span><span class="n">sheet</span><span class="o">.</span><span class="n">freeze_panes</span> <span class="o">=</span> <span class="s2">"C2"</span>
2247 <span class="gp">>>> </span><span class="n">workbook</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="s2">"sample_frozen.xlsx"</span><span class="p">)</span>
2248 </pre></div>
2249
2250 <p>If you open the <code>sample_frozen.xlsx</code> spreadsheet in your favorite spreadsheet editor, you’ll notice that row <code>1</code> and columns <code>A</code> and <code>B</code> are frozen and are always visible no matter where you navigate within the spreadsheet.</p>
2251 <p>This feature is handy, for example, to keep headers within sight, so you always know what each column represents.</p>
2252 <p>Here’s how it looks in the editor:</p>
2253 <p><a href="https://files.realpython.com/media/Screenshot_2019-06-24_18.12.20.55694a0781f8.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/Screenshot_2019-06-24_18.12.20.55694a0781f8.png" width="2160" height="1352" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_18.12.20.55694a0781f8.png&w=540&sig=5826de23e5df2e08d625844698fc3a29b32ee7b2 540w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_18.12.20.55694a0781f8.png&w=1080&sig=c3abe2321f00372d975bbbf033f3ebe3687eb09f 1080w, https://files.realpython.com/media/Screenshot_2019-06-24_18.12.20.55694a0781f8.png 2160w" sizes="75vw" alt="Example Spreadsheet With Frozen Rows and Columns"/></a></p>
2254 <p>Notice how you’re at the end of the spreadsheet, and yet, you can see both row <code>1</code> and columns <code>A</code> and <code>B</code>.</p>
2255 <h4 id="adding-filters">Adding Filters</h4>
2256 <p>You can use <code>openpyxl</code> to add filters and sorts to your spreadsheet. However, when you open the spreadsheet, the data won’t be rearranged according to these sorts and filters.</p>
2257 <p>At first, this might seem like a pretty useless feature, but when you’re programmatically creating a spreadsheet that is going to be sent and used by somebody else, it’s still nice to at least create the filters and allow people to use it afterward.</p>
2258 <p>The code below is an example of how you would add some filters to our existing <code>sample.xlsx</code> spreadsheet:</p>
2259 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="c1"># Check the used spreadsheet space using the attribute "dimensions"</span>
2260 <span class="gp">>>> </span><span class="n">sheet</span><span class="o">.</span><span class="n">dimensions</span>
2261 <span class="go">'A1:O100'</span>
2262
2263 <span class="gp">>>> </span><span class="n">sheet</span><span class="o">.</span><span class="n">auto_filter</span><span class="o">.</span><span class="n">ref</span> <span class="o">=</span> <span class="s2">"A1:O100"</span>
2264 <span class="gp">>>> </span><span class="n">workbook</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="n">filename</span><span class="o">=</span><span class="s2">"sample_with_filters.xlsx"</span><span class="p">)</span>
2265 </pre></div>
2266
2267 <p>You should now see the filters created when opening the spreadsheet in your editor:</p>
2268 <p><a href="https://files.realpython.com/media/Screenshot_2019-06-24_18.20.35.5fdbfe805194.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/Screenshot_2019-06-24_18.20.35.5fdbfe805194.png" width="2160" height="1352" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_18.20.35.5fdbfe805194.png&w=540&sig=c1d7ad4f2dfc03fc8730e3babf9000ac74170c7d 540w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_18.20.35.5fdbfe805194.png&w=1080&sig=27e888a967ddc112f1e824be671a15e2c111fe6c 1080w, https://files.realpython.com/media/Screenshot_2019-06-24_18.20.35.5fdbfe805194.png 2160w" sizes="75vw" alt="Example Spreadsheet With Filters"/></a></p>
2269 <p>You don’t have to use <code>sheet.dimensions</code> if you know precisely which part of the spreadsheet you want to apply filters to.</p>
2270 <h3 id="adding-formulas">Adding Formulas</h3>
2271 <p><strong>Formulas</strong> (or <strong>formulae</strong>) are one of the most powerful features of spreadsheets.</p>
2272 <p>They gives you the power to apply specific mathematical equations to a range of cells. Using formulas with <code>openpyxl</code> is as simple as editing the value of a cell.</p>
2273 <p>You can see the list of formulas supported by <code>openpyxl</code>:</p>
2274 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">openpyxl.utils</span> <span class="k">import</span> <span class="n">FORMULAE</span>
2275 <span class="gp">>>> </span><span class="n">FORMULAE</span>
2276 <span class="go">frozenset({'ABS',</span>
2277 <span class="go"> 'ACCRINT',</span>
2278 <span class="go"> 'ACCRINTM',</span>
2279 <span class="go"> 'ACOS',</span>
2280 <span class="go"> 'ACOSH',</span>
2281 <span class="go"> 'AMORDEGRC',</span>
2282 <span class="go"> 'AMORLINC',</span>
2283 <span class="go"> 'AND',</span>
2284 <span class="go"> ...</span>
2285 <span class="go"> 'YEARFRAC',</span>
2286 <span class="go"> 'YIELD',</span>
2287 <span class="go"> 'YIELDDISC',</span>
2288 <span class="go"> 'YIELDMAT',</span>
2289 <span class="go"> 'ZTEST'})</span>
2290 </pre></div>
2291
2292 <p>Let’s add some formulas to our <code>sample.xlsx</code> spreadsheet.</p>
2293 <p>Starting with something easy, let’s check the average star rating for the 99 reviews within the spreadsheet:</p>
2294 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="c1"># Star rating is column "H"</span>
2295 <span class="gp">>>> </span><span class="n">sheet</span><span class="p">[</span><span class="s2">"P2"</span><span class="p">]</span> <span class="o">=</span> <span class="s2">"=AVERAGE(H2:H100)"</span>
2296 <span class="gp">>>> </span><span class="n">workbook</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="n">filename</span><span class="o">=</span><span class="s2">"sample_formulas.xlsx"</span><span class="p">)</span>
2297 </pre></div>
2298
2299 <p>If you open the spreadsheet now and go to cell <code>P2</code>, you should see that its value is: <em>4.18181818181818</em>. Have a look in the editor:</p>
2300 <p><a href="https://files.realpython.com/media/Screenshot_2019-06-24_18.33.09.7c2633f706cc.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/Screenshot_2019-06-24_18.33.09.7c2633f706cc.png" width="2160" height="1352" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_18.33.09.7c2633f706cc.png&w=540&sig=5d7a9eb97acf524d5d2b9b93ae0e9214bbcf95c8 540w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_18.33.09.7c2633f706cc.png&w=1080&sig=8af67321cb101fb2f30fd0ca2bdcc62de35c9334 1080w, https://files.realpython.com/media/Screenshot_2019-06-24_18.33.09.7c2633f706cc.png 2160w" sizes="75vw" alt="Example Spreadsheet With Average Formula"/></a></p>
2301 <p>You can use the same methodology to add any formulas to your spreadsheet. For example, let’s count the number of reviews that had helpful votes:</p>
2302 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="c1"># The helpful votes are counted on column "I"</span>
2303 <span class="gp">>>> </span><span class="n">sheet</span><span class="p">[</span><span class="s2">"P3"</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'=COUNTIF(I2:I100, ">0")'</span>
2304 <span class="gp">>>> </span><span class="n">workbook</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="n">filename</span><span class="o">=</span><span class="s2">"sample_formulas.xlsx"</span><span class="p">)</span>
2305 </pre></div>
2306
2307 <p>You should get the number <code>21</code> on your <code>P3</code> spreadsheet cell like so:</p>
2308 <p><a href="https://files.realpython.com/media/Screenshot_2019-06-24_18.35.24.e26e97b0c9c0.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/Screenshot_2019-06-24_18.35.24.e26e97b0c9c0.png" width="2160" height="1352" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_18.35.24.e26e97b0c9c0.png&w=540&sig=0ec4c4c12a792a1a393e0273855282bfa0594d53 540w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_18.35.24.e26e97b0c9c0.png&w=1080&sig=67b399b8cb79ddbe7285da0325fe8f6b9edf3ecc 1080w, https://files.realpython.com/media/Screenshot_2019-06-24_18.35.24.e26e97b0c9c0.png 2160w" sizes="75vw" alt="Example Spreadsheet With Average and CountIf Formula"/></a></p>
2309 <p>You’ll have to make sure that the strings within a formula are always in double quotes, so you either have to use single quotes around the formula like in the example above or you’ll have to escape the double quotes inside the formula: <code>"=COUNTIF(I2:I100, \">0\")"</code>.</p>
2310 <p>There are a ton of other formulas you can add to your spreadsheet using the same procedure you tried above. Give it a go yourself!</p>
2311 <h3 id="adding-styles">Adding Styles</h3>
2312 <p>Even though styling a spreadsheet might not be something you would do every day, it’s still good to know how to do it.</p>
2313 <p>Using <code>openpyxl</code>, you can apply multiple styling options to your spreadsheet, including fonts, borders, colors, and so on. Have a look at the <code>openpyxl</code> <a href="https://openpyxl.readthedocs.io/en/stable/styles.html">documentation</a> to learn more.</p>
2314 <p>You can also choose to either apply a style directly to a cell or create a template and reuse it to apply styles to multiple cells.</p>
2315 <p>Let’s start by having a look at simple cell styling, using our <code>sample.xlsx</code> again as the base spreadsheet:</p>
2316 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="c1"># Import necessary style classes</span>
2317 <span class="gp">>>> </span><span class="kn">from</span> <span class="nn">openpyxl.styles</span> <span class="k">import</span> <span class="n">Font</span><span class="p">,</span> <span class="n">Color</span><span class="p">,</span> <span class="n">Alignment</span><span class="p">,</span> <span class="n">Border</span><span class="p">,</span> <span class="n">Side</span><span class="p">,</span> <span class="n">colors</span>
2318
2319 <span class="gp">>>> </span><span class="c1"># Create a few styles</span>
2320 <span class="gp">>>> </span><span class="n">bold_font</span> <span class="o">=</span> <span class="n">Font</span><span class="p">(</span><span class="n">bold</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
2321 <span class="gp">>>> </span><span class="n">big_red_text</span> <span class="o">=</span> <span class="n">Font</span><span class="p">(</span><span class="n">color</span><span class="o">=</span><span class="n">colors</span><span class="o">.</span><span class="n">RED</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="mi">20</span><span class="p">)</span>
2322 <span class="gp">>>> </span><span class="n">center_aligned_text</span> <span class="o">=</span> <span class="n">Alignment</span><span class="p">(</span><span class="n">horizontal</span><span class="o">=</span><span class="s2">"center"</span><span class="p">)</span>
2323 <span class="gp">>>> </span><span class="n">double_border_side</span> <span class="o">=</span> <span class="n">Side</span><span class="p">(</span><span class="n">border_style</span><span class="o">=</span><span class="s2">"double"</span><span class="p">)</span>
2324 <span class="gp">>>> </span><span class="n">square_border</span> <span class="o">=</span> <span class="n">Border</span><span class="p">(</span><span class="n">top</span><span class="o">=</span><span class="n">double_border_side</span><span class="p">,</span>
2325 <span class="gp">... </span> <span class="n">right</span><span class="o">=</span><span class="n">double_border_side</span><span class="p">,</span>
2326 <span class="gp">... </span> <span class="n">bottom</span><span class="o">=</span><span class="n">double_border_side</span><span class="p">,</span>
2327 <span class="gp">... </span> <span class="n">left</span><span class="o">=</span><span class="n">double_border_side</span><span class="p">)</span>
2328
2329 <span class="gp">>>> </span><span class="c1"># Style some cells!</span>
2330 <span class="gp">>>> </span><span class="n">sheet</span><span class="p">[</span><span class="s2">"A2"</span><span class="p">]</span><span class="o">.</span><span class="n">font</span> <span class="o">=</span> <span class="n">bold_font</span>
2331 <span class="gp">>>> </span><span class="n">sheet</span><span class="p">[</span><span class="s2">"A3"</span><span class="p">]</span><span class="o">.</span><span class="n">font</span> <span class="o">=</span> <span class="n">big_red_text</span>
2332 <span class="gp">>>> </span><span class="n">sheet</span><span class="p">[</span><span class="s2">"A4"</span><span class="p">]</span><span class="o">.</span><span class="n">alignment</span> <span class="o">=</span> <span class="n">center_aligned_text</span>
2333 <span class="gp">>>> </span><span class="n">sheet</span><span class="p">[</span><span class="s2">"A5"</span><span class="p">]</span><span class="o">.</span><span class="n">border</span> <span class="o">=</span> <span class="n">square_border</span>
2334 <span class="gp">>>> </span><span class="n">workbook</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="n">filename</span><span class="o">=</span><span class="s2">"sample_styles.xlsx"</span><span class="p">)</span>
2335 </pre></div>
2336
2337 <p>If you open your spreadsheet now, you should see quite a few different styles on the first 5 cells of column <code>A</code>:</p>
2338 <p><a href="https://files.realpython.com/media/Screenshot_2019-06-24_18.43.15.e3aeb3fb06e3.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/Screenshot_2019-06-24_18.43.15.e3aeb3fb06e3.png" width="2160" height="1352" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_18.43.15.e3aeb3fb06e3.png&w=540&sig=ecc21878006697a6135ae515442642a95ab2bfb6 540w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_18.43.15.e3aeb3fb06e3.png&w=1080&sig=6f0ef4a148f1ca5a588e0cb2c02b0c9aad4246f2 1080w, https://files.realpython.com/media/Screenshot_2019-06-24_18.43.15.e3aeb3fb06e3.png 2160w" sizes="75vw" alt="Example Spreadsheet With Simple Cell Styles"/></a></p>
2339 <p>There you go. You got:</p>
2340 <ul>
2341 <li><strong>A2</strong> with the text in bold</li>
2342 <li><strong>A3</strong> with the text in red and bigger font size</li>
2343 <li><strong>A4</strong> with the text centered</li>
2344 <li><strong>A5</strong> with a square border around the text</li>
2345 </ul>
2346 <div class="alert alert-primary" role="alert">
2347 <p><strong>Note:</strong> For the colors, you can also use HEX codes instead by doing <code>Font(color="C70E0F")</code>.</p>
2348 </div>
2349 <p>You can also combine styles by simply adding them to the cell at the same time:</p>
2350 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="c1"># Reusing the same styles from the example above</span>
2351 <span class="gp">>>> </span><span class="n">sheet</span><span class="p">[</span><span class="s2">"A6"</span><span class="p">]</span><span class="o">.</span><span class="n">alignment</span> <span class="o">=</span> <span class="n">center_aligned_text</span>
2352 <span class="gp">>>> </span><span class="n">sheet</span><span class="p">[</span><span class="s2">"A6"</span><span class="p">]</span><span class="o">.</span><span class="n">font</span> <span class="o">=</span> <span class="n">big_red_text</span>
2353 <span class="gp">>>> </span><span class="n">sheet</span><span class="p">[</span><span class="s2">"A6"</span><span class="p">]</span><span class="o">.</span><span class="n">border</span> <span class="o">=</span> <span class="n">square_border</span>
2354 <span class="gp">>>> </span><span class="n">workbook</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="n">filename</span><span class="o">=</span><span class="s2">"sample_styles.xlsx"</span><span class="p">)</span>
2355 </pre></div>
2356
2357 <p>Have a look at cell <code>A6</code> here:</p>
2358 <p><a href="https://files.realpython.com/media/Screenshot_2019-06-24_18.46.04.314517930065.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/Screenshot_2019-06-24_18.46.04.314517930065.png" width="2160" height="1352" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_18.46.04.314517930065.png&w=540&sig=290bbf523eb24ac8c9741daf86701ca57cad4b96 540w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_18.46.04.314517930065.png&w=1080&sig=9decedef154c2138e26b287f2186213142650f6e 1080w, https://files.realpython.com/media/Screenshot_2019-06-24_18.46.04.314517930065.png 2160w" sizes="75vw" alt="Example Spreadsheet With Coupled Cell Styles"/></a></p>
2359 <p>When you want to apply multiple styles to one or several cells, you can use a <code>NamedStyle</code> class instead, which is like a style template that you can use over and over again. Have a look at the example below:</p>
2360 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">openpyxl.styles</span> <span class="k">import</span> <span class="n">NamedStyle</span>
2361
2362 <span class="gp">>>> </span><span class="c1"># Let's create a style template for the header row</span>
2363 <span class="gp">>>> </span><span class="n">header</span> <span class="o">=</span> <span class="n">NamedStyle</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s2">"header"</span><span class="p">)</span>
2364 <span class="gp">>>> </span><span class="n">header</span><span class="o">.</span><span class="n">font</span> <span class="o">=</span> <span class="n">Font</span><span class="p">(</span><span class="n">bold</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
2365 <span class="gp">>>> </span><span class="n">header</span><span class="o">.</span><span class="n">border</span> <span class="o">=</span> <span class="n">Border</span><span class="p">(</span><span class="n">bottom</span><span class="o">=</span><span class="n">Side</span><span class="p">(</span><span class="n">border_style</span><span class="o">=</span><span class="s2">"thin"</span><span class="p">))</span>
2366 <span class="gp">>>> </span><span class="n">header</span><span class="o">.</span><span class="n">alignment</span> <span class="o">=</span> <span class="n">Alignment</span><span class="p">(</span><span class="n">horizontal</span><span class="o">=</span><span class="s2">"center"</span><span class="p">,</span> <span class="n">vertical</span><span class="o">=</span><span class="s2">"center"</span><span class="p">)</span>
2367
2368 <span class="gp">>>> </span><span class="c1"># Now let's apply this to all first row (header) cells</span>
2369 <span class="gp">>>> </span><span class="n">header_row</span> <span class="o">=</span> <span class="n">sheet</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
2370 <span class="gp">>>> </span><span class="k">for</span> <span class="n">cell</span> <span class="ow">in</span> <span class="n">header_row</span><span class="p">:</span>
2371 <span class="gp">... </span> <span class="n">cell</span><span class="o">.</span><span class="n">style</span> <span class="o">=</span> <span class="n">header</span>
2372
2373 <span class="gp">>>> </span><span class="n">workbook</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="n">filename</span><span class="o">=</span><span class="s2">"sample_styles.xlsx"</span><span class="p">)</span>
2374 </pre></div>
2375
2376 <p>If you open the spreadsheet now, you should see that its first row is bold, the text is aligned to the center, and there’s a small bottom border! Have a look below:</p>
2377 <p><a href="https://files.realpython.com/media/Screenshot_2019-06-24_18.48.33.4bc57d1b24d5.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/Screenshot_2019-06-24_18.48.33.4bc57d1b24d5.png" width="2160" height="1352" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_18.48.33.4bc57d1b24d5.png&w=540&sig=199107a0c9ea60fbf1dfcc078a7680b43faeef3a 540w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_18.48.33.4bc57d1b24d5.png&w=1080&sig=af3e5225e36a24dea088e8c175da02050bb5dda9 1080w, https://files.realpython.com/media/Screenshot_2019-06-24_18.48.33.4bc57d1b24d5.png 2160w" sizes="75vw" alt="Example Spreadsheet With Named Styles"/></a></p>
2378 <p>As you saw above, there are many options when it comes to styling, and it depends on the use case, so feel free to check <code>openpyxl</code> <a href="https://openpyxl.readthedocs.io/en/stable/styles.html">documentation</a> and see what other things you can do.</p>
2379 <h3 id="conditional-formatting">Conditional Formatting</h3>
2380 <p>This feature is one of my personal favorites when it comes to adding styles to a spreadsheet.</p>
2381 <p>It’s a much more powerful approach to styling because it dynamically applies styles according to how the data in the spreadsheet changes.</p>
2382 <p>In a nutshell, <strong>conditional formatting</strong> allows you to specify a list of styles to apply to a cell (or cell range) according to specific conditions.</p>
2383 <p>For example, a widespread use case is to have a balance sheet where all the negative totals are in red, and the positive ones are in green. This formatting makes it much more efficient to spot good vs bad periods.</p>
2384 <p>Without further ado, let’s pick our favorite spreadsheet—<code>sample.xlsx</code>—and add some conditional formatting.</p>
2385 <p>You can start by adding a simple one that adds a red background to all reviews with less than 3 stars:</p>
2386 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">openpyxl.styles</span> <span class="k">import</span> <span class="n">PatternFill</span><span class="p">,</span> <span class="n">colors</span>
2387 <span class="gp">>>> </span><span class="kn">from</span> <span class="nn">openpyxl.styles.differential</span> <span class="k">import</span> <span class="n">DifferentialStyle</span>
2388 <span class="gp">>>> </span><span class="kn">from</span> <span class="nn">openpyxl.formatting.rule</span> <span class="k">import</span> <span class="n">Rule</span>
2389
2390 <span class="gp">>>> </span><span class="n">red_background</span> <span class="o">=</span> <span class="n">PatternFill</span><span class="p">(</span><span class="n">bgColor</span><span class="o">=</span><span class="n">colors</span><span class="o">.</span><span class="n">RED</span><span class="p">)</span>
2391 <span class="gp">>>> </span><span class="n">diff_style</span> <span class="o">=</span> <span class="n">DifferentialStyle</span><span class="p">(</span><span class="n">fill</span><span class="o">=</span><span class="n">red_background</span><span class="p">)</span>
2392 <span class="gp">>>> </span><span class="n">rule</span> <span class="o">=</span> <span class="n">Rule</span><span class="p">(</span><span class="nb">type</span><span class="o">=</span><span class="s2">"expression"</span><span class="p">,</span> <span class="n">dxf</span><span class="o">=</span><span class="n">diff_style</span><span class="p">)</span>
2393 <span class="gp">>>> </span><span class="n">rule</span><span class="o">.</span><span class="n">formula</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"$H1<3"</span><span class="p">]</span>
2394 <span class="gp">>>> </span><span class="n">sheet</span><span class="o">.</span><span class="n">conditional_formatting</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="s2">"A1:O100"</span><span class="p">,</span> <span class="n">rule</span><span class="p">)</span>
2395 <span class="gp">>>> </span><span class="n">workbook</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="s2">"sample_conditional_formatting.xlsx"</span><span class="p">)</span>
2396 </pre></div>
2397
2398 <p>Now you’ll see all the reviews with a star rating below 3 marked with a red background:</p>
2399 <p><a href="https://files.realpython.com/media/Screenshot_2019-06-24_18.55.41.17f234a186c6.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/Screenshot_2019-06-24_18.55.41.17f234a186c6.png" width="2160" height="1352" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_18.55.41.17f234a186c6.png&w=540&sig=f3c141c2fe708c031c32c083cb038a736fd8da87 540w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_18.55.41.17f234a186c6.png&w=1080&sig=9ded3045cee09d34cd6f5dada7a4eea669ac4808 1080w, https://files.realpython.com/media/Screenshot_2019-06-24_18.55.41.17f234a186c6.png 2160w" sizes="75vw" alt="Example Spreadsheet With Simple Conditional Formatting"/></a></p>
2400 <p>Code-wise, the only things that are new here are the objects <code>DifferentialStyle</code> and <code>Rule</code>:</p>
2401 <ul>
2402 <li><strong><code>DifferentialStyle</code></strong> is quite similar to <code>NamedStyle</code>, which you already saw above, and it’s used to aggregate multiple styles such as fonts, borders, alignment, and so forth.</li>
2403 <li><strong><code>Rule</code></strong> is responsible for selecting the cells and applying the styles if the cells match the rule’s logic.</li>
2404 </ul>
2405 <p>Using a <code>Rule</code> object, you can create numerous conditional formatting scenarios.</p>
2406 <p>However, for simplicity sake, the <code>openpyxl</code> package offers 3 built-in formats that make it easier to create a few common conditional formatting patterns. These built-ins are:</p>
2407 <ul>
2408 <li><code>ColorScale</code></li>
2409 <li><code>IconSet</code></li>
2410 <li><code>DataBar</code></li>
2411 </ul>
2412 <p>The <strong>ColorScale</strong> gives you the ability to create color gradients:</p>
2413 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">openpyxl.formatting.rule</span> <span class="k">import</span> <span class="n">ColorScaleRule</span>
2414 <span class="gp">>>> </span><span class="n">color_scale_rule</span> <span class="o">=</span> <span class="n">ColorScaleRule</span><span class="p">(</span><span class="n">start_type</span><span class="o">=</span><span class="s2">"min"</span><span class="p">,</span>
2415 <span class="gp">... </span> <span class="n">start_color</span><span class="o">=</span><span class="n">colors</span><span class="o">.</span><span class="n">RED</span><span class="p">,</span>
2416 <span class="gp">... </span> <span class="n">end_type</span><span class="o">=</span><span class="s2">"max"</span><span class="p">,</span>
2417 <span class="gp">... </span> <span class="n">end_color</span><span class="o">=</span><span class="n">colors</span><span class="o">.</span><span class="n">GREEN</span><span class="p">)</span>
2418
2419 <span class="gp">>>> </span><span class="c1"># Again, let's add this gradient to the star ratings, column "H"</span>
2420 <span class="gp">>>> </span><span class="n">sheet</span><span class="o">.</span><span class="n">conditional_formatting</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="s2">"H2:H100"</span><span class="p">,</span> <span class="n">color_scale_rule</span><span class="p">)</span>
2421 <span class="gp">>>> </span><span class="n">workbook</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="n">filename</span><span class="o">=</span><span class="s2">"sample_conditional_formatting_color_scale.xlsx"</span><span class="p">)</span>
2422 </pre></div>
2423
2424 <p>Now you should see a color gradient on column <code>H</code>, from red to green, according to the star rating:</p>
2425 <p><a href="https://files.realpython.com/media/Screenshot_2019-06-24_19.00.57.26756963c1e9.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/Screenshot_2019-06-24_19.00.57.26756963c1e9.png" width="2160" height="1352" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_19.00.57.26756963c1e9.png&w=540&sig=782964d150a8adc1de811fab78b0accde6357f85 540w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_19.00.57.26756963c1e9.png&w=1080&sig=4c7c87c194ec0a8eb3a9e73fdabf3ead991e96ea 1080w, https://files.realpython.com/media/Screenshot_2019-06-24_19.00.57.26756963c1e9.png 2160w" sizes="75vw" alt="Example Spreadsheet With Color Scale Conditional Formatting"/></a></p>
2426 <p>You can also add a third color and make two gradients instead:</p>
2427 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">openpyxl.formatting.rule</span> <span class="k">import</span> <span class="n">ColorScaleRule</span>
2428 <span class="gp">>>> </span><span class="n">color_scale_rule</span> <span class="o">=</span> <span class="n">ColorScaleRule</span><span class="p">(</span><span class="n">start_type</span><span class="o">=</span><span class="s2">"num"</span><span class="p">,</span>
2429 <span class="gp">... </span> <span class="n">start_value</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
2430 <span class="gp">... </span> <span class="n">start_color</span><span class="o">=</span><span class="n">colors</span><span class="o">.</span><span class="n">RED</span><span class="p">,</span>
2431 <span class="gp">... </span> <span class="n">mid_type</span><span class="o">=</span><span class="s2">"num"</span><span class="p">,</span>
2432 <span class="gp">... </span> <span class="n">mid_value</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span>
2433 <span class="gp">... </span> <span class="n">mid_color</span><span class="o">=</span><span class="n">colors</span><span class="o">.</span><span class="n">YELLOW</span><span class="p">,</span>
2434 <span class="gp">... </span> <span class="n">end_type</span><span class="o">=</span><span class="s2">"num"</span><span class="p">,</span>
2435 <span class="gp">... </span> <span class="n">end_value</span><span class="o">=</span><span class="mi">5</span><span class="p">,</span>
2436 <span class="gp">... </span> <span class="n">end_color</span><span class="o">=</span><span class="n">colors</span><span class="o">.</span><span class="n">GREEN</span><span class="p">)</span>
2437
2438 <span class="gp">>>> </span><span class="c1"># Again, let's add this gradient to the star ratings, column "H"</span>
2439 <span class="gp">>>> </span><span class="n">sheet</span><span class="o">.</span><span class="n">conditional_formatting</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="s2">"H2:H100"</span><span class="p">,</span> <span class="n">color_scale_rule</span><span class="p">)</span>
2440 <span class="gp">>>> </span><span class="n">workbook</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="n">filename</span><span class="o">=</span><span class="s2">"sample_conditional_formatting_color_scale_3.xlsx"</span><span class="p">)</span>
2441 </pre></div>
2442
2443 <p>This time, you’ll notice that star ratings between 1 and 3 have a gradient from red to yellow, and star ratings between 3 and 5 have a gradient from yellow to green:</p>
2444 <p><a href="https://files.realpython.com/media/Screenshot_2019-06-24_19.03.30.0de9a2ff9866.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/Screenshot_2019-06-24_19.03.30.0de9a2ff9866.png" width="2160" height="1352" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_19.03.30.0de9a2ff9866.png&w=540&sig=17daddc356c8ff5c78497b200fe57ab69f80617d 540w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_19.03.30.0de9a2ff9866.png&w=1080&sig=6dcb18f4aa3f8ae0a395ca1e3581f8d4c59805ad 1080w, https://files.realpython.com/media/Screenshot_2019-06-24_19.03.30.0de9a2ff9866.png 2160w" sizes="75vw" alt="Example Spreadsheet With 2 Color Scales Conditional Formatting"/></a></p>
2445 <p>The <strong>IconSet</strong> allows you to add an icon to the cell according to its value:</p>
2446 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">openpyxl.formatting.rule</span> <span class="k">import</span> <span class="n">IconSetRule</span>
2447
2448 <span class="gp">>>> </span><span class="n">icon_set_rule</span> <span class="o">=</span> <span class="n">IconSetRule</span><span class="p">(</span><span class="s2">"5Arrows"</span><span class="p">,</span> <span class="s2">"num"</span><span class="p">,</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">])</span>
2449 <span class="gp">>>> </span><span class="n">sheet</span><span class="o">.</span><span class="n">conditional_formatting</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="s2">"H2:H100"</span><span class="p">,</span> <span class="n">icon_set_rule</span><span class="p">)</span>
2450 <span class="gp">>>> </span><span class="n">workbook</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="s2">"sample_conditional_formatting_icon_set.xlsx"</span><span class="p">)</span>
2451 </pre></div>
2452
2453 <p>You’ll see a colored arrow next to the star rating. This arrow is red and points down when the value of the cell is 1 and, as the rating gets better, the arrow starts pointing up and becomes green:</p>
2454 <p><a href="https://files.realpython.com/media/Screenshot_2019-06-24_19.07.29.23e75ff46771.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/Screenshot_2019-06-24_19.07.29.23e75ff46771.png" width="2160" height="1352" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_19.07.29.23e75ff46771.png&w=540&sig=388fc68ff53fa2e2d3d5678acfd41f10fa8eccde 540w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_19.07.29.23e75ff46771.png&w=1080&sig=e33f483f9758189782bb619a6714c948130375aa 1080w, https://files.realpython.com/media/Screenshot_2019-06-24_19.07.29.23e75ff46771.png 2160w" sizes="75vw" alt="Example Spreadsheet With Icon Set Conditional Formatting"/></a></p>
2455 <p>The <code>openpyxl</code> package has a <a href="https://openpyxl.readthedocs.io/en/stable/formatting.html#iconset">full list</a> of other icons you can use, besides the arrow.</p>
2456 <p>Finally, the <strong>DataBar</strong> allows you to create progress bars:</p>
2457 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">openpyxl.formatting.rule</span> <span class="k">import</span> <span class="n">DataBarRule</span>
2458
2459 <span class="gp">>>> </span><span class="n">data_bar_rule</span> <span class="o">=</span> <span class="n">DataBarRule</span><span class="p">(</span><span class="n">start_type</span><span class="o">=</span><span class="s2">"num"</span><span class="p">,</span>
2460 <span class="gp">... </span> <span class="n">start_value</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
2461 <span class="gp">... </span> <span class="n">end_type</span><span class="o">=</span><span class="s2">"num"</span><span class="p">,</span>
2462 <span class="gp">... </span> <span class="n">end_value</span><span class="o">=</span><span class="s2">"5"</span><span class="p">,</span>
2463 <span class="gp">... </span> <span class="n">color</span><span class="o">=</span><span class="n">colors</span><span class="o">.</span><span class="n">GREEN</span><span class="p">)</span>
2464 <span class="gp">>>> </span><span class="n">sheet</span><span class="o">.</span><span class="n">conditional_formatting</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="s2">"H2:H100"</span><span class="p">,</span> <span class="n">data_bar_rule</span><span class="p">)</span>
2465 <span class="gp">>>> </span><span class="n">workbook</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="s2">"sample_conditional_formatting_data_bar.xlsx"</span><span class="p">)</span>
2466 </pre></div>
2467
2468 <p>You’ll now see a green progress bar that gets fuller the closer the star rating is to the number 5:</p>
2469 <p><a href="https://files.realpython.com/media/Screenshot_2019-06-24_19.09.10.ebbe032c088d.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/Screenshot_2019-06-24_19.09.10.ebbe032c088d.png" width="2160" height="1352" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_19.09.10.ebbe032c088d.png&w=540&sig=a7b8e3515fde3ff6b662ff1780cbc290da9ce2ad 540w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_19.09.10.ebbe032c088d.png&w=1080&sig=1e3d2befc147a43c5b98061cf5888abf22ca4c4a 1080w, https://files.realpython.com/media/Screenshot_2019-06-24_19.09.10.ebbe032c088d.png 2160w" sizes="75vw" alt="Example Spreadsheet With Data Bar Conditional Formatting"/></a></p>
2470 <p>As you can see, there are a lot of cool things you can do with conditional formatting.</p>
2471 <p>Here, you saw only a few examples of what you can achieve with it, but check the <code>openpyxl</code> <a href="https://openpyxl.readthedocs.io/en/stable/formatting.html">documentation</a> to see a bunch of other options.</p>
2472 <h3 id="adding-images">Adding Images</h3>
2473 <p>Even though images are not something that you’ll often see in a spreadsheet, it’s quite cool to be able to add them. Maybe you can use it for branding purposes or to make spreadsheets more personal.</p>
2474 <p>To be able to load images to a spreadsheet using <code>openpyxl</code>, you’ll have to install <code>Pillow</code>:</p>
2475 <div class="highlight sh"><pre><span></span><span class="gp">$</span> pip install Pillow
2476 </pre></div>
2477
2478 <p>Apart from that, you’ll also need an image. For this example, you can grab the <em>Real Python</em> logo below and convert it from <code>.webp</code> to <code>.png</code> using an online converter such as <a href="https://cloudconvert.com/webp-to-png">cloudconvert.com</a>, save the final file as <code>logo.png</code>, and copy it to the root folder where you’re running your examples:</p>
2479 <p><a href="https://files.realpython.com/media/real-python-logo-round.4d95338e8944.png" target="_blank"><img class="img-fluid mx-auto d-block w-25" src="https://files.realpython.com/media/real-python-logo-round.4d95338e8944.png" width="1500" height="1500" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/real-python-logo-round.4d95338e8944.png&w=375&sig=e431a39c9d7f2d5963a81687571a41288c359142 375w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/real-python-logo-round.4d95338e8944.png&w=750&sig=a098752adfc378feee6bc69748af593ed078b8c0 750w, https://files.realpython.com/media/real-python-logo-round.4d95338e8944.png 1500w" sizes="75vw" alt="Real Python Logo"/></a></p>
2480 <p>Afterward, this is the code you need to import that image into the <code>hello_word.xlsx</code> spreadsheet:</p>
2481 <div class="highlight python"><pre><span></span><span class="kn">from</span> <span class="nn">openpyxl</span> <span class="k">import</span> <span class="n">load_workbook</span>
2482 <span class="kn">from</span> <span class="nn">openpyxl.drawing.image</span> <span class="k">import</span> <span class="n">Image</span>
2483
2484 <span class="c1"># Let's use the hello_world spreadsheet since it has less data</span>
2485 <span class="n">workbook</span> <span class="o">=</span> <span class="n">load_workbook</span><span class="p">(</span><span class="n">filename</span><span class="o">=</span><span class="s2">"hello_world.xlsx"</span><span class="p">)</span>
2486 <span class="n">sheet</span> <span class="o">=</span> <span class="n">workbook</span><span class="o">.</span><span class="n">active</span>
2487
2488 <span class="n">logo</span> <span class="o">=</span> <span class="n">Image</span><span class="p">(</span><span class="s2">"logo.png"</span><span class="p">)</span>
2489
2490 <span class="c1"># A bit of resizing to not fill the whole spreadsheet with the logo</span>
2491 <span class="n">logo</span><span class="o">.</span><span class="n">height</span> <span class="o">=</span> <span class="mi">150</span>
2492 <span class="n">logo</span><span class="o">.</span><span class="n">width</span> <span class="o">=</span> <span class="mi">150</span>
2493
2494 <span class="n">sheet</span><span class="o">.</span><span class="n">add_image</span><span class="p">(</span><span class="n">logo</span><span class="p">,</span> <span class="s2">"A3"</span><span class="p">)</span>
2495 <span class="n">workbook</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="n">filename</span><span class="o">=</span><span class="s2">"hello_world_logo.xlsx"</span><span class="p">)</span>
2496 </pre></div>
2497
2498 <p>You have an image on your spreadsheet! Here it is:</p>
2499 <p><a href="https://files.realpython.com/media/Screenshot_2019-06-24_20.05.30.2a69f2a77f68.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/Screenshot_2019-06-24_20.05.30.2a69f2a77f68.png" width="2160" height="1352" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_20.05.30.2a69f2a77f68.png&w=540&sig=574c2c425c011fa21e790f7cd4a41547f2449b01 540w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_20.05.30.2a69f2a77f68.png&w=1080&sig=e6b9c78c7daa929ae86f887d560c3f50f0851d5d 1080w, https://files.realpython.com/media/Screenshot_2019-06-24_20.05.30.2a69f2a77f68.png 2160w" sizes="75vw" alt="Example Spreadsheet With Image"/></a></p>
2500 <p>The image’s left top corner is on the cell you chose, in this case, <code>A3</code>.</p>
2501 <h3 id="adding-pretty-charts">Adding Pretty Charts</h3>
2502 <p>Another powerful thing you can do with spreadsheets is create an incredible variety of charts. </p>
2503 <p>Charts are a great way to visualize and understand loads of data quickly. There are a lot of different chart types: bar chart, pie chart, line chart, and so on. <code>openpyxl</code> has support for a lot of them.</p>
2504 <p>Here, you’ll see only a couple of examples of charts because the theory behind it is the same for every single chart type:</p>
2505 <div class="alert alert-primary" role="alert">
2506 <p><strong>Note:</strong> A few of the chart types that <code>openpyxl</code> currently doesn’t have support for are Funnel, Gantt, Pareto, Treemap, Waterfall, Map, and Sunburst.</p>
2507 </div>
2508 <p>For any chart you want to build, you’ll need to define the chart type: <code>BarChart</code>, <code>LineChart</code>, and so forth, plus the data to be used for the chart, which is called <code>Reference</code>.</p>
2509 <p>Before you can build your chart, you need to define what data you want to see represented in it. Sometimes, you can use the dataset as is, but other times you need to massage the data a bit to get additional information.</p>
2510 <p>Let’s start by building a new workbook with some sample data:</p>
2511 <div class="highlight python"><pre><span></span><span class="lineno"> 1 </span><span class="kn">from</span> <span class="nn">openpyxl</span> <span class="k">import</span> <span class="n">Workbook</span>
2512 <span class="lineno"> 2 </span><span class="kn">from</span> <span class="nn">openpyxl.chart</span> <span class="k">import</span> <span class="n">BarChart</span><span class="p">,</span> <span class="n">Reference</span>
2513 <span class="lineno"> 3 </span>
2514 <span class="lineno"> 4 </span><span class="n">workbook</span> <span class="o">=</span> <span class="n">Workbook</span><span class="p">()</span>
2515 <span class="lineno"> 5 </span><span class="n">sheet</span> <span class="o">=</span> <span class="n">workbook</span><span class="o">.</span><span class="n">active</span>
2516 <span class="lineno"> 6 </span>
2517 <span class="lineno"> 7 </span><span class="c1"># Let's create some sample sales data</span>
2518 <span class="lineno"> 8 </span><span class="n">rows</span> <span class="o">=</span> <span class="p">[</span>
2519 <span class="lineno"> 9 </span> <span class="p">[</span><span class="s2">"Product"</span><span class="p">,</span> <span class="s2">"Online"</span><span class="p">,</span> <span class="s2">"Store"</span><span class="p">],</span>
2520 <span class="lineno">10 </span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">30</span><span class="p">,</span> <span class="mi">45</span><span class="p">],</span>
2521 <span class="lineno">11 </span> <span class="p">[</span><span class="mi">2</span><span class="p">,</span> <span class="mi">40</span><span class="p">,</span> <span class="mi">30</span><span class="p">],</span>
2522 <span class="lineno">12 </span> <span class="p">[</span><span class="mi">3</span><span class="p">,</span> <span class="mi">40</span><span class="p">,</span> <span class="mi">25</span><span class="p">],</span>
2523 <span class="lineno">13 </span> <span class="p">[</span><span class="mi">4</span><span class="p">,</span> <span class="mi">50</span><span class="p">,</span> <span class="mi">30</span><span class="p">],</span>
2524 <span class="lineno">14 </span> <span class="p">[</span><span class="mi">5</span><span class="p">,</span> <span class="mi">30</span><span class="p">,</span> <span class="mi">25</span><span class="p">],</span>
2525 <span class="lineno">15 </span> <span class="p">[</span><span class="mi">6</span><span class="p">,</span> <span class="mi">25</span><span class="p">,</span> <span class="mi">35</span><span class="p">],</span>
2526 <span class="lineno">16 </span> <span class="p">[</span><span class="mi">7</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="mi">40</span><span class="p">],</span>
2527 <span class="lineno">17 </span><span class="p">]</span>
2528 <span class="lineno">18 </span>
2529 <span class="lineno">19 </span><span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">rows</span><span class="p">:</span>
2530 <span class="lineno">20 </span> <span class="n">sheet</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">row</span><span class="p">)</span>
2531 </pre></div>
2532
2533 <p>Now you’re going to start by creating a <strong>bar chart</strong> that displays the total number of sales per product:</p>
2534 <div class="highlight python"><pre><span></span><span class="lineno">22 </span><span class="n">chart</span> <span class="o">=</span> <span class="n">BarChart</span><span class="p">()</span>
2535 <span class="lineno">23 </span><span class="n">data</span> <span class="o">=</span> <span class="n">Reference</span><span class="p">(</span><span class="n">worksheet</span><span class="o">=</span><span class="n">sheet</span><span class="p">,</span>
2536 <span class="lineno">24 </span> <span class="n">min_row</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
2537 <span class="lineno">25 </span> <span class="n">max_row</span><span class="o">=</span><span class="mi">8</span><span class="p">,</span>
2538 <span class="lineno">26 </span> <span class="n">min_col</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span>
2539 <span class="lineno">27 </span> <span class="n">max_col</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span>
2540 <span class="lineno">28 </span>
2541 <span class="lineno">29 </span><span class="n">chart</span><span class="o">.</span><span class="n">add_data</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">titles_from_data</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
2542 <span class="lineno">30 </span><span class="n">sheet</span><span class="o">.</span><span class="n">add_chart</span><span class="p">(</span><span class="n">chart</span><span class="p">,</span> <span class="s2">"E2"</span><span class="p">)</span>
2543 <span class="lineno">31 </span>
2544 <span class="lineno">32 </span><span class="n">workbook</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="s2">"chart.xlsx"</span><span class="p">)</span>
2545 </pre></div>
2546
2547 <p>There you have it. Below, you can see a very straightforward bar chart showing the difference between <strong>online</strong> product sales online and <strong>in-store</strong> product sales:</p>
2548 <p><a href="https://files.realpython.com/media/Screenshot_2019-06-24_20.59.43.7eac35127b97.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/Screenshot_2019-06-24_20.59.43.7eac35127b97.png" width="2160" height="1352" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_20.59.43.7eac35127b97.png&w=540&sig=bcdfa015a56b903169702ddbeb1ec06c8d67bc87 540w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_20.59.43.7eac35127b97.png&w=1080&sig=904be9b2a172b3ae7436311c9d05f6a9ad8ae451 1080w, https://files.realpython.com/media/Screenshot_2019-06-24_20.59.43.7eac35127b97.png 2160w" sizes="75vw" alt="Example Spreadsheet With Bar Chart"/></a></p>
2549 <p>Like with images, the top left corner of the chart is on the cell you added the chart to. In your case, it was on cell <code>E2</code>.</p>
2550 <div class="alert alert-primary" role="alert">
2551 <p><strong>Note:</strong> Depending on whether you’re using Microsoft Excel or an open-source alternative (LibreOffice or OpenOffice), the chart might look slightly different.</p>
2552 </div>
2553 <p>Try creating a <strong>line chart</strong> instead, changing the data a bit:</p>
2554 <div class="highlight python"><pre><span></span><span class="lineno"> 1 </span><span class="kn">import</span> <span class="nn">random</span>
2555 <span class="lineno"> 2 </span><span class="kn">from</span> <span class="nn">openpyxl</span> <span class="k">import</span> <span class="n">Workbook</span>
2556 <span class="lineno"> 3 </span><span class="kn">from</span> <span class="nn">openpyxl.chart</span> <span class="k">import</span> <span class="n">LineChart</span><span class="p">,</span> <span class="n">Reference</span>
2557 <span class="lineno"> 4 </span>
2558 <span class="lineno"> 5 </span><span class="n">workbook</span> <span class="o">=</span> <span class="n">Workbook</span><span class="p">()</span>
2559 <span class="lineno"> 6 </span><span class="n">sheet</span> <span class="o">=</span> <span class="n">workbook</span><span class="o">.</span><span class="n">active</span>
2560 <span class="lineno"> 7 </span>
2561 <span class="lineno"> 8 </span><span class="c1"># Let's create some sample sales data</span>
2562 <span class="lineno"> 9 </span><span class="n">rows</span> <span class="o">=</span> <span class="p">[</span>
2563 <span class="lineno">10 </span> <span class="p">[</span><span class="s2">""</span><span class="p">,</span> <span class="s2">"January"</span><span class="p">,</span> <span class="s2">"February"</span><span class="p">,</span> <span class="s2">"March"</span><span class="p">,</span> <span class="s2">"April"</span><span class="p">,</span>
2564 <span class="lineno">11 </span> <span class="s2">"May"</span><span class="p">,</span> <span class="s2">"June"</span><span class="p">,</span> <span class="s2">"July"</span><span class="p">,</span> <span class="s2">"August"</span><span class="p">,</span> <span class="s2">"September"</span><span class="p">,</span>
2565 <span class="lineno">12 </span> <span class="s2">"October"</span><span class="p">,</span> <span class="s2">"November"</span><span class="p">,</span> <span class="s2">"December"</span><span class="p">],</span>
2566 <span class="lineno">13 </span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="p">],</span>
2567 <span class="lineno">14 </span> <span class="p">[</span><span class="mi">2</span><span class="p">,</span> <span class="p">],</span>
2568 <span class="lineno">15 </span> <span class="p">[</span><span class="mi">3</span><span class="p">,</span> <span class="p">],</span>
2569 <span class="lineno">16 </span><span class="p">]</span>
2570 <span class="lineno">17 </span>
2571 <span class="lineno">18 </span><span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">rows</span><span class="p">:</span>
2572 <span class="lineno">19 </span> <span class="n">sheet</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">row</span><span class="p">)</span>
2573 <span class="lineno">20 </span>
2574 <span class="lineno">21 </span><span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">sheet</span><span class="o">.</span><span class="n">iter_rows</span><span class="p">(</span><span class="n">min_row</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span>
2575 <span class="lineno">22 </span> <span class="n">max_row</span><span class="o">=</span><span class="mi">4</span><span class="p">,</span>
2576 <span class="lineno">23 </span> <span class="n">min_col</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span>
2577 <span class="lineno">24 </span> <span class="n">max_col</span><span class="o">=</span><span class="mi">13</span><span class="p">):</span>
2578 <span class="lineno">25 </span> <span class="k">for</span> <span class="n">cell</span> <span class="ow">in</span> <span class="n">row</span><span class="p">:</span>
2579 <span class="lineno">26 </span> <span class="n">cell</span><span class="o">.</span><span class="n">value</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">randrange</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="mi">100</span><span class="p">)</span>
2580 </pre></div>
2581
2582 <p>With the above code, you’ll be able to generate some random data regarding the sales of 3 different products across a whole year.</p>
2583 <p>Once that’s done, you can very easily create a line chart with the following code:</p>
2584 <div class="highlight python"><pre><span></span><span class="lineno">28 </span><span class="n">chart</span> <span class="o">=</span> <span class="n">LineChart</span><span class="p">()</span>
2585 <span class="lineno">29 </span><span class="n">data</span> <span class="o">=</span> <span class="n">Reference</span><span class="p">(</span><span class="n">worksheet</span><span class="o">=</span><span class="n">sheet</span><span class="p">,</span>
2586 <span class="lineno">30 </span> <span class="n">min_row</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span>
2587 <span class="lineno">31 </span> <span class="n">max_row</span><span class="o">=</span><span class="mi">4</span><span class="p">,</span>
2588 <span class="lineno">32 </span> <span class="n">min_col</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
2589 <span class="lineno">33 </span> <span class="n">max_col</span><span class="o">=</span><span class="mi">13</span><span class="p">)</span>
2590 <span class="lineno">34 </span>
2591 <span class="lineno">35 </span><span class="n">chart</span><span class="o">.</span><span class="n">add_data</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">from_rows</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">titles_from_data</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
2592 <span class="lineno">36 </span><span class="n">sheet</span><span class="o">.</span><span class="n">add_chart</span><span class="p">(</span><span class="n">chart</span><span class="p">,</span> <span class="s2">"C6"</span><span class="p">)</span>
2593 <span class="lineno">37 </span>
2594 <span class="lineno">38 </span><span class="n">workbook</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="s2">"line_chart.xlsx"</span><span class="p">)</span>
2595 </pre></div>
2596
2597 <p>Here’s the outcome of the above piece of code:</p>
2598 <p><a href="https://files.realpython.com/media/Screenshot_2019-06-24_21.06.42.e4e52ab1b433.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/Screenshot_2019-06-24_21.06.42.e4e52ab1b433.png" width="2160" height="1414" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_21.06.42.e4e52ab1b433.png&w=540&sig=362319a9716ded57c9567de98c52a4dc805b5346 540w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_21.06.42.e4e52ab1b433.png&w=1080&sig=cbd7b37aa77318e4ed8d2401281817fb53a7144b 1080w, https://files.realpython.com/media/Screenshot_2019-06-24_21.06.42.e4e52ab1b433.png 2160w" sizes="75vw" alt="Example Spreadsheet With Line Chart"/></a></p>
2599 <p>One thing to keep in mind here is the fact that you’re using <code>from_rows=True</code> when adding the data. This argument makes the chart plot row by row instead of column by column.</p>
2600 <p>In your sample data, you see that each product has a row with 12 values (1 column per month). That’s why you use <code>from_rows</code>. If you don’t pass that argument, by default, the chart tries to plot by column, and you’ll get a month-by-month comparison of sales.</p>
2601 <p>Another difference that has to do with the above argument change is the fact that our <code>Reference</code> now starts from the first column, <code>min_col=1</code>, instead of the second one. This change is needed because the chart now expects the first column to have the titles.</p>
2602 <p>There are a couple of other things you can also change regarding the style of the chart. For example, you can add specific categories to the chart:</p>
2603 <div class="highlight python"><pre><span></span><span class="n">cats</span> <span class="o">=</span> <span class="n">Reference</span><span class="p">(</span><span class="n">worksheet</span><span class="o">=</span><span class="n">sheet</span><span class="p">,</span>
2604 <span class="n">min_row</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
2605 <span class="n">max_row</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
2606 <span class="n">min_col</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span>
2607 <span class="n">max_col</span><span class="o">=</span><span class="mi">13</span><span class="p">)</span>
2608 <span class="n">chart</span><span class="o">.</span><span class="n">set_categories</span><span class="p">(</span><span class="n">cats</span><span class="p">)</span>
2609 </pre></div>
2610
2611 <p>Add this piece of code before saving the workbook, and you should see the month names appearing instead of numbers:</p>
2612 <p><a href="https://files.realpython.com/media/Screenshot_2019-06-24_21.08.05.8867e2cced85.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/Screenshot_2019-06-24_21.08.05.8867e2cced85.png" width="2160" height="1414" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_21.08.05.8867e2cced85.png&w=540&sig=6e48719b75e585dcb58fec3768def0630bb367ff 540w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_21.08.05.8867e2cced85.png&w=1080&sig=9fed1350289fe067d8f50c07516bfcc460f4a720 1080w, https://files.realpython.com/media/Screenshot_2019-06-24_21.08.05.8867e2cced85.png 2160w" sizes="75vw" alt="Example Spreadsheet With Line Chart and Categories"/></a></p>
2613 <p>Code-wise, this is a minimal change. But in terms of the readability of the spreadsheet, this makes it much easier for someone to open the spreadsheet and understand the chart straight away.</p>
2614 <p>Another thing you can do to improve the chart readability is to add an axis. You can do it using the attributes <code>x_axis</code> and <code>y_axis</code>:</p>
2615 <div class="highlight python"><pre><span></span><span class="n">chart</span><span class="o">.</span><span class="n">x_axis</span><span class="o">.</span><span class="n">title</span> <span class="o">=</span> <span class="s2">"Months"</span>
2616 <span class="n">chart</span><span class="o">.</span><span class="n">y_axis</span><span class="o">.</span><span class="n">title</span> <span class="o">=</span> <span class="s2">"Sales (per unit)"</span>
2617 </pre></div>
2618
2619 <p>This will generate a spreadsheet like the below one:</p>
2620 <p><a href="https://files.realpython.com/media/Screenshot_2019-06-24_21.09.46.ce55f629b073.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/Screenshot_2019-06-24_21.09.46.ce55f629b073.png" width="2160" height="1414" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_21.09.46.ce55f629b073.png&w=540&sig=ac73f73702b55a957c77d6da224cde46c2c9a802 540w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_21.09.46.ce55f629b073.png&w=1080&sig=408f6929a900e4a5ccb5cb1cca8647cf64d0d069 1080w, https://files.realpython.com/media/Screenshot_2019-06-24_21.09.46.ce55f629b073.png 2160w" sizes="75vw" alt="Example Spreadsheet With Line Chart, Categories and Axis Titles"/></a></p>
2621 <p>As you can see, small changes like the above make reading your chart a much easier and quicker task.</p>
2622 <p>There is also a way to style your chart by using Excel’s default <code>ChartStyle</code> property. In this case, you have to choose a number between 1 and 48. Depending on your choice, the colors of your chart change as well:</p>
2623 <div class="highlight python"><pre><span></span><span class="c1"># You can play with this by choosing any number between 1 and 48</span>
2624 <span class="n">chart</span><span class="o">.</span><span class="n">style</span> <span class="o">=</span> <span class="mi">24</span>
2625 </pre></div>
2626
2627 <p>With the style selected above, all lines have some shade of orange:</p>
2628 <p><a href="https://files.realpython.com/media/Screenshot_2019-06-24_21.16.31.7df18bbe94cb.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/Screenshot_2019-06-24_21.16.31.7df18bbe94cb.png" width="2160" height="1414" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_21.16.31.7df18bbe94cb.png&w=540&sig=4b0439c6c48d0b3f96f411e6198f6fd49d5c7026 540w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_21.16.31.7df18bbe94cb.png&w=1080&sig=4d0aad0a27321bb321dc9c88158f357155968121 1080w, https://files.realpython.com/media/Screenshot_2019-06-24_21.16.31.7df18bbe94cb.png 2160w" sizes="75vw" alt="Example Spreadsheet With Line Chart, Categories, Axis Titles and Style"/></a></p>
2629 <p>There is no clear documentation on what each style number looks like, but <a href="https://1drv.ms/x/s!Asf0Y5Y4GI3Mg6kZNRd1IA09NLWv9A">this spreadsheet</a> has a few examples of the styles available.</p>
2630 <div class="card mb-3" id="collapse_card0fb191">
2631 <div class="card-header border-0"><p class="m-0"><button class="btn" data-toggle="collapse" data-target="#collapse0fb191" aria-expanded="false" aria-controls="collapse0fb191">Complete Code Example</button> <button class="btn btn-link float-right" data-toggle="collapse" data-target="#collapse0fb191" aria-expanded="false" aria-controls="collapse0fb191">Show/Hide</button></p></div>
2632 <div id="collapse0fb191" class="collapse" data-parent="#collapse_card0fb191"><div class="card-body" markdown="1">
2633
2634 <p>Here’s the full code used to generate the line chart with categories, axis titles, and style:</p>
2635 <div class="highlight python"><pre><span></span><span class="kn">import</span> <span class="nn">random</span>
2636 <span class="kn">from</span> <span class="nn">openpyxl</span> <span class="k">import</span> <span class="n">Workbook</span>
2637 <span class="kn">from</span> <span class="nn">openpyxl.chart</span> <span class="k">import</span> <span class="n">LineChart</span><span class="p">,</span> <span class="n">Reference</span>
2638
2639 <span class="n">workbook</span> <span class="o">=</span> <span class="n">Workbook</span><span class="p">()</span>
2640 <span class="n">sheet</span> <span class="o">=</span> <span class="n">workbook</span><span class="o">.</span><span class="n">active</span>
2641
2642 <span class="c1"># Let's create some sample sales data</span>
2643 <span class="n">rows</span> <span class="o">=</span> <span class="p">[</span>
2644 <span class="p">[</span><span class="s2">""</span><span class="p">,</span> <span class="s2">"January"</span><span class="p">,</span> <span class="s2">"February"</span><span class="p">,</span> <span class="s2">"March"</span><span class="p">,</span> <span class="s2">"April"</span><span class="p">,</span>
2645 <span class="s2">"May"</span><span class="p">,</span> <span class="s2">"June"</span><span class="p">,</span> <span class="s2">"July"</span><span class="p">,</span> <span class="s2">"August"</span><span class="p">,</span> <span class="s2">"September"</span><span class="p">,</span>
2646 <span class="s2">"October"</span><span class="p">,</span> <span class="s2">"November"</span><span class="p">,</span> <span class="s2">"December"</span><span class="p">],</span>
2647 <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="p">],</span>
2648 <span class="p">[</span><span class="mi">2</span><span class="p">,</span> <span class="p">],</span>
2649 <span class="p">[</span><span class="mi">3</span><span class="p">,</span> <span class="p">],</span>
2650 <span class="p">]</span>
2651
2652 <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">rows</span><span class="p">:</span>
2653 <span class="n">sheet</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">row</span><span class="p">)</span>
2654
2655 <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">sheet</span><span class="o">.</span><span class="n">iter_rows</span><span class="p">(</span><span class="n">min_row</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span>
2656 <span class="n">max_row</span><span class="o">=</span><span class="mi">4</span><span class="p">,</span>
2657 <span class="n">min_col</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span>
2658 <span class="n">max_col</span><span class="o">=</span><span class="mi">13</span><span class="p">):</span>
2659 <span class="k">for</span> <span class="n">cell</span> <span class="ow">in</span> <span class="n">row</span><span class="p">:</span>
2660 <span class="n">cell</span><span class="o">.</span><span class="n">value</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">randrange</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="mi">100</span><span class="p">)</span>
2661
2662 <span class="c1"># Create a LineChart and add the main data</span>
2663 <span class="n">chart</span> <span class="o">=</span> <span class="n">LineChart</span><span class="p">()</span>
2664 <span class="n">data</span> <span class="o">=</span> <span class="n">Reference</span><span class="p">(</span><span class="n">worksheet</span><span class="o">=</span><span class="n">sheet</span><span class="p">,</span>
2665 <span class="n">min_row</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span>
2666 <span class="n">max_row</span><span class="o">=</span><span class="mi">4</span><span class="p">,</span>
2667 <span class="n">min_col</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
2668 <span class="n">max_col</span><span class="o">=</span><span class="mi">13</span><span class="p">)</span>
2669 <span class="n">chart</span><span class="o">.</span><span class="n">add_data</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">titles_from_data</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">from_rows</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
2670
2671 <span class="c1"># Add categories to the chart</span>
2672 <span class="n">cats</span> <span class="o">=</span> <span class="n">Reference</span><span class="p">(</span><span class="n">worksheet</span><span class="o">=</span><span class="n">sheet</span><span class="p">,</span>
2673 <span class="n">min_row</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
2674 <span class="n">max_row</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
2675 <span class="n">min_col</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span>
2676 <span class="n">max_col</span><span class="o">=</span><span class="mi">13</span><span class="p">)</span>
2677 <span class="n">chart</span><span class="o">.</span><span class="n">set_categories</span><span class="p">(</span><span class="n">cats</span><span class="p">)</span>
2678
2679 <span class="c1"># Rename the X and Y Axis</span>
2680 <span class="n">chart</span><span class="o">.</span><span class="n">x_axis</span><span class="o">.</span><span class="n">title</span> <span class="o">=</span> <span class="s2">"Months"</span>
2681 <span class="n">chart</span><span class="o">.</span><span class="n">y_axis</span><span class="o">.</span><span class="n">title</span> <span class="o">=</span> <span class="s2">"Sales (per unit)"</span>
2682
2683 <span class="c1"># Apply a specific Style</span>
2684 <span class="n">chart</span><span class="o">.</span><span class="n">style</span> <span class="o">=</span> <span class="mi">24</span>
2685
2686 <span class="c1"># Save!</span>
2687 <span class="n">sheet</span><span class="o">.</span><span class="n">add_chart</span><span class="p">(</span><span class="n">chart</span><span class="p">,</span> <span class="s2">"C6"</span><span class="p">)</span>
2688 <span class="n">workbook</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="s2">"line_chart.xlsx"</span><span class="p">)</span>
2689 </pre></div>
2690
2691 </div></div>
2692
2693 </div>
2694 <p>There are a lot more chart types and customization you can apply, so be sure to check out the <a href="https://openpyxl.readthedocs.io/en/stable/charts/introduction.html">package documentation</a> on this if you need some specific formatting.</p>
2695 <h3 id="convert-python-classes-to-excel-spreadsheet">Convert Python Classes to Excel Spreadsheet</h3>
2696 <p>You already saw how to convert an Excel spreadsheet’s data into Python classes, but now let’s do the opposite.</p>
2697 <p>Let’s imagine you have a database and are using some Object-Relational Mapping (ORM) to map DB objects into Python classes. Now, you want to export those same objects into a spreadsheet.</p>
2698 <p>Let’s assume the following <a href="https://realpython.com/python-data-classes/">data classes</a> to represent the data coming from your database regarding product sales:</p>
2699 <div class="highlight python"><pre><span></span><span class="kn">from</span> <span class="nn">dataclasses</span> <span class="k">import</span> <span class="n">dataclass</span>
2700 <span class="kn">from</span> <span class="nn">typing</span> <span class="k">import</span> <span class="n">List</span>
2701
2702 <span class="nd">@dataclass</span>
2703 <span class="k">class</span> <span class="nc">Sale</span><span class="p">:</span>
2704 <span class="nb">id</span><span class="p">:</span> <span class="nb">str</span>
2705 <span class="n">quantity</span><span class="p">:</span> <span class="nb">int</span>
2706
2707 <span class="nd">@dataclass</span>
2708 <span class="k">class</span> <span class="nc">Product</span><span class="p">:</span>
2709 <span class="nb">id</span><span class="p">:</span> <span class="nb">str</span>
2710 <span class="n">name</span><span class="p">:</span> <span class="nb">str</span>
2711 <span class="n">sales</span><span class="p">:</span> <span class="n">List</span><span class="p">[</span><span class="n">Sale</span><span class="p">]</span>
2712 </pre></div>
2713
2714 <p>Now, let’s generate some random data, assuming the above classes are stored in a <code>db_classes.py</code> file:</p>
2715 <div class="highlight python"><pre><span></span><span class="lineno"> 1 </span><span class="kn">import</span> <span class="nn">random</span>
2716 <span class="lineno"> 2 </span>
2717 <span class="lineno"> 3 </span><span class="c1"># Ignore these for now. You'll use them in a sec ;)</span>
2718 <span class="lineno"> 4 </span><span class="kn">from</span> <span class="nn">openpyxl</span> <span class="k">import</span> <span class="n">Workbook</span>
2719 <span class="lineno"> 5 </span><span class="kn">from</span> <span class="nn">openpyxl.chart</span> <span class="k">import</span> <span class="n">LineChart</span><span class="p">,</span> <span class="n">Reference</span>
2720 <span class="lineno"> 6 </span>
2721 <span class="lineno"> 7 </span><span class="kn">from</span> <span class="nn">db_classes</span> <span class="k">import</span> <span class="n">Product</span><span class="p">,</span> <span class="n">Sale</span>
2722 <span class="lineno"> 8 </span>
2723 <span class="lineno"> 9 </span><span class="n">products</span> <span class="o">=</span> <span class="p">[]</span>
2724 <span class="lineno">10 </span>
2725 <span class="lineno">11 </span><span class="c1"># Let's create 5 products</span>
2726 <span class="lineno">12 </span><span class="k">for</span> <span class="n">idx</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">6</span><span class="p">):</span>
2727 <span class="lineno">13 </span> <span class="n">sales</span> <span class="o">=</span> <span class="p">[]</span>
2728 <span class="lineno">14 </span>
2729 <span class="lineno">15 </span> <span class="c1"># Create 5 months of sales</span>
2730 <span class="lineno">16 </span> <span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">):</span>
2731 <span class="lineno">17 </span> <span class="n">sale</span> <span class="o">=</span> <span class="n">Sale</span><span class="p">(</span><span class="n">quantity</span><span class="o">=</span><span class="n">random</span><span class="o">.</span><span class="n">randrange</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="mi">100</span><span class="p">))</span>
2732 <span class="lineno">18 </span> <span class="n">sales</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">sale</span><span class="p">)</span>
2733 <span class="lineno">19 </span>
2734 <span class="lineno">20 </span> <span class="n">product</span> <span class="o">=</span> <span class="n">Product</span><span class="p">(</span><span class="nb">id</span><span class="o">=</span><span class="nb">str</span><span class="p">(</span><span class="n">idx</span><span class="p">),</span>
2735 <span class="lineno">21 </span> <span class="n">name</span><span class="o">=</span><span class="s2">"Product </span><span class="si">%s</span><span class="s2">"</span> <span class="o">%</span> <span class="n">idx</span><span class="p">,</span>
2736 <span class="lineno">22 </span> <span class="n">sales</span><span class="o">=</span><span class="n">sales</span><span class="p">)</span>
2737 <span class="lineno">23 </span> <span class="n">products</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">product</span><span class="p">)</span>
2738 </pre></div>
2739
2740 <p>By running this piece of code, you should get 5 products with 5 months of sales with a random quantity of sales for each month.</p>
2741 <p>Now, to convert this into a spreadsheet, you need to iterate over the data and append it to the spreadsheet:</p>
2742 <div class="highlight python"><pre><span></span><span class="lineno">25 </span><span class="n">workbook</span> <span class="o">=</span> <span class="n">Workbook</span><span class="p">()</span>
2743 <span class="lineno">26 </span><span class="n">sheet</span> <span class="o">=</span> <span class="n">workbook</span><span class="o">.</span><span class="n">active</span>
2744 <span class="lineno">27 </span>
2745 <span class="lineno">28 </span><span class="c1"># Append column names first</span>
2746 <span class="lineno">29 </span><span class="n">sheet</span><span class="o">.</span><span class="n">append</span><span class="p">([</span><span class="s2">"Product ID"</span><span class="p">,</span> <span class="s2">"Product Name"</span><span class="p">,</span> <span class="s2">"Month 1"</span><span class="p">,</span>
2747 <span class="lineno">30 </span> <span class="s2">"Month 2"</span><span class="p">,</span> <span class="s2">"Month 3"</span><span class="p">,</span> <span class="s2">"Month 4"</span><span class="p">,</span> <span class="s2">"Month 5"</span><span class="p">])</span>
2748 <span class="lineno">31 </span>
2749 <span class="lineno">32 </span><span class="c1"># Append the data</span>
2750 <span class="lineno">33 </span><span class="k">for</span> <span class="n">product</span> <span class="ow">in</span> <span class="n">products</span><span class="p">:</span>
2751 <span class="lineno">34 </span> <span class="n">data</span> <span class="o">=</span> <span class="p">[</span><span class="n">product</span><span class="o">.</span><span class="n">id</span><span class="p">,</span> <span class="n">product</span><span class="o">.</span><span class="n">name</span><span class="p">]</span>
2752 <span class="lineno">35 </span> <span class="k">for</span> <span class="n">sale</span> <span class="ow">in</span> <span class="n">product</span><span class="o">.</span><span class="n">sales</span><span class="p">:</span>
2753 <span class="lineno">36 </span> <span class="n">data</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">sale</span><span class="o">.</span><span class="n">quantity</span><span class="p">)</span>
2754 <span class="lineno">37 </span> <span class="n">sheet</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
2755 </pre></div>
2756
2757 <p>That’s it. That should allow you to create a spreadsheet with some data coming from your database.</p>
2758 <p>However, why not use some of that cool knowledge you gained recently to add a chart as well to display that data more visually?</p>
2759 <p>All right, then you could probably do something like this:</p>
2760 <div class="highlight python"><pre><span></span><span class="lineno">38 </span><span class="n">chart</span> <span class="o">=</span> <span class="n">LineChart</span><span class="p">()</span>
2761 <span class="lineno">39 </span><span class="n">data</span> <span class="o">=</span> <span class="n">Reference</span><span class="p">(</span><span class="n">worksheet</span><span class="o">=</span><span class="n">sheet</span><span class="p">,</span>
2762 <span class="lineno">40 </span> <span class="n">min_row</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span>
2763 <span class="lineno">41 </span> <span class="n">max_row</span><span class="o">=</span><span class="mi">6</span><span class="p">,</span>
2764 <span class="lineno">42 </span> <span class="n">min_col</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span>
2765 <span class="lineno">43 </span> <span class="n">max_col</span><span class="o">=</span><span class="mi">7</span><span class="p">)</span>
2766 <span class="lineno">44 </span>
2767 <span class="lineno">45 </span><span class="n">chart</span><span class="o">.</span><span class="n">add_data</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">titles_from_data</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">from_rows</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
2768 <span class="lineno">46 </span><span class="n">sheet</span><span class="o">.</span><span class="n">add_chart</span><span class="p">(</span><span class="n">chart</span><span class="p">,</span> <span class="s2">"B8"</span><span class="p">)</span>
2769 <span class="lineno">47 </span>
2770 <span class="lineno">48 </span><span class="n">cats</span> <span class="o">=</span> <span class="n">Reference</span><span class="p">(</span><span class="n">worksheet</span><span class="o">=</span><span class="n">sheet</span><span class="p">,</span>
2771 <span class="lineno">49 </span> <span class="n">min_row</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
2772 <span class="lineno">50 </span> <span class="n">max_row</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
2773 <span class="lineno">51 </span> <span class="n">min_col</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span>
2774 <span class="lineno">52 </span> <span class="n">max_col</span><span class="o">=</span><span class="mi">7</span><span class="p">)</span>
2775 <span class="lineno">53 </span><span class="n">chart</span><span class="o">.</span><span class="n">set_categories</span><span class="p">(</span><span class="n">cats</span><span class="p">)</span>
2776 <span class="lineno">54 </span>
2777 <span class="lineno">55 </span><span class="n">chart</span><span class="o">.</span><span class="n">x_axis</span><span class="o">.</span><span class="n">title</span> <span class="o">=</span> <span class="s2">"Months"</span>
2778 <span class="lineno">56 </span><span class="n">chart</span><span class="o">.</span><span class="n">y_axis</span><span class="o">.</span><span class="n">title</span> <span class="o">=</span> <span class="s2">"Sales (per unit)"</span>
2779 <span class="lineno">57 </span>
2780 <span class="lineno">58 </span><span class="n">workbook</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="n">filename</span><span class="o">=</span><span class="s2">"oop_sample.xlsx"</span><span class="p">)</span>
2781 </pre></div>
2782
2783 <p>Now we’re talking! Here’s a spreadsheet generated from database objects and with a chart and everything:</p>
2784 <p><a href="https://files.realpython.com/media/Screenshot_2019-06-24_21.26.23.1f355e76586d.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/Screenshot_2019-06-24_21.26.23.1f355e76586d.png" width="2160" height="1414" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_21.26.23.1f355e76586d.png&w=540&sig=135f4ee5413467c91f65bbb6e914724cdf1fd413 540w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_21.26.23.1f355e76586d.png&w=1080&sig=5b7dd165f92c237ddd049350ca6fc6a165e10512 1080w, https://files.realpython.com/media/Screenshot_2019-06-24_21.26.23.1f355e76586d.png 2160w" sizes="75vw" alt="Example Spreadsheet With Conversion from Python Data Classes"/></a></p>
2785 <p>That’s a great way for you to wrap up your new knowledge of charts!</p>
2786 <h3 id="bonus-working-with-pandas">Bonus: Working With Pandas</h3>
2787 <p>Even though you can use <a href="https://realpython.com/working-with-large-excel-files-in-pandas/">Pandas to handle Excel files</a>, there are few things that you either can’t accomplish with Pandas or that you’d be better off just using <code>openpyxl</code> directly.</p>
2788 <p>For example, some of the advantages of using <code>openpyxl</code> are the ability to easily customize your spreadsheet with styles, conditional formatting, and such.</p>
2789 <p>But guess what, you don’t have to worry about picking. In fact, <code>openpyxl</code> has support for both converting data from a Pandas DataFrame into a workbook or the opposite, converting an <code>openpyxl</code> workbook into a Pandas DataFrame.</p>
2790 <div class="alert alert-primary" role="alert">
2791 <p><strong>Note:</strong> If you’re new to Pandas, check our <a href="https://realpython.com/courses/pandas-dataframes-101/">course on Pandas DataFrames</a> beforehand.</p>
2792 </div>
2793 <p>First things first, remember to install the <code>pandas</code> package:</p>
2794 <div class="highlight sh"><pre><span></span><span class="gp">$</span> pip install pandas
2795 </pre></div>
2796
2797 <p>Then, let’s create a sample DataFrame:</p>
2798 <div class="highlight python"><pre><span></span><span class="lineno"> 1 </span><span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
2799 <span class="lineno"> 2 </span>
2800 <span class="lineno"> 3 </span><span class="n">data</span> <span class="o">=</span> <span class="p">{</span>
2801 <span class="lineno"> 4 </span> <span class="s2">"Product Name"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"Product 1"</span><span class="p">,</span> <span class="s2">"Product 2"</span><span class="p">],</span>
2802 <span class="lineno"> 5 </span> <span class="s2">"Sales Month 1"</span><span class="p">:</span> <span class="p">[</span><span class="mi">10</span><span class="p">,</span> <span class="mi">20</span><span class="p">],</span>
2803 <span class="lineno"> 6 </span> <span class="s2">"Sales Month 2"</span><span class="p">:</span> <span class="p">[</span><span class="mi">5</span><span class="p">,</span> <span class="mi">35</span><span class="p">],</span>
2804 <span class="lineno"> 7 </span><span class="p">}</span>
2805 <span class="lineno"> 8 </span><span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
2806 </pre></div>
2807
2808 <p>Now that you have some data, you can use <code>.dataframe_to_rows()</code> to convert it from a DataFrame into a worksheet:</p>
2809 <div class="highlight python"><pre><span></span><span class="lineno">10 </span><span class="kn">from</span> <span class="nn">openpyxl</span> <span class="k">import</span> <span class="n">Workbook</span>
2810 <span class="lineno">11 </span><span class="kn">from</span> <span class="nn">openpyxl.utils.dataframe</span> <span class="k">import</span> <span class="n">dataframe_to_rows</span>
2811 <span class="lineno">12 </span>
2812 <span class="lineno">13 </span><span class="n">workbook</span> <span class="o">=</span> <span class="n">Workbook</span><span class="p">()</span>
2813 <span class="lineno">14 </span><span class="n">sheet</span> <span class="o">=</span> <span class="n">workbook</span><span class="o">.</span><span class="n">active</span>
2814 <span class="lineno">15 </span>
2815 <span class="lineno">16 </span><span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">dataframe_to_rows</span><span class="p">(</span><span class="n">df</span><span class="p">,</span> <span class="n">index</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">header</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
2816 <span class="lineno">17 </span> <span class="n">sheet</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">row</span><span class="p">)</span>
2817 <span class="lineno">18 </span>
2818 <span class="lineno">19 </span><span class="n">workbook</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="s2">"pandas.xlsx"</span><span class="p">)</span>
2819 </pre></div>
2820
2821 <p>You should see a spreadsheet that looks like this:</p>
2822 <p><a href="https://files.realpython.com/media/Screenshot_2019-06-24_21.42.15.0a4208db25f0.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/Screenshot_2019-06-24_21.42.15.0a4208db25f0.png" width="2160" height="1414" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_21.42.15.0a4208db25f0.png&w=540&sig=636303dd8f99512651c5868f4ef572b2afa75d3c 540w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screenshot_2019-06-24_21.42.15.0a4208db25f0.png&w=1080&sig=c2ddf015c46566fc545ad03187cc5780a61938f9 1080w, https://files.realpython.com/media/Screenshot_2019-06-24_21.42.15.0a4208db25f0.png 2160w" sizes="75vw" alt="Example Spreadsheet With Data from Pandas Data Frame"/></a></p>
2823 <p>If you want to add the <a href="https://realpython.com/python-data-cleaning-numpy-pandas/#changing-the-index-of-a-dataframe">DataFrame’s index</a>, you can change <code>index=True</code>, and it adds each row’s index into your spreadsheet.</p>
2824 <p>On the other hand, if you want to convert a spreadsheet into a DataFrame, you can also do it in a very straightforward way like so:</p>
2825 <div class="highlight python"><pre><span></span><span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
2826 <span class="kn">from</span> <span class="nn">openpyxl</span> <span class="k">import</span> <span class="n">load_workbook</span>
2827
2828 <span class="n">workbook</span> <span class="o">=</span> <span class="n">load_workbook</span><span class="p">(</span><span class="n">filename</span><span class="o">=</span><span class="s2">"sample.xlsx"</span><span class="p">)</span>
2829 <span class="n">sheet</span> <span class="o">=</span> <span class="n">workbook</span><span class="o">.</span><span class="n">active</span>
2830
2831 <span class="n">values</span> <span class="o">=</span> <span class="n">sheet</span><span class="o">.</span><span class="n">values</span>
2832 <span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">values</span><span class="p">)</span>
2833 </pre></div>
2834
2835 <p>Alternatively, if you want to add the correct headers and use the review ID as the index, for example, then you can also do it like this instead:</p>
2836 <div class="highlight python"><pre><span></span><span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
2837 <span class="kn">from</span> <span class="nn">openpyxl</span> <span class="k">import</span> <span class="n">load_workbook</span>
2838 <span class="kn">from</span> <span class="nn">mapping</span> <span class="k">import</span> <span class="n">REVIEW_ID</span>
2839
2840 <span class="n">workbook</span> <span class="o">=</span> <span class="n">load_workbook</span><span class="p">(</span><span class="n">filename</span><span class="o">=</span><span class="s2">"sample.xlsx"</span><span class="p">)</span>
2841 <span class="n">sheet</span> <span class="o">=</span> <span class="n">workbook</span><span class="o">.</span><span class="n">active</span>
2842
2843 <span class="n">data</span> <span class="o">=</span> <span class="n">sheet</span><span class="o">.</span><span class="n">values</span>
2844
2845 <span class="c1"># Set the first row as the columns for the DataFrame</span>
2846 <span class="n">cols</span> <span class="o">=</span> <span class="nb">next</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
2847 <span class="n">data</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
2848
2849 <span class="c1"># Set the field "review_id" as the indexes for each row</span>
2850 <span class="n">idx</span> <span class="o">=</span> <span class="p">[</span><span class="n">row</span><span class="p">[</span><span class="n">REVIEW_ID</span><span class="p">]</span> <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">data</span><span class="p">]</span>
2851
2852 <span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">index</span><span class="o">=</span><span class="n">idx</span><span class="p">,</span> <span class="n">columns</span><span class="o">=</span><span class="n">cols</span><span class="p">)</span>
2853 </pre></div>
2854
2855 <p>Using indexes and columns allows you to access data from your DataFrame easily:</p>
2856 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">df</span><span class="o">.</span><span class="n">columns</span>
2857 <span class="go">Index(['marketplace', 'customer_id', 'review_id', 'product_id',</span>
2858 <span class="go"> 'product_parent', 'product_title', 'product_category', 'star_rating',</span>
2859 <span class="go"> 'helpful_votes', 'total_votes', 'vine', 'verified_purchase',</span>
2860 <span class="go"> 'review_headline', 'review_body', 'review_date'],</span>
2861 <span class="go"> dtype='object')</span>
2862
2863 <span class="gp">>>> </span><span class="c1"># Get first 10 reviews' star rating</span>
2864 <span class="gp">>>> </span><span class="n">df</span><span class="p">[</span><span class="s2">"star_rating"</span><span class="p">][:</span><span class="mi">10</span><span class="p">]</span>
2865 <span class="go">R3O9SGZBVQBV76 5</span>
2866 <span class="go">RKH8BNC3L5DLF 5</span>
2867 <span class="go">R2HLE8WKZSU3NL 2</span>
2868 <span class="go">R31U3UH5AZ42LL 5</span>
2869 <span class="go">R2SV659OUJ945Y 4</span>
2870 <span class="go">RA51CP8TR5A2L 5</span>
2871 <span class="go">RB2Q7DLDN6TH6 5</span>
2872 <span class="go">R2RHFJV0UYBK3Y 1</span>
2873 <span class="go">R2Z6JOQ94LFHEP 5</span>
2874 <span class="go">RX27XIIWY5JPB 4</span>
2875 <span class="go">Name: star_rating, dtype: int64</span>
2876
2877 <span class="gp">>>> </span><span class="c1"># Grab review with id "R2EQL1V1L6E0C9", using the index</span>
2878 <span class="gp">>>> </span><span class="n">df</span><span class="o">.</span><span class="n">loc</span><span class="p">[</span><span class="s2">"R2EQL1V1L6E0C9"</span><span class="p">]</span>
2879 <span class="go">marketplace US</span>
2880 <span class="go">customer_id 15305006</span>
2881 <span class="go">review_id R2EQL1V1L6E0C9</span>
2882 <span class="go">product_id B004LURNO6</span>
2883 <span class="go">product_parent 892860326</span>
2884 <span class="go">review_headline Five Stars</span>
2885 <span class="go">review_body Love it</span>
2886 <span class="go">review_date 2015-08-31</span>
2887 <span class="go">Name: R2EQL1V1L6E0C9, dtype: object</span>
2888 </pre></div>
2889
2890 <p>There you go, whether you want to use <code>openpyxl</code> to prettify your Pandas dataset or use Pandas to do some hardcore algebra, you now know how to switch between both packages.</p>
2891 <h2 id="conclusion">Conclusion</h2>
2892 <p><em>Phew</em>, after that long read, you now know how to work with spreadsheets in Python! You can rely on <code>openpyxl</code>, your trustworthy companion, to:</p>
2893 <ul>
2894 <li>Extract valuable information from spreadsheets in a Pythonic manner</li>
2895 <li>Create your own spreadsheets, no matter the complexity level</li>
2896 <li>Add cool features such as conditional formatting or charts to your spreadsheets</li>
2897 </ul>
2898 <p>There are a few other things you can do with <code>openpyxl</code> that might not have been covered in this tutorial, but you can always check the package’s official <a href="https://openpyxl.readthedocs.io/en/stable/index.html">documentation website</a> to learn more about it. You can even venture into checking its <a href="https://bitbucket.org/openpyxl/openpyxl/src/default/">source code</a> and improving the package further.</p>
2899 <p>Feel free to leave any comments below if you have any questions, or if there’s any section you’d love to hear more about.</p>
2900 <div class="alert alert-warning" role="alert"><p><strong>Download Dataset:</strong> <a href="https://realpython.com/optins/view/openpyxl-sample-dataset/" class="alert-link" data-toggle="modal" data-target="#modal-openpyxl-sample-dataset" data-focus="false">Click here to download the dataset for the openpyxl exercise you'll be following in this tutorial.</a></p></div>
2901 <hr />
2902 <p><em>[ Improve Your Python With π Python Tricks π β Get a short & sweet Python Trick delivered to your inbox every couple of days. <a href="https://realpython.com/python-tricks/?utm_source=realpython&utm_medium=rss&utm_campaign=footer">>> Click here to learn more and see examples</a> ]</em></p>#
2903 datePublished: #Mon Aug 26 14:00:00 2019#
2904 dateUpdated: #Mon Aug 26 14:00:00 2019#
2905 # Person begin ####################
2906 name: #Real Python#
2907 # Person end ######################
2908 # Item end ########################
2909 # Item begin ######################
2910 id: #https://realpython.com/cpython-source-code-guide/#
2911 title: #Your Guide to the CPython Source Code#
2912 link: #https://realpython.com/cpython-source-code-guide/#
2913 description: #In this detailed Python tutorial, you'll explore the CPython source code. By following this step-by-step walkthrough, you'll take a deep dive into how the CPython compiler works and how your Python code gets executed.#
2914 content: #<p>Are there certain parts of Python that just seem magic? Like how are dictionaries so much faster than looping over a list to find an item. How does a generator remember the state of the variables each time it yields a value and why do you never have to allocate memory like other languages? It turns out, CPython, the most popular Python runtime is written in human-readable C and Python code. This tutorial will walk you through the CPython source code. </p>
2915 <p>You’ll cover all the concepts behind the internals of CPython, how they work and visual explanations as you go.</p>
2916 <p><strong>You’ll learn how to:</strong></p>
2917 <ul>
2918 <li>Read and navigate the source code</li>
2919 <li>Compile CPython from source code</li>
2920 <li>Navigate and comprehend the inner workings of concepts like lists, dictionaries, and generators</li>
2921 <li>Run the test suite</li>
2922 <li>Modify or upgrade components of the CPython library to contribute them to future versions</li>
2923 </ul>
2924 <p>Yes, this is a very long article. If you just made yourself a fresh cup of tea, coffee or your favorite beverage, it’s going to be cold by the end of Part 1. </p>
2925 <p>This tutorial is split into five parts. Take your time for each part and make sure you try out the demos and the interactive components. You can feel a sense of achievement that you grasp the core concepts of Python that can make you a better Python programmer.</p>
2926 <div class="alert alert-warning" role="alert"><p><strong>Free Bonus:</strong> <a href="" class="alert-link" data-toggle="modal" data-target="#modal-python-mastery-course" data-focus="false">5 Thoughts On Python Mastery</a>, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.</p></div>
2927
2928 <h2 h1="h1" id="part-1-introduction-to-cpython">Part 1: Introduction to CPython</h2>
2929 <p>When you type <code>python</code> at the console or install a Python distribution from <a href="https://www.python.org">python.org</a>, you are running <strong>CPython</strong>. CPython is one of the many Python runtimes, maintained and written by different teams of developers. Some other runtimes you may have heard are <a href="https://pypy.org/">PyPy</a>, <a href="https://cython.org/">Cython</a>, and <a href="https://www.jython.org/">Jython</a>.</p>
2930 <p>The unique thing about CPython is that it contains both a runtime and the shared language specification that all Python runtimes use. CPython is the “official,” or reference implementation of Python.</p>
2931 <p>The Python language specification is the document that the description of the Python language. For example, it says that <code>assert</code> is a reserved keyword, and that <code>[]</code> is used for indexing, slicing, and creating empty lists.</p>
2932 <p>Think about what you expect to be inside the Python distribution on your computer:</p>
2933 <ul>
2934 <li>When you type <code>python</code> without a file or module, it gives an interactive prompt.</li>
2935 <li>You can import built-in modules from the standard library like <code>json</code>.</li>
2936 <li>You can install packages from the internet using <code>pip</code>.</li>
2937 <li>You can test your applications using the built-in <code>unittest</code> library.</li>
2938 </ul>
2939 <p>These are all part of the CPython distribution. There’s a lot more than just a compiler.</p>
2940 <div class="alert alert-primary" role="alert">
2941 <p><strong>Note:</strong> This article is written against version <a href="https://github.com/python/cpython/tree/v3.8.0b4">3.8.0b4</a> of the CPython source code.</p>
2942 </div>
2943 <h3 id="whats-in-the-source-code">What’s in the Source Code?</h3>
2944 <p>The CPython source distribution comes with a whole range of tools, libraries, and components. We’ll explore those in this article. First we are going to focus on the compiler.</p>
2945 <p>To download a copy of the CPython source code, you can use <code>git</code> to pull the latest version to a working copy locally:</p>
2946 <div class="highlight sh"><pre><span></span><span class="go">git clone https://github.com/python/cpython</span>
2947 <span class="go">cd cpython</span>
2948 <span class="go">git checkout v3.8.0b4</span>
2949 </pre></div>
2950
2951 <div class="alert alert-primary" role="alert">
2952 <p><strong>Note:</strong> If you don’t have Git available, you can download the source in a <a href="https://github.com/python/cpython/archive/v3.8.0b4.zip">ZIP</a> file directly from the GitHub website.</p>
2953 </div>
2954 <p>Inside of the newly downloaded <code>cpython</code> directory, you will find the following subdirectories:</p>
2955 <div class="highlight"><pre><span></span>cpython/
2956 β
2957 βββ Doc β Source for the documentation
2958 βββ Grammar β The computer-readable language definition
2959 βββ Include β The C header files
2960 βββ Lib β Standard library modules written in Python
2961 βββ Mac β macOS support files
2962 βββ Misc β Miscellaneous files
2963 βββ Modules β Standard Library Modules written in C
2964 βββ Objects β Core types and the object model
2965 βββ Parser β The Python parser source code
2966 βββ PC β Windows build support files
2967 βββ PCbuild β Windows build support files for older Windows versions
2968 βββ Programs β Source code for the python executable and other binaries
2969 βββ Python β The CPython interpreter source code
2970 βββ Tools β Standalone tools useful for building or extending Python
2971 </pre></div>
2972
2973 <p>Next, we’ll compile CPython from the source code. This step requires a C compiler, and some build tools, which depend on the operating system you’re using.</p>
2974 <h3 id="compiling-cpython-macos">Compiling CPython (macOS)</h3>
2975 <p>Compiling CPython on macOS is straightforward. You will first need the essential C compiler toolkit. The Command Line Development Tools is an app that you can update in macOS through the App Store. You need to perform the initial installation on the terminal.</p>
2976 <p>To open up a terminal in macOS, go to the Launchpad, then <em>Other</em> then choose the <em>Terminal</em> app. You will want to save this app to your Dock, so right-click the Icon and select <em>Keep in Dock</em>.</p>
2977 <p>Now, within the terminal, install the C compiler and toolkit by running the following:</p>
2978 <div class="highlight sh"><pre><span></span><span class="gp">$</span> xcode-select --install
2979 </pre></div>
2980
2981 <p>This command will pop up with a prompt to download and install a set of tools, including Git, Make, and the GNU C compiler.</p>
2982 <p>You will also need a working copy of <a href="https://www.openssl.org/">OpenSSL</a> to use for fetching packages from the PyPi.org website. If you later plan on using this build to install additional packages, SSL validation is required.</p>
2983 <p>The simplest way to install OpenSSL on macOS is by using <a href="https://brew.sh">HomeBrew</a>. If you already have HomeBrew installed, you can install the dependencies for CPython with the <code>brew install</code> command:</p>
2984 <div class="highlight sh"><pre><span></span><span class="gp">$</span> brew install openssl xz zlib
2985 </pre></div>
2986
2987 <p>Now that you have the dependencies, you can run the <code>configure</code> script, enabling SSL support by discovering the location that HomeBrew installed to and enabling the debug hooks <code>--with-pydebug</code>:</p>
2988 <div class="highlight sh"><pre><span></span><span class="gp">$</span> <span class="nv">CPPFLAGS</span><span class="o">=</span><span class="s2">"-I</span><span class="k">$(</span>brew --prefix zlib<span class="k">)</span><span class="s2">/include"</span> <span class="se">\</span>
2989 <span class="nv">LDFLAGS</span><span class="o">=</span><span class="s2">"-L</span><span class="k">$(</span>brew --prefix zlib<span class="k">)</span><span class="s2">/lib"</span> <span class="se">\</span>
2990 ./configure --with-openssl<span class="o">=</span><span class="k">$(</span>brew --prefix openssl<span class="k">)</span> --with-pydebug
2991 </pre></div>
2992
2993 <p>This will generate a <code>Makefile</code> in the root of the repository that you can use to automate the build process. The <code>./configure</code> step only needs to be run once. You can build the CPython binary by running:</p>
2994 <div class="highlight sh"><pre><span></span><span class="gp">$</span> make -j2 -s
2995 </pre></div>
2996
2997 <p>The <code>-j2</code> flag allows <code>make</code> to run 2 jobs simultaneously. If you have 4 cores, you can change this to 4. The <code>-s</code> flag stops the <code>Makefile</code> from printing every command it runs to the console. You can remove this, but the output is very verbose.</p>
2998 <p>During the build, you may receive some errors, and in the summary, it will notify you that not all packages could be built. For example, <code>_dbm</code>, <code>_sqlite3</code>, <code>_uuid</code>, <code>nis</code>, <code>ossaudiodev</code>, <code>spwd</code>, and <code>_tkinter</code> would fail to build with this set of instructions. That’s okay if you aren’t planning on developing against those packages. If you are, then check out the <a href="https://devguide.python.org/">dev guide</a> website for more information.</p>
2999 <p>The build will take a few minutes and generate a binary called <code>python.exe</code>. Every time you make changes to the source code, you will need to re-run <code>make</code> with the same flags.
3000 The <code>python.exe</code> binary is the debug binary of CPython. Execute <code>python.exe</code> to see a working REPL:</p>
3001 <div class="highlight sh"><pre><span></span><span class="gp">$</span> ./python.exe
3002 <span class="go">Python 3.8.0b4 (tags/v3.8.0b4:d93605de72, Aug 30 2019, 10:00:03) </span>
3003 <span class="go">[Clang 10.0.1 (clang-1001.0.46.4)] on darwin</span>
3004 <span class="go">Type "help", "copyright", "credits" or "license" for more information.</span>
3005 <span class="gp">></span>>>
3006 </pre></div>
3007
3008 <div class="alert alert-primary" role="alert">
3009 <p><strong>Note:</strong>
3010 Yes, that’s right, the macOS build has a file extension for <code>.exe</code>. This is <em>not</em> because it’s a Windows binary. Because macOS has a case-insensitive filesystem and when working with the binary, the developers didn’t want people to accidentally refer to the directory <code>Python/</code> so <code>.exe</code> was appended to avoid ambiguity.
3011 If you later run <code>make install</code> or <code>make altinstall</code>, it will rename the file back to <code>python</code>.</p>
3012 </div>
3013 <h3 id="compiling-cpython-linux">Compiling CPython (Linux)</h3>
3014 <p>For Linux, the first step is to download and install <code>make</code>, <code>gcc</code>, <code>configure</code>, and <code>pkgconfig</code>. </p>
3015 <p>For Fedora Core, RHEL, CentOS, or other yum-based systems: </p>
3016 <div class="highlight sh"><pre><span></span><span class="gp">$</span> sudo yum install yum-utils
3017 </pre></div>
3018
3019 <p>For Debian, Ubuntu, or other <code>apt</code>-based systems:</p>
3020 <div class="highlight sh"><pre><span></span><span class="gp">$</span> sudo apt install build-essential
3021 </pre></div>
3022
3023 <p>Then install the required packages, for Fedora Core, RHEL, CentOS or other yum-based systems: </p>
3024 <div class="highlight sh"><pre><span></span><span class="gp">$</span> sudo yum-builddep python3
3025 </pre></div>
3026
3027 <p>For Debian, Ubuntu, or other <code>apt</code>-based systems:</p>
3028 <div class="highlight sh"><pre><span></span><span class="gp">$</span> sudo apt install libssl-dev zlib1g-dev libncurses5-dev <span class="se">\</span>
3029 libncursesw5-dev libreadline-dev libsqlite3-dev libgdbm-dev <span class="se">\</span>
3030 libdb5.3-dev libbz2-dev libexpat1-dev liblzma-dev libffi-dev
3031 </pre></div>
3032
3033 <p>Now that you have the dependencies, you can run the <code>configure</code> script, enabling the debug hooks <code>--with-pydebug</code>:</p>
3034 <div class="highlight sh"><pre><span></span><span class="gp">$</span> ./configure --with-pydebug
3035 </pre></div>
3036
3037 <p>Review the output to ensure that OpenSSL support was marked as <code>YES</code>. Otherwise, check with your distribution for instructions on installing the headers for OpenSSL.</p>
3038 <p>Next, you can build the CPython binary by running the generated <code>Makefile</code>:</p>
3039 <div class="highlight sh"><pre><span></span><span class="gp">$</span> make -j2 -s
3040 </pre></div>
3041
3042 <p>During the build, you may receive some errors, and in the summary, it will notify you that not all packages could be built. That’s okay if you aren’t planning on developing against those packages. If you are, then check out the <a href="https://devguide.python.org/">dev guide</a> website for more information.</p>
3043 <p>The build will take a few minutes and generate a binary called <code>python</code>. This is the debug binary of CPython. Execute <code>./python</code> to see a working REPL:</p>
3044 <div class="highlight sh"><pre><span></span><span class="gp">$</span> ./python
3045 <span class="go">Python 3.8.0b4 (tags/v3.8.0b4:d93605de72, Aug 30 2019, 10:00:03) </span>
3046 <span class="go">[Clang 10.0.1 (clang-1001.0.46.4)] on darwin</span>
3047 <span class="go">Type "help", "copyright", "credits" or "license" for more information.</span>
3048 <span class="gp">></span>>>
3049 </pre></div>
3050
3051 <h3 id="compiling-cpython-windows">Compiling CPython (Windows)</h3>
3052 <p>Inside the PC folder is a Visual Studio project file for building and exploring CPython. To use this, you need to have Visual Studio installed on your PC.</p>
3053 <p>The newest version of Visual Studio, Visual Studio 2019, makes it easier to work with Python and the CPython source code, so it is recommended for use in this tutorial. If you already have Visual Studio 2017 installed, that would also work fine.</p>
3054 <p>None of the paid features are required for compiling CPython or this tutorial. You can use the Community edition of Visual Studio, which is available for free from <a href="https://visualstudio.microsoft.com/vs/">Microsoft’s Visual Studio website</a>.</p>
3055 <p>Once you’ve downloaded the installer, you’ll be asked to select which components you want to install. The bare minimum for this tutorial is:</p>
3056 <ul>
3057 <li>The <strong>Python Development</strong> workload</li>
3058 <li>The optional <strong>Python native development tools</strong></li>
3059 <li>Python 3 64-bit (3.7.2) (can be deselected if you already have Python 3.7 installed)</li>
3060 </ul>
3061 <p>Any other optional features can be deselected if you want to be more conscientious with disk space:</p>
3062 <p><a href="https://files.realpython.com/media/Screen_Shot_2019-08-22_at_2.47.23_pm.5e8682a89503.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/Screen_Shot_2019-08-22_at_2.47.23_pm.5e8682a89503.png" width="2504" height="1260" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screen_Shot_2019-08-22_at_2.47.23_pm.5e8682a89503.png&w=626&sig=86eb9f82580a69f533983087ba0fa4faf0d5bf96 626w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screen_Shot_2019-08-22_at_2.47.23_pm.5e8682a89503.png&w=1252&sig=fe2157486f81073eabe043b3c441af23dd67b78a 1252w, https://files.realpython.com/media/Screen_Shot_2019-08-22_at_2.47.23_pm.5e8682a89503.png 2504w" sizes="75vw" alt="Visual Studio Options Window"/></a></p>
3063 <p>The installer will then download and install all of the required components. The installation could take an hour, so you may want to read on and come back to this section.</p>
3064 <p>Once the installer has completed, click the <em>Launch</em> button to start Visual Studio. You will be prompted to sign in. If you have a Microsoft account you can log in, or skip that step.</p>
3065 <p>Once Visual Studio starts, you will be prompted to Open a Project. A shortcut to getting started with the Git configuration and cloning CPython is to choose the <em>Clone or check out code</em> option:</p>
3066 <p><a href="https://files.realpython.com/media/Capture3.e19765d74ec4.PNG" target="_blank"><img class="img-fluid mx-auto d-block w-50" src="https://files.realpython.com/media/Capture3.e19765d74ec4.PNG" width="2048" height="1420" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Capture3.e19765d74ec4.PNG&w=512&sig=e475e2a09cd780894f108850f91736c53ac95f27 512w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Capture3.e19765d74ec4.PNG&w=1024&sig=47d66a316c2546c9787e5af4cb4a9a006dca936d 1024w, https://files.realpython.com/media/Capture3.e19765d74ec4.PNG 2048w" sizes="75vw" alt="Choosing a Project Type in Visual Studio"/></a></p>
3067 <p>For the project URL, type <code>https://github.com/python/cpython</code> to clone:</p>
3068 <p><a href="https://files.realpython.com/media/Capture4.ea01418a971c.PNG" target="_blank"><img class="img-fluid mx-auto d-block w-50" src="https://files.realpython.com/media/Capture4.ea01418a971c.PNG" width="2048" height="1420" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Capture4.ea01418a971c.PNG&w=512&sig=47ed81b234652446a4f0e77e2a3f70e0074ac222 512w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Capture4.ea01418a971c.PNG&w=1024&sig=de393823fe557847f295d040573bd061b2ccd557 1024w, https://files.realpython.com/media/Capture4.ea01418a971c.PNG 2048w" sizes="75vw" alt="Cloning projects in Visual Studio"/></a></p>
3069 <p>Visual Studio will then download a copy of CPython from GitHub using the version of Git bundled with Visual Studio. This step also saves you the hassle of having to install Git on Windows. The download may take 10 minutes.</p>
3070 <p>Once the project has downloaded, you need to point it to the <strong><code>pcbuild</code></strong> Solution file, by clicking on <em>Solutions and Projects</em> and selecting <code>pcbuild.sln</code>:</p>
3071 <p><a href="https://files.realpython.com/media/Capture6.3d06a62b8e87.PNG" target="_blank"><img class="img-fluid mx-auto d-block border w-50" src="https://files.realpython.com/media/Capture6.3d06a62b8e87.PNG" width="863" height="565" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Capture6.3d06a62b8e87.PNG&w=215&sig=0d4c23758a58848ea65b74514aca9e15a72748fa 215w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Capture6.3d06a62b8e87.PNG&w=431&sig=05f0ebe121ef54abf51bcdb55fba695d28656868 431w, https://files.realpython.com/media/Capture6.3d06a62b8e87.PNG 863w" sizes="75vw" alt="Selecting a solution"/></a></p>
3072 <p>When the solution is loaded, it will prompt you to retarget the project’s inside the solution to the version of the C/C++ compiler you have installed. Visual Studio will also target the version of the Windows SDK you have installed.</p>
3073 <p>Ensure that you change the Windows SDK version to the newest installed version and the platform toolset to the latest version. If you missed this window, you can right-click on the Solution in the <em>Solutions and Projects</em> window and click <em>Retarget Solution</em>.</p>
3074 <p>Once this is complete, you need to download some source files to be able to build the whole CPython package. Inside the <code>PCBuild</code> folder there is a <code>.bat</code> file that automates this for you. <a href="https://www.youtube.com/watch?v=bgSSJQolR0E">Open up a command-line prompt inside</a> the downloaded <code>PCBuild</code> and run <code>get_externals.bat</code>:</p>
3075 <div class="highlight sh"><pre><span></span><span class="go"> > get_externals.bat</span>
3076 <span class="go">Using py -3.7 (found 3.7 with py.exe)</span>
3077 <span class="go">Fetching external libraries...</span>
3078 <span class="go">Fetching bzip2-1.0.6...</span>
3079 <span class="go">Fetching sqlite-3.21.0.0...</span>
3080 <span class="go">Fetching xz-5.2.2...</span>
3081 <span class="go">Fetching zlib-1.2.11...</span>
3082 <span class="go">Fetching external binaries...</span>
3083 <span class="go">Fetching openssl-bin-1.1.0j...</span>
3084 <span class="go">Fetching tcltk-8.6.9.0...</span>
3085 <span class="go">Finished.</span>
3086 </pre></div>
3087
3088 <p>Next, back within Visual Studio, build CPython by pressing <span class="keys"><kbd class="key-control">Ctrl</kbd><span>+</span><kbd class="key-shift">Shift</kbd><span>+</span><kbd class="key-b">B</kbd></span>, or choosing <em>Build Solution</em> from the top menu. If you receive any errors about the Windows SDK being missing, make sure you set the right targeting settings in the <em>Retarget Solution</em> window. You should also see <em>Windows Kits</em> inside your Start Menu, and <em>Windows Software Development Kit</em> inside of that menu.</p>
3089 <p>The build stage could take 10 minutes or more for the first time. Once the build is completed, you may see a few warnings that you can ignore and eventual completion.</p>
3090 <p>To start the debug version of CPython, press <span class="keys"><kbd class="key-f5">F5</kbd></span> and CPython will start in Debug mode straight into the REPL:</p>
3091 <p><a href="https://files.realpython.com/media/Capture8.967a3606daf0.PNG" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/Capture8.967a3606daf0.PNG" width="3360" height="2100" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Capture8.967a3606daf0.PNG&w=840&sig=1822fde8ffe6946fc91e47dd8975aa23add8b23a 840w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Capture8.967a3606daf0.PNG&w=1680&sig=a093e68320ad548384c585840002e4294ab4bf94 1680w, https://files.realpython.com/media/Capture8.967a3606daf0.PNG 3360w" sizes="75vw" alt="CPython debugging Windows"/></a></p>
3092 <p>Once this is completed, you can run the Release build by changing the build configuration from <em>Debug</em> to <em>Release</em> on the top menu bar and rerunning Build Solution again.
3093 You now have both Debug and Release versions of the CPython binary within <code>PCBuild\win32\</code>.</p>
3094 <p>You can set up Visual Studio to be able to open a REPL with either the Release or Debug build by choosing <em><code>Tools</code>-><code>Python</code>-><code>Python Environments</code></em> from the top menu:</p>
3095 <p><a href="https://files.realpython.com/media/Environments.96a819ecf0b3.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/Environments.96a819ecf0b3.png" width="3360" height="2033" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Environments.96a819ecf0b3.png&w=840&sig=f8dd9b3b31d44c25cbe06d56078b0462cb0fa753 840w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Environments.96a819ecf0b3.png&w=1680&sig=e8ca69be87363b62ccda44bcf53bedd4e4e320c2 1680w, https://files.realpython.com/media/Environments.96a819ecf0b3.png 3360w" sizes="75vw" alt="Choosing Python environments"/></a></p>
3096 <p>Then click <em>Add Environment</em> and then target the Debug or Release binary. The Debug binary will end in <code>_d.exe</code>, for example, <code>python_d.exe</code> and <code>pythonw_d.exe</code>. You will most likely want to use the debug binary as it comes with Debugging support in Visual Studio and will be useful for this tutorial.</p>
3097 <p>In the Add Environment window, target the <code>python_d.exe</code> file as the interpreter inside the <code>PCBuild/win32</code> and the <code>pythonw_d.exe</code> as the windowed interpreter:</p>
3098 <p><a href="https://files.realpython.com/media/environment3.d33858c1f6aa.PNG" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/environment3.d33858c1f6aa.PNG" width="2048" height="1352" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/environment3.d33858c1f6aa.PNG&w=512&sig=ffc6b359ac60d689f40f98233466cef35354d239 512w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/environment3.d33858c1f6aa.PNG&w=1024&sig=d881ce7ecabc62624fb2a5154102a5be2ff6a4db 1024w, https://files.realpython.com/media/environment3.d33858c1f6aa.PNG 2048w" sizes="75vw" alt="Adding an environment in VS2019"/></a></p>
3099 <p>Now, you can start a REPL session by clicking <em>Open Interactive Window</em> in the Python Environments window and you will see the REPL for the compiled version of Python:</p>
3100 <p><a href="https://files.realpython.com/media/environment4.7c9eade3b74e.PNG" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/environment4.7c9eade3b74e.PNG" width="3360" height="2033" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/environment4.7c9eade3b74e.PNG&w=840&sig=9e384e72bcfdebb39fe6dc23f21c239a0895ad51 840w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/environment4.7c9eade3b74e.PNG&w=1680&sig=be56aac5b9baffc9b8b0de4701527954ed32ef53 1680w, https://files.realpython.com/media/environment4.7c9eade3b74e.PNG 3360w" sizes="75vw" alt="Python Environment REPL"/></a></p>
3101 <p>During this tutorial there will be REPL sessions with example commands. I encourage you to use the Debug binary to run these REPL sessions in case you want to put in any breakpoints within the code.</p>
3102 <p>Lastly, to make it easier to navigate the code, in the Solution View, click on the toggle button next to the Home icon to switch to Folder view:</p>
3103 <p><a href="https://files.realpython.com/media/environments5.6462694398e3.6fb872a5f57d.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/environments5.6462694398e3.6fb872a5f57d.png" width="1231" height="692" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/environments5.6462694398e3.6fb872a5f57d.png&w=307&sig=14bf2b50bd86dfabedc3ee1ec75a60ebccd574c4 307w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/environments5.6462694398e3.6fb872a5f57d.png&w=615&sig=0d5644853d8c429ccc78fc1bced16d8f571f555d 615w, https://files.realpython.com/media/environments5.6462694398e3.6fb872a5f57d.png 1231w" sizes="75vw" alt="Switching Environment Mode"/></a></p>
3104 <p>Now you have a version of CPython compiled and ready to go, let’s find out how the CPython compiler works.</p>
3105 <h3 id="what-does-a-compiler-do">What Does a Compiler Do?</h3>
3106 <p>The purpose of a compiler is to convert one language into another. Think of a compiler like a translator. You would hire a translator to listen to you speaking in English and then speak in Japanese:</p>
3107 <p><a href="https://files.realpython.com/media/t.38be306a7e83.png" target="_blank"><img class="img-fluid mx-auto d-block w-75" src="https://files.realpython.com/media/t.38be306a7e83.png" width="960" height="540" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/t.38be306a7e83.png&w=240&sig=2ad6eec49af1eaba79b83c099925e01a970d5efd 240w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/t.38be306a7e83.png&w=480&sig=033587107c8dceca7bcc292527a4c815fffb0b8d 480w, https://files.realpython.com/media/t.38be306a7e83.png 960w" sizes="75vw" alt="Translating from English to Japanese"/></a></p>
3108 <p>Some compilers will compile into a low-level machine code which can be executed directly on a system. Other compilers will compile into an intermediary language, to be executed by a virtual machine.</p>
3109 <p>One important decision to make when choosing a compiler is the system portability requirements. <a href="https://en.wikipedia.org/wiki/Java_bytecode">Java</a> and <a href="https://en.wikipedia.org/wiki/Common_Language_Runtime">.NET CLR</a> will compile into an Intermediary Language so that the compiled code is portable across multiple systems architectures. C, Go, C++, and Pascal will compile into a low-level executable that will only work on systems similar to the one it was compiled. </p>
3110 <p>Because Python applications are typically distributed as source code, the role of the Python runtime is to convert the Python source code and execute it in one step. Internally, the CPython runtime does compile your code. A popular misconception is that Python is an interpreted language. It is actually compiled.</p>
3111 <p>Python code is not compiled into machine-code. It is compiled into a special low-level intermediary language called <strong>bytecode</strong> that only CPython understands. This code is stored in <code>.pyc</code> files in a hidden directory and cached for execution. If you run the same Python application twice without changing the source code, it’ll always be much faster the second time. This is because it loads the compiled bytecode and executes it directly.</p>
3112 <h3 id="why-is-cpython-written-in-c-and-not-python">Why Is CPython Written in C and Not Python?</h3>
3113 <p>The <strong>C</strong> in CPython is a reference to the C programming language, implying that this Python distribution is written in the C language.</p>
3114 <p>This statement is largely true: the compiler in CPython is written in pure C. However, many of the standard library modules are written in pure Python or a combination of C and Python.</p>
3115 <p><strong>So why is CPython written in C and not Python?</strong></p>
3116 <p>The answer is located in how compilers work. There are two types of compiler:</p>
3117 <ol>
3118 <li><strong><a href="https://en.wikipedia.org/wiki/Self-hosting">Self-hosted compilers</a></strong> are compilers written in the language they compile, such as the Go compiler.</li>
3119 <li><strong><a href="https://en.wikipedia.org/wiki/Source-to-source_compiler">Source-to-source compilers</a></strong> are compilers written in another language that already have a compiler.</li>
3120 </ol>
3121 <p>If you’re writing a new programming language from scratch, you need an executable application to compile your compiler! You need a compiler to execute anything, so when new languages are developed, they’re often written first in an older, more established language.</p>
3122 <p>A good example would be the Go programming language. The first Go compiler was written in C, then once Go could be compiled, the compiler was rewritten in Go. </p>
3123 <p>CPython kept its C heritage: many of the standard library modules, like the <code>ssl</code> module or the <code>sockets</code> module, are written in C to access low-level operating system APIs.
3124 The APIs in the Windows and Linux kernels for <a href="https://realpython.com/python-sockets/">creating network sockets</a>, <a href="https://realpython.com/working-with-files-in-python/">working with the filesystem</a> or <a href="https://realpython.com/python-gui-with-wxpython/">interacting with the display</a> are all written in C. It made sense for Python’s extensibility layer to be focused on the C language. Later in this article, we will cover the Python Standard Library and the C modules.</p>
3125 <p>There is a Python compiler written in Python called <a href="https://pypy.org/">PyPy</a>. PyPy’s logo is an <a href="https://en.wikipedia.org/wiki/Ouroboros">Ouroboros</a> to represent the self-hosting nature of the compiler.</p>
3126 <p>Another example of a cross-compiler for Python is <a href="https://www.jython.org/">Jython</a>. Jython is written in Java and compiles from Python source code into Java bytecode. In the same way that CPython makes it easy to import C libraries and use them from Python, Jython makes it easy to import and reference Java modules and classes.</p>
3127 <h3 id="the-python-language-specification">The Python Language Specification</h3>
3128 <p>Contained within the CPython source code is the definition of the Python language. This is the reference specification used by all the Python interpreters.</p>
3129 <p>The specification is in both human-readable and machine-readable format. Inside the documentation is a detailed explanation of the Python language, what is allowed, and how each statement should behave.</p>
3130 <h4 id="documentation">Documentation</h4>
3131 <p>Located inside the <code>Doc/reference</code> directory are <a href="http://docutils.sourceforge.net/rst.html">reStructuredText</a> explanations of each of the features in the Python language. This forms the official Python reference guide on <a href="https://docs.python.org/3/reference/">docs.python.org</a>.</p>
3132 <p>Inside the directory are the files you need to understand the whole language, structure, and keywords:</p>
3133 <div class="highlight"><pre><span></span>cpython/Doc/reference
3134 |
3135 βββ compound_stmts.rst
3136 βββ datamodel.rst
3137 βββ executionmodel.rst
3138 βββ expressions.rst
3139 βββ grammar.rst
3140 βββ import.rst
3141 βββ index.rst
3142 βββ introduction.rst
3143 βββ lexical_analysis.rst
3144 βββ simple_stmts.rst
3145 βββ toplevel_components.rst
3146 </pre></div>
3147
3148 <p>Inside <code>compound_stmts.rst</code>, the documentation for compound statements, you can see a simple example defining the <code>with</code> statement.</p>
3149 <p>The <code>with</code> statement can be used in multiple ways in Python, the simplest being the <a href="https://dbader.org/blog/python-context-managers-and-with-statement">instantiation of a context-manager</a> and a nested block of code:</p>
3150 <div class="highlight python"><pre><span></span><span class="k">with</span> <span class="n">x</span><span class="p">():</span>
3151 <span class="o">...</span>
3152 </pre></div>
3153
3154 <p>You can assign the result to a variable using the <code>as</code> keyword:</p>
3155 <div class="highlight python"><pre><span></span><span class="k">with</span> <span class="n">x</span><span class="p">()</span> <span class="k">as</span> <span class="n">y</span><span class="p">:</span>
3156 <span class="o">...</span>
3157 </pre></div>
3158
3159 <p>You can also chain context managers together with a comma:</p>
3160 <div class="highlight python"><pre><span></span><span class="k">with</span> <span class="n">x</span><span class="p">()</span> <span class="k">as</span> <span class="n">y</span><span class="p">,</span> <span class="n">z</span><span class="p">()</span> <span class="k">as</span> <span class="n">jk</span><span class="p">:</span>
3161 <span class="o">...</span>
3162 </pre></div>
3163
3164 <p>Next, we’ll explore the computer-readable documentation of the Python language.</p>
3165 <h4 id="grammar">Grammar</h4>
3166 <p>The documentation contains the human-readable specification of the language, and the machine-readable specification is housed in a single file, <a href="https://github.com/python/cpython/blob/master/Grammar/Grammar"><code>Grammar/Grammar</code></a>. </p>
3167 <p>The Grammar file is written in a context-notation called <a href="https://en.m.wikipedia.org/wiki/Backus%E2%80%93Naur_form">Backus-Naur Form (BNF)</a>. BNF is not specific to Python and is often used as the notation for grammars in many other languages.</p>
3168 <p>The concept of grammatical structure in a programming language is inspired by <a href="https://en.wikipedia.org/wiki/Syntactic_Structures">Noam Chomsky’s work on Syntactic Structures</a> in the 1950s!</p>
3169 <p>Python’s grammar file uses the Extended-BNF (EBNF) specification with regular-expression syntax. So, in the grammar file you can use:</p>
3170 <ul>
3171 <li><strong><code>*</code></strong> for repetition</li>
3172 <li><strong><code>+</code></strong> for at-least-once repetition</li>
3173 <li><strong><code>[]</code></strong> for optional parts</li>
3174 <li><strong><code>|</code></strong> for alternatives</li>
3175 <li><strong><code>()</code></strong> for grouping</li>
3176 </ul>
3177 <p>If you search for the <code>with</code> statement in the grammar file, at around line 80 you’ll see the definitions for the <code>with</code> statement:</p>
3178 <div class="highlight text"><pre><span></span>with_stmt: 'with' with_item (',' with_item)* ':' suite
3179 with_item: test ['as' expr]
3180 </pre></div>
3181
3182 <p>Anything in quotes is a string literal, which is how keywords are defined. So the <code>with_stmt</code> is specified as:</p>
3183 <ol>
3184 <li>Starting with the word <code>with</code></li>
3185 <li>Followed by a <code>with_item</code>, which is a <code>test</code> and (optionally), the word <code>as</code>, and an expression</li>
3186 <li>Following one or many items, each separated by a comma</li>
3187 <li>Ending with a <code>:</code></li>
3188 <li>Followed by a <code>suite</code></li>
3189 </ol>
3190 <p>There are references to some other definitions in these two lines:</p>
3191 <ul>
3192 <li><strong><code>suite</code></strong> refers to a block of code with one or multiple statements</li>
3193 <li><strong><code>test</code></strong> refers to a simple statement that is evaluated</li>
3194 <li><strong><code>expr</code></strong> refers to a simple expression</li>
3195 </ul>
3196 <p>If you want to explore those in detail, the whole of the Python grammar is defined in this single file.</p>
3197 <p>If you want to see a recent example of how grammar is used, in PEP 572 the <strong>colon equals</strong> operator was added to the grammar file in <a href="https://github.com/python/cpython/commit/8f59ee01be3d83d5513a9a3f654a237d77d80d9a#diff-cb0b9d6312c0d67f6d4aa1966766cedd">this Git commit</a>.</p>
3198 <h4 id="using-pgen">Using <code>pgen</code></h4>
3199 <p>The grammar file itself is never used by the Python compiler. Instead, a parser table created by a tool called <code>pgen</code> is used. <code>pgen</code> reads the grammar file and converts it into a parser table. If you make changes to the grammar file, you must regenerate the parser table and recompile Python.</p>
3200 <div class="alert alert-primary" role="alert">
3201 <p><strong>Note:</strong> The <code>pgen</code> application was rewritten in Python 3.8 from C to <a href="https://github.com/python/cpython/blob/master/Parser/pgen/pgen.py">pure Python</a>.</p>
3202 </div>
3203 <p>To see <code>pgen</code> in action, let’s change part of the Python grammar. Around line 51 you will see the definition of a <code>pass</code> statement:</p>
3204 <div class="highlight text"><pre><span></span>pass_stmt: 'pass'
3205 </pre></div>
3206
3207 <p>Change that line to accept the keyword <code>'pass'</code> or <code>'proceed'</code> as keywords:</p>
3208 <div class="highlight text"><pre><span></span>pass_stmt: 'pass' | 'proceed'
3209 </pre></div>
3210
3211 <p>Now you need to rebuild the grammar files.
3212 On macOS and Linux, run <code>make regen-grammar</code> to run <code>pgen</code> over the altered grammar file. For Windows, there is no officially supported way of running <code>pgen</code>. However, you can clone <a href="https://github.com/tonybaloney/cpython/tree/pcbuildregen">my fork</a> and run <code>build.bat --regen</code> from within the <code>PCBuild</code> directory.</p>
3213 <p>You should see an output similar to this, showing that the new <code>Include/graminit.h</code> and <code>Python/graminit.c</code> files have been generated:</p>
3214 <div class="highlight text"><pre><span></span># Regenerate Doc/library/token-list.inc from Grammar/Tokens
3215 # using Tools/scripts/generate_token.py
3216 ...
3217 python3 ./Tools/scripts/update_file.py ./Include/graminit.h ./Include/graminit.h.new
3218 python3 ./Tools/scripts/update_file.py ./Python/graminit.c ./Python/graminit.c.new
3219 </pre></div>
3220
3221 <div class="alert alert-primary" role="alert">
3222 <p><strong>Note:</strong> <code>pgen</code> works by converting the EBNF statements into a <a href="https://en.wikipedia.org/wiki/Nondeterministic_finite_automaton">Non-deterministic Finite Automaton (NFA)</a>, which is then turned into a <a href="https://en.wikipedia.org/wiki/Deterministic_finite_automaton">Deterministic Finite Automaton (DFA)</a>.
3223 The DFAs are used by the parser as parsing tables in a special way that’s unique to CPython. This technique was <a href="http://infolab.stanford.edu/~ullman/dragon/slides1.pdf">formed at Stanford University</a> and developed in the 1980s, just before the advent of Python.</p>
3224 </div>
3225 <p>With the regenerated parser tables, you need to recompile CPython to see the new syntax. Use the same compilation steps you used earlier for your operating system.</p>
3226 <p>If the code compiled successfully, you can execute your new CPython binary and start a REPL.</p>
3227 <p>In the REPL, you can now try defining a function and instead of using the <code>pass</code> statement, use the <code>proceed</code> keyword alternative that you compiled into the Python grammar:</p>
3228 <div class="highlight text"><pre><span></span>Python 3.8.0b4 (tags/v3.8.0b4:d93605de72, Aug 30 2019, 10:00:03)
3229 [Clang 10.0.1 (clang-1001.0.46.4)] on darwin
3230 Type "help", "copyright", "credits" or "license" for more information.
3231 >>> def example():
3232 ... proceed
3233 ...
3234 >>> example()
3235 </pre></div>
3236
3237 <p>Well done! You’ve changed the CPython syntax and compiled your own version of CPython. Ship it!</p>
3238 <p>Next, we’ll explore tokens and their relationship to grammar.</p>
3239 <h4 id="tokens">Tokens</h4>
3240 <p>Alongside the grammar file in the <code>Grammar</code> folder is a <a href="https://github.com/python/cpython/blob/master/Grammar/Tokens"><code>Tokens</code></a> file, which contains each of the unique types found as a leaf node in a parse tree. We will cover parser trees in depth later.
3241 Each token also has a name and a generated unique ID. The names are used to make it simpler to refer to in the tokenizer.</p>
3242 <div class="alert alert-primary" role="alert">
3243 <p><strong>Note:</strong> The <code>Tokens</code> file is a new feature in Python 3.8.</p>
3244 </div>
3245 <p>For example, the left parenthesis is called <code>LPAR</code>, and semicolons are called <code>SEMI</code>. You’ll see these tokens later in the article:</p>
3246 <div class="highlight text"><pre><span></span>LPAR '('
3247 RPAR ')'
3248 LSQB '['
3249 RSQB ']'
3250 COLON ':'
3251 COMMA ','
3252 SEMI ';'
3253 </pre></div>
3254
3255 <p>As with the <code>Grammar</code> file, if you change the <code>Tokens</code> file, you need to run <code>pgen</code> again. </p>
3256 <p>To see tokens in action, you can use the <code>tokenize</code> module in CPython. Create a simple Python script called <code>test_tokens.py</code>:</p>
3257 <div class="highlight python"><pre><span></span><span class="c1"># Hello world!</span>
3258 <span class="k">def</span> <span class="nf">my_function</span><span class="p">():</span>
3259 <span class="n">proceed</span>
3260 </pre></div>
3261
3262 <div class="alert alert-primary" role="alert">
3263 <p>For the rest of this tutorial, <code>./python.exe</code> will refer to the compiled version of CPython. However, the actual command will depend on your system.</p>
3264 <p>For Windows:</p>
3265 <div class="highlight sh"><pre><span></span><span class="go"> > python.exe</span>
3266 </pre></div>
3267
3268 <p>For Linux:</p>
3269 <div class="highlight sh"><pre><span></span><span class="go"> > ./python</span>
3270 </pre></div>
3271
3272 <p>For macOS:</p>
3273 <div class="highlight sh"><pre><span></span><span class="go"> > ./python.exe</span>
3274 </pre></div>
3275
3276 </div>
3277 <p>Then pass this file through a module built into the standard library called <code>tokenize</code>. You will see the list of tokens, by line and character. Use the <code>-e</code> flag to output the exact token name:</p>
3278 <div class="highlight sh"><pre><span></span><span class="gp">$</span> ./python.exe -m tokenize -e test_tokens.py
3279
3280 <span class="go">0,0-0,0: ENCODING 'utf-8' </span>
3281 <span class="go">1,0-1,14: COMMENT '# Hello world!'</span>
3282 <span class="go">1,14-1,15: NL '\n' </span>
3283 <span class="go">2,0-2,3: NAME 'def' </span>
3284 <span class="go">2,4-2,15: NAME 'my_function' </span>
3285 <span class="go">2,15-2,16: LPAR '(' </span>
3286 <span class="go">2,16-2,17: RPAR ')' </span>
3287 <span class="go">2,17-2,18: COLON ':' </span>
3288 <span class="go">2,18-2,19: NEWLINE '\n' </span>
3289 <span class="go">3,0-3,3: INDENT ' ' </span>
3290 <span class="go">3,3-3,7: NAME 'proceed' </span>
3291 <span class="go">3,7-3,8: NEWLINE '\n' </span>
3292 <span class="go">4,0-4,0: DEDENT '' </span>
3293 <span class="go">4,0-4,0: ENDMARKER '' </span>
3294 </pre></div>
3295
3296 <p>In the output, the first column is the range of the line/column coordinates, the second column is the name of the token, and the final column is the value of the token.</p>
3297 <p>In the output, the <code>tokenize</code> module has implied some tokens that were not in the file. The <code>ENCODING</code> token for <code>utf-8</code>, and a blank line at the end, giving <code>DEDENT</code> to close the function declaration and an <code>ENDMARKER</code> to end the file.</p>
3298 <p>It is best practice to have a blank line at the end of your Python source files. If you omit it, CPython adds it for you, with a tiny performance penalty.</p>
3299 <p>The <code>tokenize</code> module is written in pure Python and is located in <a href="https://github.com/python/cpython/blob/master/Lib/tokenize.py"><code>Lib/tokenize.py</code></a> within the CPython source code.</p>
3300 <div class="alert alert-primary" role="alert">
3301 <p><strong>Important:</strong> There are two tokenizers in the CPython source code: one written in Python, demonstrated here, and another written in C.
3302 The tokenizer written in Python is meant as a utility, and the one written in C is used by the Python compiler. They have identical output and behavior. The version written in C is designed for performance and the module in Python is designed for debugging.</p>
3303 </div>
3304 <p>To see a verbose readout of the C tokenizer, you can run Python with the <code>-d</code> flag. Using the <code>test_tokens.py</code> script you created earlier, run it with the following:</p>
3305 <div class="highlight sh"><pre><span></span><span class="gp">$</span> ./python.exe -d test_tokens.py
3306
3307 <span class="go">Token NAME/'def' ... It's a keyword</span>
3308 <span class="go"> DFA 'file_input', state 0: Push 'stmt'</span>
3309 <span class="go"> DFA 'stmt', state 0: Push 'compound_stmt'</span>
3310 <span class="go"> DFA 'compound_stmt', state 0: Push 'funcdef'</span>
3311 <span class="go"> DFA 'funcdef', state 0: Shift.</span>
3312 <span class="go">Token NAME/'my_function' ... It's a token we know</span>
3313 <span class="go"> DFA 'funcdef', state 1: Shift.</span>
3314 <span class="go">Token LPAR/'(' ... It's a token we know</span>
3315 <span class="go"> DFA 'funcdef', state 2: Push 'parameters'</span>
3316 <span class="go"> DFA 'parameters', state 0: Shift.</span>
3317 <span class="go">Token RPAR/')' ... It's a token we know</span>
3318 <span class="go"> DFA 'parameters', state 1: Shift.</span>
3319 <span class="go"> DFA 'parameters', state 2: Direct pop.</span>
3320 <span class="go">Token COLON/':' ... It's a token we know</span>
3321 <span class="go"> DFA 'funcdef', state 3: Shift.</span>
3322 <span class="go">Token NEWLINE/'' ... It's a token we know</span>
3323 <span class="go"> DFA 'funcdef', state 5: [switch func_body_suite to suite] Push 'suite'</span>
3324 <span class="go"> DFA 'suite', state 0: Shift.</span>
3325 <span class="go">Token INDENT/'' ... It's a token we know</span>
3326 <span class="go"> DFA 'suite', state 1: Shift.</span>
3327 <span class="hll"><span class="go">Token NAME/'proceed' ... It's a keyword</span>
3328 </span><span class="go"> DFA 'suite', state 3: Push 'stmt'</span>
3329 <span class="go">...</span>
3330 <span class="go"> ACCEPT.</span>
3331 </pre></div>
3332
3333 <p>In the output, you can see that it highlighted <code>proceed</code> as a keyword. In the next chapter, we’ll see how executing the Python binary gets to the tokenizer and what happens from there to execute your code.</p>
3334 <p>Now that you have an overview of the Python grammar and the relationship between tokens and statements, there is a way to convert the <code>pgen</code> output into an interactive graph.</p>
3335 <p>Here is a screenshot of the Python 3.8a2 grammar:</p>
3336 <p><a href="https://files.realpython.com/media/Screen_Shot_2019-03-12_at_2.31.16_pm.f36c3e99b8b4.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/Screen_Shot_2019-03-12_at_2.31.16_pm.f36c3e99b8b4.png" width="3258" height="2248" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screen_Shot_2019-03-12_at_2.31.16_pm.f36c3e99b8b4.png&w=814&sig=21666b4228a46a6bcc7aeca6d5263e62a3aeb6d5 814w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screen_Shot_2019-03-12_at_2.31.16_pm.f36c3e99b8b4.png&w=1629&sig=01404157eddc09548f8cf7d4e995da9806c36fac 1629w, https://files.realpython.com/media/Screen_Shot_2019-03-12_at_2.31.16_pm.f36c3e99b8b4.png 3258w" sizes="75vw" alt="Python 3.8 DFA node graph"/></a></p>
3337 <p>The Python package used to generate this graph, <code>instaviz</code>, will be covered in a later chapter.</p>
3338 <h3 id="memory-management-in-cpython">Memory Management in CPython</h3>
3339 <p>Throughout this article, you will see references to a <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pyarena.c#L128"><code>PyArena</code></a> object. The arena is one of CPython’s memory management structures. The code is within <code>Python/pyarena.c</code> and contains a wrapper around C’s memory allocation and deallocation functions.</p>
3340 <p>In a traditionally written C program, the developer <em>should</em> allocate memory for data structures before writing into that data. This allocation marks the memory as belonging to the process with the operating system.</p>
3341 <p>It is also up to the developer to deallocate, or “free,” the allocated memory when its no longer being used and return it to the operating system’s block table of free memory.
3342 If a process allocates memory for a variable, say within a function or loop, when that function has completed, the memory is not automatically given back to the operating system in C. So if it hasn’t been explicitly deallocated in the C code, it causes a memory leak. The process will continue to take more memory each time that function runs until eventually, the system runs out of memory, and crashes!</p>
3343 <p>Python takes that responsibility away from the programmer and uses two algorithms: <a href="https://realpython.com/python-memory-management/">a reference counter and a garbage collector</a>.</p>
3344 <p>Whenever an interpreter is instantiated, a <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pyarena.c#L128"><code>PyArena</code></a> is created and attached one of the fields in the interpreter. During the lifecycle of a CPython interpreter, many arenas could be allocated. They are connected with a linked list. The arena stores a list of pointers to Python Objects as a <code>PyListObject</code>. Whenever a new Python object is created, a pointer to it is added using <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pyarena.c#L203"><code>PyArena_AddPyObject()</code></a>. This function call stores a pointer in the arena’s list, <code>a_objects</code>.</p>
3345 <div class="alert alert-primary" role="alert">
3346 <p>Even though Python doesn’t have pointers, there are some <a href="https://realpython.com/pointers-in-python/">interesting techniques</a> to simulate the behavior of pointers.</p>
3347 </div>
3348 <p>The <code>PyArena</code> serves a second function, which is to allocate and reference a list of raw memory blocks. For example, a <code>PyList</code> would need extra memory if you added thousands of additional values. The <code>PyList</code> object’s C code does not allocate memory directly. The object gets raw blocks of memory from the <code>PyArena</code> by calling <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pyarena.c#L180"><code>PyArena_Malloc()</code></a> from the <code>PyObject</code> with the required memory size. This task is completed by another abstraction in <code>Objects/obmalloc.c</code>. In the object allocation module, memory can be allocated, freed, and reallocated for a Python Object.</p>
3349 <p>A linked list of allocated blocks is stored inside the arena, so that when an interpreter is stopped, all managed memory blocks can be deallocated in one go using <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pyarena.c#L157"><code>PyArena_Free()</code></a>.</p>
3350 <p>Take the <code>PyListObject</code> example. If you were to <code>.append()</code> an object to the end of a Python list, you don’t need to reallocate the memory used in the existing list beforehand. The <code>.append()</code> method calls <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Objects/listobject.c#L36"><code>list_resize()</code></a> which handles memory allocation for lists. Each list object keeps a list of the amount of memory allocated. If the item you’re appending will fit inside the existing free memory, it is simply added. If the list needs more memory space, it is expanded. Lists are expanded in length as 0, 4, 8, 16, 25, 35, 46, 58, 72, 88.</p>
3351 <p><a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Objects/obmalloc.c#L618"><code>PyMem_Realloc()</code></a> is called to expand the memory allocated in a list. <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Objects/obmalloc.c#L618"><code>PyMem_Realloc()</code></a> is an API wrapper for <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Objects/obmalloc.c#L1913"><code>pymalloc_realloc()</code></a>.</p>
3352 <p>Python also has a special wrapper for the C call <code>malloc()</code>, which sets the max size of the memory allocation to help prevent buffer overflow errors (See <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Modules/overlapped.c#L28"><code>PyMem_RawMalloc()</code></a>).</p>
3353 <p>In summary: </p>
3354 <ul>
3355 <li>Allocation of raw memory blocks is done via <code>PyMem_RawAlloc()</code>.</li>
3356 <li>The pointers to Python objects are stored within the <code>PyArena</code>.</li>
3357 <li><code>PyArena</code> also stores a linked-list of allocated memory blocks.</li>
3358 </ul>
3359 <p>More information on the API is detailed on the <a href="https://docs.python.org/3/c-api/memory.html">CPython documentation</a>.</p>
3360 <h4 id="reference-counting">Reference Counting</h4>
3361 <p>To create a variable in Python, you have to assign a value to a <em>uniquely</em> named variable:</p>
3362 <div class="highlight python"><pre><span></span><span class="n">my_variable</span> <span class="o">=</span> <span class="mi">180392</span>
3363 </pre></div>
3364
3365 <p>Whenever a value is assigned to a variable in Python, the name of the variable is checked within the locals and globals scope to see if it already exists.</p>
3366 <p>Because <code>my_variable</code> is not already within the <code>locals()</code> or <code>globals()</code> dictionary, this new object is created, and the value is assigned as being the numeric constant <code>180392</code>.</p>
3367 <p>There is now one reference to <code>my_variable</code>, so the reference counter for <code>my_variable</code> is incremented by 1. </p>
3368 <p>You will see function calls <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Objects/object.c#L239"><code>Py_INCREF()</code></a> and <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Objects/object.c#L245"><code>Py_DECREF()</code></a> throughout the C source code for CPython. These functions increment and decrement the count of references to that object.</p>
3369 <p>References to an object are decremented when a variable falls outside of the scope in which it was declared. Scope in Python can refer to a function or method, a comprehension, or a lambda function. These are some of the more literal scopes, but there are many other implicit scopes, like passing variables to a function call.</p>
3370 <p>The handling of incrementing and decrementing references based on the language is built into the CPython compiler and the core execution loop, <code>ceval.c</code>, which we will cover in detail later in this article.</p>
3371 <p>Whenever <code>Py_DECREF()</code> is called, and the counter becomes 0, the <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Objects/obmalloc.c#L707"><code>PyObject_Free()</code></a> function is called. For that object <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pyarena.c#L157"><code>PyArena_Free()</code></a> is called for all of the memory that was allocated. </p>
3372 <h4 id="garbage-collection">Garbage Collection</h4>
3373 <p>How often does your garbage get collected? Weekly, or fortnightly? </p>
3374 <p>When you’re finished with something, you discard it and throw it in the trash. But that trash won’t get collected straight away. You need to wait for the garbage trucks to come and pick it up.</p>
3375 <p>CPython has the same principle, using a garbage collection algorithm. CPython’s garbage collector is enabled by default, happens in the background and works to deallocate memory that’s been used for objects which are no longer in use.</p>
3376 <p>Because the garbage collection algorithm is a lot more complex than the reference counter, it doesn’t happen all the time, otherwise, it would consume a huge amount of CPU resources. It happens periodically, after a set number of operations.</p>
3377 <p>CPython’s standard library comes with a Python module to interface with the arena and the garbage collector, the <code>gc</code> module. Here’s how to use the <code>gc</code> module in debug mode:</p>
3378 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">gc</span>
3379 <span class="gp">>>> </span><span class="n">gc</span><span class="o">.</span><span class="n">set_debug</span><span class="p">(</span><span class="n">gc</span><span class="o">.</span><span class="n">DEBUG_STATS</span><span class="p">)</span>
3380 </pre></div>
3381
3382 <p>This will print the statistics whenever the garbage collector is run.</p>
3383 <p>You can get the threshold after which the garbage collector is run by calling <code>get_threshold()</code>:</p>
3384 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">gc</span><span class="o">.</span><span class="n">get_threshold</span><span class="p">()</span>
3385 <span class="go">(700, 10, 10)</span>
3386 </pre></div>
3387
3388 <p>You can also get the current threshold counts:</p>
3389 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">gc</span><span class="o">.</span><span class="n">get_count</span><span class="p">()</span>
3390 <span class="go">(688, 1, 1)</span>
3391 </pre></div>
3392
3393 <p>Lastly, you can run the collection algorithm manually:</p>
3394 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">gc</span><span class="o">.</span><span class="n">collect</span><span class="p">()</span>
3395 <span class="go">24</span>
3396 </pre></div>
3397
3398 <p>This will call <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Modules/gcmodule.c#L987"><code>collect()</code></a> inside the <code>Modules/gcmodule.c</code> file which contains the implementation of the garbage collector algorithm.</p>
3399 <h3 id="conclusion">Conclusion</h3>
3400 <p>In Part 1, you covered the structure of the source code repository, how to compile from source, and the Python language specification. These core concepts will be critical in Part 2 as you dive deeper into the Python interpreter process.</p>
3401 <h2 h1="h1" id="part-2-the-python-interpreter-process">Part 2: The Python Interpreter Process</h2>
3402 <p>Now that you’ve seen the Python grammar and memory management, you can follow the process from typing <code>python</code> to the part where your code is executed.</p>
3403 <p>There are five ways the <code>python</code> binary can be called:</p>
3404 <ol>
3405 <li>To run a single command with <code>-c</code> and a Python command</li>
3406 <li>To start a module with <code>-m</code> and the name of a module</li>
3407 <li>To run a file with the filename</li>
3408 <li>To run the <code>stdin</code> input using a shell pipe</li>
3409 <li>To start the REPL and execute commands one at a time</li>
3410 </ol>
3411 <div class="alert alert-primary" role="alert">
3412 <p>Python has so many ways to execute scripts, it can be a little overwhelming. Darren Jones has put together a <a href="https://realpython.com/courses/running-python-scripts/">great course on running Python scripts</a> if you want to learn more.</p>
3413 </div>
3414 <p>The three source files you need to inspect to see this process are:</p>
3415 <ol>
3416 <li><strong><code>Programs/python.c</code></strong> is a simple entry point.</li>
3417 <li><strong><code>Modules/main.c</code></strong> contains the code to bring together the whole process, loading configuration, executing code and clearing up memory.</li>
3418 <li><strong><code>Python/initconfig.c</code></strong> loads the configuration from the system environment and merges it with any command-line flags.</li>
3419 </ol>
3420 <p>This diagram shows how each of those functions is called:</p>
3421 <p><a href="https://files.realpython.com/media/swim-lanes-chart-1.9fb3000aad85.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/swim-lanes-chart-1.9fb3000aad85.png" width="1046" height="851" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/swim-lanes-chart-1.9fb3000aad85.png&w=261&sig=8aa36cedaf32be0236896cce2c32b6c4c4ec7e05 261w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/swim-lanes-chart-1.9fb3000aad85.png&w=523&sig=a447b53d2afa96dfd204b9265c8b439bd71272d7 523w, https://files.realpython.com/media/swim-lanes-chart-1.9fb3000aad85.png 1046w" sizes="75vw" alt="Python run swim lane diagram"/></a></p>
3422 <p>The execution mode is determined from the configuration.</p>
3423 <div class="alert alert-primary" role="alert">
3424 <p><strong>The CPython source code style:</strong></p>
3425 <p>Similar to the <a href="https://realpython.com/courses/writing-beautiful-python-code-pep-8/">PEP8 style guide for Python code</a>, there is an <a href="https://www.python.org/dev/peps/pep-0007/">official style guide</a> for the CPython C code, designed originally in 2001 and updated for modern versions. </p>
3426 <p>There are some naming standards which help when navigating the source code:</p>
3427 <ul>
3428 <li>
3429 <p>Use a <code>Py</code> prefix for public functions, never for static functions. The <code>Py_</code> prefix is reserved for global service routines like <code>Py_FatalError</code>. Specific groups of routines (like specific object type APIs) use a longer prefix, such as <code>PyString_</code> for string functions.</p>
3430 </li>
3431 <li>
3432 <p>Public functions and variables use MixedCase with underscores, like this: <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Objects/object.c#L924"><code>PyObject_GetAttr</code></a>, <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Include/modsupport.h#L20"><code>Py_BuildValue</code></a>, <code>PyExc_TypeError</code>.</p>
3433 </li>
3434 <li>
3435 <p>Occasionally an “internal” function has to be visible to the loader. We use the <code>_Py</code> prefix for this, for example, <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Objects/object.c#L464"><code>_PyObject_Dump</code></a>.</p>
3436 </li>
3437 <li>
3438 <p>Macros should have a MixedCase prefix and then use upper case, for example <code>PyString_AS_STRING</code>, <code>Py_PRINT_RAW</code>.</p>
3439 </li>
3440 </ul>
3441 </div>
3442 <h3 id="establishing-runtime-configuration">Establishing Runtime Configuration</h3>
3443 <p><a href="https://files.realpython.com/media/swim-lanes-chart-1.9fb3000aad85.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/swim-lanes-chart-1.9fb3000aad85.png" width="1046" height="851" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/swim-lanes-chart-1.9fb3000aad85.png&w=261&sig=8aa36cedaf32be0236896cce2c32b6c4c4ec7e05 261w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/swim-lanes-chart-1.9fb3000aad85.png&w=523&sig=a447b53d2afa96dfd204b9265c8b439bd71272d7 523w, https://files.realpython.com/media/swim-lanes-chart-1.9fb3000aad85.png 1046w" sizes="75vw" alt="Python run swim lane diagram"/></a></p>
3444 <p>In the swimlanes, you can see that before any Python code is executed, the runtime first establishes the configuration.
3445 The configuration of the runtime is a data structure defined in <code>Include/cpython/initconfig.h</code> named <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Include/cpython/initconfig.h#L407"><code>PyConfig</code></a>.</p>
3446 <p>The configuration data structure includes things like:</p>
3447 <ul>
3448 <li>Runtime flags for various modes like debug and optimized mode</li>
3449 <li>The execution mode, such as whether a filename was passed, <code>stdin</code> was provided or a module name</li>
3450 <li>Extended option, specified by <code>-X <option></code></li>
3451 <li>Environment variables for runtime settings</li>
3452 </ul>
3453 <p>The configuration data is primarily used by the CPython runtime to enable and disable various features.</p>
3454 <p>Python also comes with several <a href="https://docs.python.org/3/using/cmdline.html">Command Line Interface Options</a>. In Python you can enable verbose mode with the <code>-v</code> flag. In verbose mode, Python will print messages to the screen when modules are loaded:</p>
3455 <div class="highlight sh"><pre><span></span><span class="gp">$</span> ./python.exe -v -c <span class="s2">"print('hello world')"</span>
3456
3457
3458 <span class="gp">#</span> installing zipimport hook
3459 <span class="go">import zipimport # builtin</span>
3460 <span class="gp">#</span> installed zipimport hook
3461 <span class="go">...</span>
3462 </pre></div>
3463
3464 <p>You will see a hundred lines or more with all the imports of your user site-packages and anything else in the system environment.</p>
3465 <p>You can see the definition of this flag within <code>Include/cpython/initconfig.h</code> inside the <code>struct</code> for <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Include/cpython/initconfig.h#L407"><code>PyConfig</code></a>:</p>
3466 <div class="highlight c"><pre><span></span><span class="cm">/* --- PyConfig ---------------------------------------------- */</span>
3467
3468 <span class="k">typedef</span> <span class="k">struct</span> <span class="p">{</span>
3469 <span class="kt">int</span> <span class="n">_config_version</span><span class="p">;</span> <span class="cm">/* Internal configuration version,</span>
3470 <span class="cm"> used for ABI compatibility */</span>
3471 <span class="kt">int</span> <span class="n">_config_init</span><span class="p">;</span> <span class="cm">/* _PyConfigInitEnum value */</span>
3472
3473 <span class="p">...</span>
3474
3475 <span class="cm">/* If greater than 0, enable the verbose mode: print a message each time a</span>
3476 <span class="cm"> module is initialized, showing the place (filename or built-in module)</span>
3477 <span class="cm"> from which it is loaded.</span>
3478
3479 <span class="cm"> If greater or equal to 2, print a message for each file that is checked</span>
3480 <span class="cm"> for when searching for a module. Also provides information on module</span>
3481 <span class="cm"> cleanup at exit.</span>
3482
3483 <span class="cm"> Incremented by the -v option. Set by the PYTHONVERBOSE environment</span>
3484 <span class="cm"> variable. If set to -1 (default), inherit Py_VerboseFlag value. */</span>
3485 <span class="kt">int</span> <span class="n">verbose</span><span class="p">;</span>
3486 </pre></div>
3487
3488 <p>In <code>Python/initconfig.c</code>, the logic for reading settings from environment variables and runtime command-line flags is established.</p>
3489 <p>In the <code>config_read_env_vars</code> function, the environment variables are read and used to assign the values for the configuration settings:</p>
3490 <div class="highlight c"><pre><span></span><span class="k">static</span> <span class="n">PyStatus</span>
3491 <span class="nf">config_read_env_vars</span><span class="p">(</span><span class="n">PyConfig</span> <span class="o">*</span><span class="n">config</span><span class="p">)</span>
3492 <span class="p">{</span>
3493 <span class="n">PyStatus</span> <span class="n">status</span><span class="p">;</span>
3494 <span class="kt">int</span> <span class="n">use_env</span> <span class="o">=</span> <span class="n">config</span><span class="o">-></span><span class="n">use_environment</span><span class="p">;</span>
3495
3496 <span class="cm">/* Get environment variables */</span>
3497 <span class="hll"> <span class="n">_Py_get_env_flag</span><span class="p">(</span><span class="n">use_env</span><span class="p">,</span> <span class="o">&</span><span class="n">config</span><span class="o">-></span><span class="n">parser_debug</span><span class="p">,</span> <span class="s">"PYTHONDEBUG"</span><span class="p">);</span>
3498 </span> <span class="n">_Py_get_env_flag</span><span class="p">(</span><span class="n">use_env</span><span class="p">,</span> <span class="o">&</span><span class="n">config</span><span class="o">-></span><span class="n">verbose</span><span class="p">,</span> <span class="s">"PYTHONVERBOSE"</span><span class="p">);</span>
3499 <span class="n">_Py_get_env_flag</span><span class="p">(</span><span class="n">use_env</span><span class="p">,</span> <span class="o">&</span><span class="n">config</span><span class="o">-></span><span class="n">optimization_level</span><span class="p">,</span> <span class="s">"PYTHONOPTIMIZE"</span><span class="p">);</span>
3500 <span class="n">_Py_get_env_flag</span><span class="p">(</span><span class="n">use_env</span><span class="p">,</span> <span class="o">&</span><span class="n">config</span><span class="o">-></span><span class="n">inspect</span><span class="p">,</span> <span class="s">"PYTHONINSPECT"</span><span class="p">);</span>
3501 </pre></div>
3502
3503 <p>For the verbose setting, you can see that the value of <code>PYTHONVERBOSE</code> is used to set the value of <code>&config->verbose</code>, if <code>PYTHONVERBOSE</code> is found. If the environment variable does not exist, then the default value of <code>-1</code> will remain.</p>
3504 <p>Then in <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/initconfig.c#L1828"><code>config_parse_cmdline</code></a> within <code>initconfig.c</code> again, the command-line flag is used to set the value, if provided:</p>
3505 <div class="highlight c"><pre><span></span><span class="k">static</span> <span class="n">PyStatus</span>
3506 <span class="nf">config_parse_cmdline</span><span class="p">(</span><span class="n">PyConfig</span> <span class="o">*</span><span class="n">config</span><span class="p">,</span> <span class="n">PyWideStringList</span> <span class="o">*</span><span class="n">warnoptions</span><span class="p">,</span>
3507 <span class="n">Py_ssize_t</span> <span class="o">*</span><span class="n">opt_index</span><span class="p">)</span>
3508 <span class="p">{</span>
3509 <span class="p">...</span>
3510
3511 <span class="k">switch</span> <span class="p">(</span><span class="n">c</span><span class="p">)</span> <span class="p">{</span>
3512 <span class="p">...</span>
3513
3514 <span class="k">case</span> <span class="sc">'v'</span><span class="o">:</span>
3515 <span class="hll"> <span class="n">config</span><span class="o">-></span><span class="n">verbose</span><span class="o">++</span><span class="p">;</span>
3516 </span> <span class="k">break</span><span class="p">;</span>
3517 <span class="p">...</span>
3518 <span class="cm">/* This space reserved for other options */</span>
3519
3520 <span class="k">default</span><span class="o">:</span>
3521 <span class="cm">/* unknown argument: parsing failed */</span>
3522 <span class="n">config_usage</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">program</span><span class="p">);</span>
3523 <span class="k">return</span> <span class="n">_PyStatus_EXIT</span><span class="p">(</span><span class="mi">2</span><span class="p">);</span>
3524 <span class="p">}</span>
3525 <span class="p">}</span> <span class="k">while</span> <span class="p">(</span><span class="mi">1</span><span class="p">);</span>
3526 </pre></div>
3527
3528 <p>This value is later copied to a global variable <code>Py_VerboseFlag</code> by the <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/initconfig.c#L134"><code>_Py_GetGlobalVariablesAsDict</code></a> function.</p>
3529 <p>Within a Python session, you can access the runtime flags, like verbose mode, quiet mode, using the <code>sys.flags</code> named tuple.
3530 The <code>-X</code> flags are all available inside the <code>sys._xoptions</code> dictionary:</p>
3531 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="go">$ ./python.exe -X dev -q </span>
3532
3533 <span class="gp">>>> </span><span class="kn">import</span> <span class="nn">sys</span>
3534 <span class="gp">>>> </span><span class="n">sys</span><span class="o">.</span><span class="n">flags</span>
3535 <span class="go">sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, </span>
3536 <span class="go"> no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, </span>
3537 <span class="go"> quiet=1, hash_randomization=1, isolated=0, dev_mode=True, utf8_mode=0)</span>
3538
3539 <span class="gp">>>> </span><span class="n">sys</span><span class="o">.</span><span class="n">_xoptions</span>
3540 <span class="go">{'dev': True}</span>
3541 </pre></div>
3542
3543 <p>As well as the runtime configuration in <code>initconfig.h</code>, there is also the build configuration, which is located inside <code>pyconfig.h</code> in the root folder. This file is created dynamically in the <code>configure</code> step in the build process, or by Visual Studio for Windows systems.</p>
3544 <p>You can see the build configuration by running:</p>
3545 <div class="highlight sh"><pre><span></span><span class="gp">$</span> ./python.exe -m sysconfig
3546 </pre></div>
3547
3548 <h3 id="reading-filesinput">Reading Files/Input</h3>
3549 <p>Once CPython has the runtime configuration and the command-line arguments, it can establish what it needs to execute.</p>
3550 <p>This task is handled by the <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Modules/main.c#L665"><code>pymain_main</code></a> function inside <code>Modules/main.c</code>. Depending on the newly created <code>config</code> instance, CPython will now execute code provided via several options.</p>
3551 <h4 id="input-via-c">Input via <code>-c</code></h4>
3552 <p>The simplest is providing CPython a command with the <code>-c</code> option and a Python program inside quotes.</p>
3553 <p>For example:</p>
3554 <div class="highlight sh"><pre><span></span><span class="gp">$</span> ./python.exe -c <span class="s2">"print('hi')"</span>
3555 <span class="go">hi</span>
3556 </pre></div>
3557
3558 <p>Here is the full flowchart of how this happens:</p>
3559 <p><a href="https://files.realpython.com/media/pymain_run_command.f5da561ba7d5.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/pymain_run_command.f5da561ba7d5.png" width="1041" height="751" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pymain_run_command.f5da561ba7d5.png&w=260&sig=f7f802b23e900bf42b29804ee80ed0dd0eaec6a4 260w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pymain_run_command.f5da561ba7d5.png&w=520&sig=3079060f305945fd3556ce7cd453fac965a6ec27 520w, https://files.realpython.com/media/pymain_run_command.f5da561ba7d5.png 1041w" sizes="75vw" alt="Flow chart of pymain_run_command"/></a></p>
3560 <p>First, the <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Modules/main.c#L240"><code>pymain_run_command()</code></a> function is executed inside <code>Modules/main.c</code> taking the command passed in <code>-c</code> as an argument in the C type <code>wchar_t*</code>. The <code>wchar_t*</code> type is often used as a low-level storage type for Unicode data across CPython as the size of the type can store UTF8 characters.</p>
3561 <p>When converting the <code>wchar_t*</code> to a Python string, the <code>Objects/unicodeobject.c</code> file has a helper function <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Objects/unicodeobject.c#L2088"><code>PyUnicode_FromWideChar()</code></a> that returns a <code>PyObject</code>, of type <code>str</code>. The encoding to UTF8 is then done by <code>PyUnicode_AsUTF8String()</code> on the Python <code>str</code> object to convert it to a Python <code>bytes</code> object. </p>
3562 <p>Once this is complete, <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Modules/main.c#L240"><code>pymain_run_command()</code></a> will then pass the Python bytes object to <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pythonrun.c#L453"><code>PyRun_SimpleStringFlags()</code></a> for execution, but first converting the <code>bytes</code> to a <code>str</code> type again:</p>
3563 <div class="highlight c"><pre><span></span><span class="k">static</span> <span class="kt">int</span>
3564 <span class="nf">pymain_run_command</span><span class="p">(</span><span class="kt">wchar_t</span> <span class="o">*</span><span class="n">command</span><span class="p">,</span> <span class="n">PyCompilerFlags</span> <span class="o">*</span><span class="n">cf</span><span class="p">)</span>
3565 <span class="p">{</span>
3566 <span class="n">PyObject</span> <span class="o">*</span><span class="n">unicode</span><span class="p">,</span> <span class="o">*</span><span class="n">bytes</span><span class="p">;</span>
3567 <span class="kt">int</span> <span class="n">ret</span><span class="p">;</span>
3568
3569 <span class="n">unicode</span> <span class="o">=</span> <span class="n">PyUnicode_FromWideChar</span><span class="p">(</span><span class="n">command</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">);</span>
3570 <span class="k">if</span> <span class="p">(</span><span class="n">unicode</span> <span class="o">==</span> <span class="nb">NULL</span><span class="p">)</span> <span class="p">{</span>
3571 <span class="k">goto</span> <span class="n">error</span><span class="p">;</span>
3572 <span class="p">}</span>
3573
3574 <span class="k">if</span> <span class="p">(</span><span class="n">PySys_Audit</span><span class="p">(</span><span class="s">"cpython.run_command"</span><span class="p">,</span> <span class="s">"O"</span><span class="p">,</span> <span class="n">unicode</span><span class="p">)</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
3575 <span class="k">return</span> <span class="n">pymain_exit_err_print</span><span class="p">();</span>
3576 <span class="p">}</span>
3577
3578 <span class="n">bytes</span> <span class="o">=</span> <span class="n">PyUnicode_AsUTF8String</span><span class="p">(</span><span class="n">unicode</span><span class="p">);</span>
3579 <span class="n">Py_DECREF</span><span class="p">(</span><span class="n">unicode</span><span class="p">);</span>
3580 <span class="k">if</span> <span class="p">(</span><span class="n">bytes</span> <span class="o">==</span> <span class="nb">NULL</span><span class="p">)</span> <span class="p">{</span>
3581 <span class="k">goto</span> <span class="n">error</span><span class="p">;</span>
3582 <span class="p">}</span>
3583
3584 <span class="n">ret</span> <span class="o">=</span> <span class="n">PyRun_SimpleStringFlags</span><span class="p">(</span><span class="n">PyBytes_AsString</span><span class="p">(</span><span class="n">bytes</span><span class="p">),</span> <span class="n">cf</span><span class="p">);</span>
3585 <span class="n">Py_DECREF</span><span class="p">(</span><span class="n">bytes</span><span class="p">);</span>
3586 <span class="k">return</span> <span class="p">(</span><span class="n">ret</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">);</span>
3587
3588 <span class="nl">error</span><span class="p">:</span>
3589 <span class="n">PySys_WriteStderr</span><span class="p">(</span><span class="s">"Unable to decode the command from the command line:</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span>
3590 <span class="k">return</span> <span class="n">pymain_exit_err_print</span><span class="p">();</span>
3591 <span class="p">}</span>
3592 </pre></div>
3593
3594 <p>The conversion of <code>wchar_t*</code> to Unicode, bytes, and then a string is roughly equivalent to the following:</p>
3595 <div class="highlight python"><pre><span></span><span class="n">unicode</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="n">command</span><span class="p">)</span>
3596 <span class="n">bytes_</span> <span class="o">=</span> <span class="nb">bytes</span><span class="p">(</span><span class="n">unicode</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'utf8'</span><span class="p">))</span>
3597 <span class="c1"># call PyRun_SimpleStringFlags with bytes_</span>
3598 </pre></div>
3599
3600 <p>The <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pythonrun.c#L453"><code>PyRun_SimpleStringFlags()</code></a> function is part of <code>Python/pythonrun.c</code>. It’s purpose is to turn this simple command into a Python module and then send it on to be executed.
3601 Since a Python module needs to have <code>__main__</code> to be executed as a standalone module, it creates that automatically:</p>
3602 <div class="highlight c"><pre><span></span><span class="kt">int</span>
3603 <span class="nf">PyRun_SimpleStringFlags</span><span class="p">(</span><span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">command</span><span class="p">,</span> <span class="n">PyCompilerFlags</span> <span class="o">*</span><span class="n">flags</span><span class="p">)</span>
3604 <span class="p">{</span>
3605 <span class="n">PyObject</span> <span class="o">*</span><span class="n">m</span><span class="p">,</span> <span class="o">*</span><span class="n">d</span><span class="p">,</span> <span class="o">*</span><span class="n">v</span><span class="p">;</span>
3606 <span class="hll"> <span class="n">m</span> <span class="o">=</span> <span class="n">PyImport_AddModule</span><span class="p">(</span><span class="s">"__main__"</span><span class="p">);</span>
3607 </span> <span class="k">if</span> <span class="p">(</span><span class="n">m</span> <span class="o">==</span> <span class="nb">NULL</span><span class="p">)</span>
3608 <span class="k">return</span> <span class="o">-</span><span class="mi">1</span><span class="p">;</span>
3609 <span class="hll"> <span class="n">d</span> <span class="o">=</span> <span class="n">PyModule_GetDict</span><span class="p">(</span><span class="n">m</span><span class="p">);</span>
3610 </span><span class="hll"> <span class="n">v</span> <span class="o">=</span> <span class="n">PyRun_StringFlags</span><span class="p">(</span><span class="n">command</span><span class="p">,</span> <span class="n">Py_file_input</span><span class="p">,</span> <span class="n">d</span><span class="p">,</span> <span class="n">d</span><span class="p">,</span> <span class="n">flags</span><span class="p">);</span>
3611 </span> <span class="k">if</span> <span class="p">(</span><span class="n">v</span> <span class="o">==</span> <span class="nb">NULL</span><span class="p">)</span> <span class="p">{</span>
3612 <span class="n">PyErr_Print</span><span class="p">();</span>
3613 <span class="k">return</span> <span class="o">-</span><span class="mi">1</span><span class="p">;</span>
3614 <span class="p">}</span>
3615 <span class="n">Py_DECREF</span><span class="p">(</span><span class="n">v</span><span class="p">);</span>
3616 <span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
3617 <span class="p">}</span>
3618 </pre></div>
3619
3620 <p>Once <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pythonrun.c#L453"><code>PyRun_SimpleStringFlags()</code></a> has created a module and a dictionary, it calls <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pythonrun.c#L1008"><code>PyRun_StringFlags()</code></a>, which creates a fake filename and then calls the Python parser to create an AST from the string and return a module, <code>mod</code>:</p>
3621 <div class="highlight c"><pre><span></span><span class="n">PyObject</span> <span class="o">*</span>
3622 <span class="nf">PyRun_StringFlags</span><span class="p">(</span><span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">str</span><span class="p">,</span> <span class="kt">int</span> <span class="n">start</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">globals</span><span class="p">,</span>
3623 <span class="n">PyObject</span> <span class="o">*</span><span class="n">locals</span><span class="p">,</span> <span class="n">PyCompilerFlags</span> <span class="o">*</span><span class="n">flags</span><span class="p">)</span>
3624 <span class="p">{</span>
3625 <span class="p">...</span>
3626 <span class="n">mod</span> <span class="o">=</span> <span class="n">PyParser_ASTFromStringObject</span><span class="p">(</span><span class="n">str</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="n">start</span><span class="p">,</span> <span class="n">flags</span><span class="p">,</span> <span class="n">arena</span><span class="p">);</span>
3627 <span class="k">if</span> <span class="p">(</span><span class="n">mod</span> <span class="o">!=</span> <span class="nb">NULL</span><span class="p">)</span>
3628 <span class="n">ret</span> <span class="o">=</span> <span class="n">run_mod</span><span class="p">(</span><span class="n">mod</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="n">globals</span><span class="p">,</span> <span class="n">locals</span><span class="p">,</span> <span class="n">flags</span><span class="p">,</span> <span class="n">arena</span><span class="p">);</span>
3629 <span class="n">PyArena_Free</span><span class="p">(</span><span class="n">arena</span><span class="p">);</span>
3630 <span class="k">return</span> <span class="n">ret</span><span class="p">;</span>
3631 </pre></div>
3632
3633 <p>You’ll dive into the AST and Parser code in the next section.</p>
3634 <h4 id="input-via-m">Input via <code>-m</code></h4>
3635 <p>Another way to execute Python commands is by using the <code>-m</code> option with the name of a module.
3636 A typical example is <code>python -m unittest</code> to run the unittest module in the standard library.</p>
3637 <p>Being able to execute modules as scripts were initially proposed in <a href="https://www.python.org/dev/peps/pep-0338">PEP 338</a> and then the standard for explicit relative imports defined in <a href="https://www.python.org/dev/peps/pep-0366">PEP366</a>. </p>
3638 <p>The use of the <code>-m</code> flag implies that within the module package, you want to execute whatever is inside <a href="https://realpython.com/python-main-function/"><code>__main__</code></a>. It also implies that you want to search <code>sys.path</code> for the named module.</p>
3639 <p>This search mechanism is why you don’t need to remember where the <code>unittest</code> module is stored on your filesystem.</p>
3640 <p>Inside <code>Modules/main.c</code> there is a function called when the command-line is run with the <code>-m</code> flag. The name of the module is passed as the <code>modname</code> argument.</p>
3641 <p>CPython will then import a standard library module, <code>runpy</code> and execute it using <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Objects/call.c#L214"><code>PyObject_Call()</code></a>. The import is done using the C API function <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/import.c#L1409"><code>PyImport_ImportModule()</code></a>, found within the <code>Python/import.c</code> file:</p>
3642 <div class="highlight c"><pre><span></span><span class="k">static</span> <span class="kt">int</span>
3643 <span class="nf">pymain_run_module</span><span class="p">(</span><span class="k">const</span> <span class="kt">wchar_t</span> <span class="o">*</span><span class="n">modname</span><span class="p">,</span> <span class="kt">int</span> <span class="n">set_argv0</span><span class="p">)</span>
3644 <span class="p">{</span>
3645 <span class="n">PyObject</span> <span class="o">*</span><span class="n">module</span><span class="p">,</span> <span class="o">*</span><span class="n">runpy</span><span class="p">,</span> <span class="o">*</span><span class="n">runmodule</span><span class="p">,</span> <span class="o">*</span><span class="n">runargs</span><span class="p">,</span> <span class="o">*</span><span class="n">result</span><span class="p">;</span>
3646 <span class="n">runpy</span> <span class="o">=</span> <span class="n">PyImport_ImportModule</span><span class="p">(</span><span class="s">"runpy"</span><span class="p">);</span>
3647 <span class="p">...</span>
3648 <span class="n">runmodule</span> <span class="o">=</span> <span class="n">PyObject_GetAttrString</span><span class="p">(</span><span class="n">runpy</span><span class="p">,</span> <span class="s">"_run_module_as_main"</span><span class="p">);</span>
3649 <span class="p">...</span>
3650 <span class="n">module</span> <span class="o">=</span> <span class="n">PyUnicode_FromWideChar</span><span class="p">(</span><span class="n">modname</span><span class="p">,</span> <span class="n">wcslen</span><span class="p">(</span><span class="n">modname</span><span class="p">));</span>
3651 <span class="p">...</span>
3652 <span class="n">runargs</span> <span class="o">=</span> <span class="n">Py_BuildValue</span><span class="p">(</span><span class="s">"(Oi)"</span><span class="p">,</span> <span class="n">module</span><span class="p">,</span> <span class="n">set_argv0</span><span class="p">);</span>
3653 <span class="p">...</span>
3654 <span class="n">result</span> <span class="o">=</span> <span class="n">PyObject_Call</span><span class="p">(</span><span class="n">runmodule</span><span class="p">,</span> <span class="n">runargs</span><span class="p">,</span> <span class="nb">NULL</span><span class="p">);</span>
3655 <span class="p">...</span>
3656 <span class="k">if</span> <span class="p">(</span><span class="n">result</span> <span class="o">==</span> <span class="nb">NULL</span><span class="p">)</span> <span class="p">{</span>
3657 <span class="k">return</span> <span class="n">pymain_exit_err_print</span><span class="p">();</span>
3658 <span class="p">}</span>
3659 <span class="n">Py_DECREF</span><span class="p">(</span><span class="n">result</span><span class="p">);</span>
3660 <span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
3661 <span class="p">}</span>
3662 </pre></div>
3663
3664 <p>In this function you’ll also see 2 other C API functions: <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Objects/call.c#L214"><code>PyObject_Call()</code></a> and <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Objects/object.c#L831"><code>PyObject_GetAttrString()</code></a>. Because <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/import.c#L1409"><code>PyImport_ImportModule()</code></a> returns a <code>PyObject*</code>, the core object type, you need to call special functions to get attributes and to call it.</p>
3665 <p>In Python, if you had an object and wanted to get an attribute, then you could call <code>getattr()</code>. In the C API, this call is <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Objects/object.c#L831"><code>PyObject_GetAttrString()</code></a>, which is found in <code>Objects/object.c</code>. If you wanted to run a callable, you would give it parentheses, or you can run the <code>__call__()</code> property on any Python object. The <code>__call__()</code> method is implemented inside <code>Objects/object.c</code>:</p>
3666 <div class="highlight python"><pre><span></span><span class="n">hi</span> <span class="o">=</span> <span class="s2">"hi!"</span>
3667 <span class="n">hi</span><span class="o">.</span><span class="n">upper</span><span class="p">()</span> <span class="o">==</span> <span class="n">hi</span><span class="o">.</span><span class="n">upper</span><span class="o">.</span><span class="fm">__call__</span><span class="p">()</span> <span class="c1"># this is the same</span>
3668 </pre></div>
3669
3670 <p>The <code>runpy</code> module is written in pure Python and located in <code>Lib/runpy.py</code>.</p>
3671 <p>Executing <code>python -m <module></code> is equivalent to running <code>python -m runpy <module></code>. The <code>runpy</code> module was created to abstract the process of locating and executing modules on an operating system.</p>
3672 <p><code>runpy</code> does a few things to run the target module:</p>
3673 <ul>
3674 <li>Calls <code>__import__()</code> for the module name you provided</li>
3675 <li>Sets <code>__name__</code> (the module name) to a namespace called <code>__main__</code></li>
3676 <li>Executes the module within the <code>__main__</code> namespace</li>
3677 </ul>
3678 <p>The <code>runpy</code> module also supports executing directories and zip files.</p>
3679 <h4 id="input-via-filename">Input via Filename</h4>
3680 <p>If the first argument to <code>python</code> was a filename, such as <code>python test.py</code>, then CPython will open a file handle, similar to using <code>open()</code> in Python and pass the handle to <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pythonrun.c#L372"><code>PyRun_SimpleFileExFlags()</code></a> inside <code>Python/pythonrun.c</code>.</p>
3681 <p>There are 3 paths this function can take:</p>
3682 <ol>
3683 <li>If the file path is a <code>.pyc</code> file, it will call <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pythonrun.c#L1145"><code>run_pyc_file()</code></a>.</li>
3684 <li>If the file path is a script file (<code>.py</code>) it will run <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pythonrun.c#L1032"><code>PyRun_FileExFlags()</code></a>.</li>
3685 <li>If the filepath is <code>stdin</code> because the user ran <code>command | python</code> then treat <code>stdin</code> as a file handle and run <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pythonrun.c#L1032"><code>PyRun_FileExFlags()</code></a>.</li>
3686 </ol>
3687 <div class="highlight c"><pre><span></span><span class="kt">int</span>
3688 <span class="nf">PyRun_SimpleFileExFlags</span><span class="p">(</span><span class="kt">FILE</span> <span class="o">*</span><span class="n">fp</span><span class="p">,</span> <span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">filename</span><span class="p">,</span> <span class="kt">int</span> <span class="n">closeit</span><span class="p">,</span>
3689 <span class="n">PyCompilerFlags</span> <span class="o">*</span><span class="n">flags</span><span class="p">)</span>
3690 <span class="p">{</span>
3691 <span class="p">...</span>
3692 <span class="n">m</span> <span class="o">=</span> <span class="n">PyImport_AddModule</span><span class="p">(</span><span class="s">"__main__"</span><span class="p">);</span>
3693 <span class="p">...</span>
3694 <span class="hll"> <span class="k">if</span> <span class="p">(</span><span class="n">maybe_pyc_file</span><span class="p">(</span><span class="n">fp</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="n">ext</span><span class="p">,</span> <span class="n">closeit</span><span class="p">))</span> <span class="p">{</span>
3695 </span> <span class="p">...</span>
3696 <span class="hll"> <span class="n">v</span> <span class="o">=</span> <span class="n">run_pyc_file</span><span class="p">(</span><span class="n">pyc_fp</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="n">d</span><span class="p">,</span> <span class="n">d</span><span class="p">,</span> <span class="n">flags</span><span class="p">);</span>
3697 </span> <span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
3698 <span class="cm">/* When running from stdin, leave __main__.__loader__ alone */</span>
3699 <span class="k">if</span> <span class="p">(</span><span class="n">strcmp</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s">"<stdin>"</span><span class="p">)</span> <span class="o">!=</span> <span class="mi">0</span> <span class="o">&&</span>
3700 <span class="n">set_main_loader</span><span class="p">(</span><span class="n">d</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="s">"SourceFileLoader"</span><span class="p">)</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
3701 <span class="n">fprintf</span><span class="p">(</span><span class="n">stderr</span><span class="p">,</span> <span class="s">"python: failed to set __main__.__loader__</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span>
3702 <span class="n">ret</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span><span class="p">;</span>
3703 <span class="k">goto</span> <span class="n">done</span><span class="p">;</span>
3704 <span class="p">}</span>
3705 <span class="hll"> <span class="n">v</span> <span class="o">=</span> <span class="n">PyRun_FileExFlags</span><span class="p">(</span><span class="n">fp</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="n">Py_file_input</span><span class="p">,</span> <span class="n">d</span><span class="p">,</span> <span class="n">d</span><span class="p">,</span>
3706 </span><span class="hll"> <span class="n">closeit</span><span class="p">,</span> <span class="n">flags</span><span class="p">);</span>
3707 </span> <span class="p">}</span>
3708 <span class="p">...</span>
3709 <span class="k">return</span> <span class="n">ret</span><span class="p">;</span>
3710 <span class="p">}</span>
3711 </pre></div>
3712
3713 <h4 id="input-via-file-with-pyrun_fileexflags">Input via File With <code>PyRun_FileExFlags()</code></h4>
3714 <p>For <code>stdin</code> and basic script files, CPython will pass the file handle to <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pythonrun.c#L1032"><code>PyRun_FileExFlags()</code></a> located in the <code>pythonrun.c</code> file.</p>
3715 <p>The purpose of <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pythonrun.c#L1032"><code>PyRun_FileExFlags()</code></a> is similar to <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pythonrun.c#L453"><code>PyRun_SimpleStringFlags()</code></a> used for the <code>-c</code> input. CPython will load the file handle into <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pythonrun.c#L1369"><code>PyParser_ASTFromFileObject()</code></a>. We’ll cover the Parser and AST modules in the next section.
3716 Because this is a full script, it doesn’t need the <code>PyImport_AddModule("__main__");</code> step used by <code>-c</code>:</p>
3717 <div class="highlight c"><pre><span></span><span class="n">PyObject</span> <span class="o">*</span>
3718 <span class="nf">PyRun_FileExFlags</span><span class="p">(</span><span class="kt">FILE</span> <span class="o">*</span><span class="n">fp</span><span class="p">,</span> <span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">filename_str</span><span class="p">,</span> <span class="kt">int</span> <span class="n">start</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">globals</span><span class="p">,</span>
3719 <span class="n">PyObject</span> <span class="o">*</span><span class="n">locals</span><span class="p">,</span> <span class="kt">int</span> <span class="n">closeit</span><span class="p">,</span> <span class="n">PyCompilerFlags</span> <span class="o">*</span><span class="n">flags</span><span class="p">)</span>
3720 <span class="p">{</span>
3721 <span class="p">...</span>
3722 <span class="n">mod</span> <span class="o">=</span> <span class="n">PyParser_ASTFromFileObject</span><span class="p">(</span><span class="n">fp</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="nb">NULL</span><span class="p">,</span> <span class="n">start</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span>
3723 <span class="n">flags</span><span class="p">,</span> <span class="nb">NULL</span><span class="p">,</span> <span class="n">arena</span><span class="p">);</span>
3724 <span class="p">...</span>
3725 <span class="n">ret</span> <span class="o">=</span> <span class="n">run_mod</span><span class="p">(</span><span class="n">mod</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="n">globals</span><span class="p">,</span> <span class="n">locals</span><span class="p">,</span> <span class="n">flags</span><span class="p">,</span> <span class="n">arena</span><span class="p">);</span>
3726 <span class="p">}</span>
3727 </pre></div>
3728
3729 <p>Identical to <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pythonrun.c#L453"><code>PyRun_SimpleStringFlags()</code></a>, once <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pythonrun.c#L1032"><code>PyRun_FileExFlags()</code></a> has created a Python module from the file, it sent it to <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pythonrun.c#L1125"><code>run_mod()</code></a> to be executed.</p>
3730 <p><a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pythonrun.c#L1125"><code>run_mod()</code></a> is found within <code>Python/pythonrun.c</code>, and sends the module to the AST to be compiled into a code object. Code objects are a format used to store the bytecode operations and the format kept in <code>.pyc</code> files:</p>
3731 <div class="highlight c"><pre><span></span><span class="k">static</span> <span class="n">PyObject</span> <span class="o">*</span>
3732 <span class="nf">run_mod</span><span class="p">(</span><span class="n">mod_ty</span> <span class="n">mod</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">filename</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">globals</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">locals</span><span class="p">,</span>
3733 <span class="n">PyCompilerFlags</span> <span class="o">*</span><span class="n">flags</span><span class="p">,</span> <span class="n">PyArena</span> <span class="o">*</span><span class="n">arena</span><span class="p">)</span>
3734 <span class="p">{</span>
3735 <span class="n">PyCodeObject</span> <span class="o">*</span><span class="n">co</span><span class="p">;</span>
3736 <span class="n">PyObject</span> <span class="o">*</span><span class="n">v</span><span class="p">;</span>
3737 <span class="n">co</span> <span class="o">=</span> <span class="n">PyAST_CompileObject</span><span class="p">(</span><span class="n">mod</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="n">flags</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="n">arena</span><span class="p">);</span>
3738 <span class="k">if</span> <span class="p">(</span><span class="n">co</span> <span class="o">==</span> <span class="nb">NULL</span><span class="p">)</span>
3739 <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
3740
3741 <span class="k">if</span> <span class="p">(</span><span class="n">PySys_Audit</span><span class="p">(</span><span class="s">"exec"</span><span class="p">,</span> <span class="s">"O"</span><span class="p">,</span> <span class="n">co</span><span class="p">)</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
3742 <span class="n">Py_DECREF</span><span class="p">(</span><span class="n">co</span><span class="p">);</span>
3743 <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
3744 <span class="p">}</span>
3745
3746 <span class="n">v</span> <span class="o">=</span> <span class="n">run_eval_code_obj</span><span class="p">(</span><span class="n">co</span><span class="p">,</span> <span class="n">globals</span><span class="p">,</span> <span class="n">locals</span><span class="p">);</span>
3747 <span class="n">Py_DECREF</span><span class="p">(</span><span class="n">co</span><span class="p">);</span>
3748 <span class="k">return</span> <span class="n">v</span><span class="p">;</span>
3749 <span class="p">}</span>
3750 </pre></div>
3751
3752 <p>We will cover the CPython compiler and bytecodes in the next section. The call to <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pythonrun.c#L1094"><code>run_eval_code_obj()</code></a> is a simple wrapper function that calls <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/ceval.c#L716"><code>PyEval_EvalCode()</code></a> in the <code>Python/eval.c</code> file. The <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/ceval.c#L716"><code>PyEval_EvalCode()</code></a> function is the main evaluation loop for CPython, it iterates over each bytecode statement and executes it on your local machine.</p>
3753 <h4 id="input-via-compiled-bytecode-with-run_pyc_file">Input via Compiled Bytecode With <code>run_pyc_file()</code></h4>
3754 <p>In the <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pythonrun.c#L372"><code>PyRun_SimpleFileExFlags()</code></a> there was a clause for the user providing a file path to a <code>.pyc</code> file. If the file path ended in <code>.pyc</code> then instead of loading the file as a plain text file and parsing it, it will assume that the <code>.pyc</code> file contains a code object written to disk. </p>
3755 <p>The <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pythonrun.c#L1145"><code>run_pyc_file()</code></a> function inside <code>Python/pythonrun.c</code> then marshals the code object from the <code>.pyc</code> file by using the file handle. <strong>Marshaling</strong> is a technical term for copying the contents of a file into memory and converting them to a specific data structure. The code object data structure on the disk is the CPython compiler’s way to caching compiled code so that it doesn’t need to parse it every time the script is called:</p>
3756 <div class="highlight c"><pre><span></span><span class="k">static</span> <span class="n">PyObject</span> <span class="o">*</span>
3757 <span class="nf">run_pyc_file</span><span class="p">(</span><span class="kt">FILE</span> <span class="o">*</span><span class="n">fp</span><span class="p">,</span> <span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">filename</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">globals</span><span class="p">,</span>
3758 <span class="n">PyObject</span> <span class="o">*</span><span class="n">locals</span><span class="p">,</span> <span class="n">PyCompilerFlags</span> <span class="o">*</span><span class="n">flags</span><span class="p">)</span>
3759 <span class="p">{</span>
3760 <span class="n">PyCodeObject</span> <span class="o">*</span><span class="n">co</span><span class="p">;</span>
3761 <span class="n">PyObject</span> <span class="o">*</span><span class="n">v</span><span class="p">;</span>
3762 <span class="p">...</span>
3763 <span class="hll"> <span class="n">v</span> <span class="o">=</span> <span class="n">PyMarshal_ReadLastObjectFromFile</span><span class="p">(</span><span class="n">fp</span><span class="p">);</span>
3764 </span> <span class="p">...</span>
3765 <span class="k">if</span> <span class="p">(</span><span class="n">v</span> <span class="o">==</span> <span class="nb">NULL</span> <span class="o">||</span> <span class="o">!</span><span class="n">PyCode_Check</span><span class="p">(</span><span class="n">v</span><span class="p">))</span> <span class="p">{</span>
3766 <span class="n">Py_XDECREF</span><span class="p">(</span><span class="n">v</span><span class="p">);</span>
3767 <span class="n">PyErr_SetString</span><span class="p">(</span><span class="n">PyExc_RuntimeError</span><span class="p">,</span>
3768 <span class="s">"Bad code object in .pyc file"</span><span class="p">);</span>
3769 <span class="k">goto</span> <span class="n">error</span><span class="p">;</span>
3770 <span class="p">}</span>
3771 <span class="n">fclose</span><span class="p">(</span><span class="n">fp</span><span class="p">);</span>
3772 <span class="hll"> <span class="n">co</span> <span class="o">=</span> <span class="p">(</span><span class="n">PyCodeObject</span> <span class="o">*</span><span class="p">)</span><span class="n">v</span><span class="p">;</span>
3773 </span><span class="hll"> <span class="n">v</span> <span class="o">=</span> <span class="n">run_eval_code_obj</span><span class="p">(</span><span class="n">co</span><span class="p">,</span> <span class="n">globals</span><span class="p">,</span> <span class="n">locals</span><span class="p">);</span>
3774 </span> <span class="k">if</span> <span class="p">(</span><span class="n">v</span> <span class="o">&&</span> <span class="n">flags</span><span class="p">)</span>
3775 <span class="n">flags</span><span class="o">-></span><span class="n">cf_flags</span> <span class="o">|=</span> <span class="p">(</span><span class="n">co</span><span class="o">-></span><span class="n">co_flags</span> <span class="o">&</span> <span class="n">PyCF_MASK</span><span class="p">);</span>
3776 <span class="n">Py_DECREF</span><span class="p">(</span><span class="n">co</span><span class="p">);</span>
3777 <span class="k">return</span> <span class="n">v</span><span class="p">;</span>
3778 <span class="p">}</span>
3779 </pre></div>
3780
3781 <p>Once the code object has been marshaled to memory, it is sent to <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pythonrun.c#L1094"><code>run_eval_code_obj()</code></a>, which calls <code>Python/ceval.c</code> to execute the code.</p>
3782 <h3 id="lexing-and-parsing">Lexing and Parsing</h3>
3783 <p>In the exploration of reading and executing Python files, we dived as deep as the parser and AST modules, with function calls to <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pythonrun.c#L1369"><code>PyParser_ASTFromFileObject()</code></a>.</p>
3784 <p>Sticking within <code>Python/pythonrun.c</code>, the <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pythonrun.c#L1369"><code>PyParser_ASTFromFileObject()</code></a> function will take a file handle, compiler flags and a <code>PyArena</code> instance and convert the file object into a node object using <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Parser/parsetok.c#L163"><code>PyParser_ParseFileObject()</code></a>.</p>
3785 <p>With the node object, it will then convert that into a module using the AST function <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/ast.c#L772"><code>PyAST_FromNodeObject()</code></a>:</p>
3786 <div class="highlight c"><pre><span></span><span class="n">mod_ty</span>
3787 <span class="nf">PyParser_ASTFromFileObject</span><span class="p">(</span><span class="kt">FILE</span> <span class="o">*</span><span class="n">fp</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">filename</span><span class="p">,</span> <span class="k">const</span> <span class="kt">char</span><span class="o">*</span> <span class="n">enc</span><span class="p">,</span>
3788 <span class="kt">int</span> <span class="n">start</span><span class="p">,</span> <span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">ps1</span><span class="p">,</span>
3789 <span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">ps2</span><span class="p">,</span> <span class="n">PyCompilerFlags</span> <span class="o">*</span><span class="n">flags</span><span class="p">,</span> <span class="kt">int</span> <span class="o">*</span><span class="n">errcode</span><span class="p">,</span>
3790 <span class="n">PyArena</span> <span class="o">*</span><span class="n">arena</span><span class="p">)</span>
3791 <span class="p">{</span>
3792 <span class="p">...</span>
3793 <span class="hll"> <span class="n">node</span> <span class="o">*</span><span class="n">n</span> <span class="o">=</span> <span class="n">PyParser_ParseFileObject</span><span class="p">(</span><span class="n">fp</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="n">enc</span><span class="p">,</span>
3794 </span><span class="hll"> <span class="o">&</span><span class="n">_PyParser_Grammar</span><span class="p">,</span>
3795 </span><span class="hll"> <span class="n">start</span><span class="p">,</span> <span class="n">ps1</span><span class="p">,</span> <span class="n">ps2</span><span class="p">,</span> <span class="o">&</span><span class="n">err</span><span class="p">,</span> <span class="o">&</span><span class="n">iflags</span><span class="p">);</span>
3796 </span> <span class="p">...</span>
3797 <span class="k">if</span> <span class="p">(</span><span class="n">n</span><span class="p">)</span> <span class="p">{</span>
3798 <span class="hll"> <span class="n">flags</span><span class="o">-></span><span class="n">cf_flags</span> <span class="o">|=</span> <span class="n">iflags</span> <span class="o">&</span> <span class="n">PyCF_MASK</span><span class="p">;</span>
3799 </span> <span class="n">mod</span> <span class="o">=</span> <span class="n">PyAST_FromNodeObject</span><span class="p">(</span><span class="n">n</span><span class="p">,</span> <span class="n">flags</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="n">arena</span><span class="p">);</span>
3800 <span class="n">PyNode_Free</span><span class="p">(</span><span class="n">n</span><span class="p">);</span>
3801 <span class="p">...</span>
3802 <span class="k">return</span> <span class="n">mod</span><span class="p">;</span>
3803 <span class="p">}</span>
3804 </pre></div>
3805
3806 <p>For <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Parser/parsetok.c#L163"><code>PyParser_ParseFileObject()</code></a> we switch to <code>Parser/parsetok.c</code> and the parser-tokenizer stage of the CPython interpreter. This function has two important tasks:</p>
3807 <ol>
3808 <li>Instantiate a tokenizer state <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Parser/tokenizer.h#L23"><code>tok_state</code></a> using <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Parser/tokenizer.h#L78"><code>PyTokenizer_FromFile()</code></a> in <code>Parser/tokenizer.c</code></li>
3809 <li>Convert the tokens into a concrete parse tree (a list of <code>node</code>) using <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Parser/parsetok.c#L232"><code>parsetok()</code></a> in <code>Parser/parsetok.c</code> </li>
3810 </ol>
3811 <div class="highlight c"><pre><span></span><span class="n">node</span> <span class="o">*</span>
3812 <span class="nf">PyParser_ParseFileObject</span><span class="p">(</span><span class="kt">FILE</span> <span class="o">*</span><span class="n">fp</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">filename</span><span class="p">,</span>
3813 <span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">enc</span><span class="p">,</span> <span class="n">grammar</span> <span class="o">*</span><span class="n">g</span><span class="p">,</span> <span class="kt">int</span> <span class="n">start</span><span class="p">,</span>
3814 <span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">ps1</span><span class="p">,</span> <span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">ps2</span><span class="p">,</span>
3815 <span class="n">perrdetail</span> <span class="o">*</span><span class="n">err_ret</span><span class="p">,</span> <span class="kt">int</span> <span class="o">*</span><span class="n">flags</span><span class="p">)</span>
3816 <span class="p">{</span>
3817 <span class="hll"> <span class="k">struct</span> <span class="n">tok_state</span> <span class="o">*</span><span class="n">tok</span><span class="p">;</span>
3818 </span><span class="p">...</span>
3819 <span class="hll"> <span class="k">if</span> <span class="p">((</span><span class="n">tok</span> <span class="o">=</span> <span class="n">PyTokenizer_FromFile</span><span class="p">(</span><span class="n">fp</span><span class="p">,</span> <span class="n">enc</span><span class="p">,</span> <span class="n">ps1</span><span class="p">,</span> <span class="n">ps2</span><span class="p">))</span> <span class="o">==</span> <span class="nb">NULL</span><span class="p">)</span> <span class="p">{</span>
3820 </span> <span class="n">err_ret</span><span class="o">-></span><span class="n">error</span> <span class="o">=</span> <span class="n">E_NOMEM</span><span class="p">;</span>
3821 <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
3822 <span class="p">}</span>
3823 <span class="p">...</span>
3824 <span class="hll"> <span class="k">return</span> <span class="n">parsetok</span><span class="p">(</span><span class="n">tok</span><span class="p">,</span> <span class="n">g</span><span class="p">,</span> <span class="n">start</span><span class="p">,</span> <span class="n">err_ret</span><span class="p">,</span> <span class="n">flags</span><span class="p">);</span>
3825 </span><span class="p">}</span>
3826 </pre></div>
3827
3828 <p><code>tok_state</code> (defined in <code>Parser/tokenizer.h</code>) is the data structure to store all temporary data generated by the tokenizer. It is returned to the parser-tokenizer as the data structure is required by <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Parser/parsetok.c#L232"><code>parsetok()</code></a> to develop the concrete syntax tree.</p>
3829 <p>Inside <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Parser/parsetok.c#L232"><code>parsetok()</code></a>, it will use the <code>tok_state</code> structure and make calls to <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Parser/tokenizer.c#L1110"><code>tok_get()</code></a> in a loop until the file is exhausted and no more tokens can be found.</p>
3830 <p><a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Parser/tokenizer.c#L1110"><code>tok_get()</code></a>, defined in <code>Parser/tokenizer.c</code> behaves like an iterator. It will keep returning the next token in the parse tree.</p>
3831 <p><a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Parser/tokenizer.c#L1110"><code>tok_get()</code></a> is one of the most complex functions in the whole CPython codebase. It has over 640 lines and includes decades of heritage with edge cases, new language features, and syntax.</p>
3832 <p>One of the simpler examples would be the part that converts a newline break into a NEWLINE token:</p>
3833 <div class="highlight c"><pre><span></span><span class="k">static</span> <span class="kt">int</span>
3834 <span class="nf">tok_get</span><span class="p">(</span><span class="k">struct</span> <span class="n">tok_state</span> <span class="o">*</span><span class="n">tok</span><span class="p">,</span> <span class="kt">char</span> <span class="o">**</span><span class="n">p_start</span><span class="p">,</span> <span class="kt">char</span> <span class="o">**</span><span class="n">p_end</span><span class="p">)</span>
3835 <span class="p">{</span>
3836 <span class="p">...</span>
3837 <span class="cm">/* Newline */</span>
3838 <span class="k">if</span> <span class="p">(</span><span class="n">c</span> <span class="o">==</span> <span class="sc">'\n'</span><span class="p">)</span> <span class="p">{</span>
3839 <span class="n">tok</span><span class="o">-></span><span class="n">atbol</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
3840 <span class="k">if</span> <span class="p">(</span><span class="n">blankline</span> <span class="o">||</span> <span class="n">tok</span><span class="o">-></span><span class="n">level</span> <span class="o">></span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
3841 <span class="k">goto</span> <span class="n">nextline</span><span class="p">;</span>
3842 <span class="p">}</span>
3843 <span class="o">*</span><span class="n">p_start</span> <span class="o">=</span> <span class="n">tok</span><span class="o">-></span><span class="n">start</span><span class="p">;</span>
3844 <span class="o">*</span><span class="n">p_end</span> <span class="o">=</span> <span class="n">tok</span><span class="o">-></span><span class="n">cur</span> <span class="o">-</span> <span class="mi">1</span><span class="p">;</span> <span class="cm">/* Leave '\n' out of the string */</span>
3845 <span class="n">tok</span><span class="o">-></span><span class="n">cont_line</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
3846 <span class="k">if</span> <span class="p">(</span><span class="n">tok</span><span class="o">-></span><span class="n">async_def</span><span class="p">)</span> <span class="p">{</span>
3847 <span class="cm">/* We're somewhere inside an 'async def' function, and</span>
3848 <span class="cm"> we've encountered a NEWLINE after its signature. */</span>
3849 <span class="n">tok</span><span class="o">-></span><span class="n">async_def_nl</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
3850 <span class="p">}</span>
3851 <span class="k">return</span> <span class="n">NEWLINE</span><span class="p">;</span>
3852 <span class="p">}</span>
3853 <span class="p">...</span>
3854 <span class="p">}</span>
3855 </pre></div>
3856
3857 <p>In this case, <code>NEWLINE</code> is a token, with a value defined in <code>Include/token.h</code>. All tokens are constant <code>int</code> values, and the <code>Include/token.h</code> file was generated earlier when we ran <code>make regen-grammar</code>.</p>
3858 <p>The <code>node</code> type returned by <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Parser/parsetok.c#L163"><code>PyParser_ParseFileObject()</code></a> is going to be essential for the next stage, converting a parse tree into an Abstract-Syntax-Tree (AST):</p>
3859 <div class="highlight c"><pre><span></span><span class="k">typedef</span> <span class="k">struct</span> <span class="n">_node</span> <span class="p">{</span>
3860 <span class="kt">short</span> <span class="n">n_type</span><span class="p">;</span>
3861 <span class="kt">char</span> <span class="o">*</span><span class="n">n_str</span><span class="p">;</span>
3862 <span class="kt">int</span> <span class="n">n_lineno</span><span class="p">;</span>
3863 <span class="kt">int</span> <span class="n">n_col_offset</span><span class="p">;</span>
3864 <span class="kt">int</span> <span class="n">n_nchildren</span><span class="p">;</span>
3865 <span class="k">struct</span> <span class="n">_node</span> <span class="o">*</span><span class="n">n_child</span><span class="p">;</span>
3866 <span class="kt">int</span> <span class="n">n_end_lineno</span><span class="p">;</span>
3867 <span class="kt">int</span> <span class="n">n_end_col_offset</span><span class="p">;</span>
3868 <span class="p">}</span> <span class="n">node</span><span class="p">;</span>
3869 </pre></div>
3870
3871 <p>Since the CST is a tree of syntax, token IDs, and symbols, it would be difficult for the compiler to make quick decisions based on the Python language.</p>
3872 <p>That is why the next stage is to convert the CST into an AST, a much higher-level structure. This task is performed by the <code>Python/ast.c</code> module, which has both a C and Python API.</p>
3873 <p>Before you jump into the AST, there is a way to access the output from the parser stage. CPython has a standard library module <code>parser</code>, which exposes the C functions with a Python API.</p>
3874 <p>The module is documented as an implementation detail of CPython so that you won’t see it in other Python interpreters. Also the output from the functions is not that easy to read.</p>
3875 <p>The output will be in the numeric form, using the token and symbol numbers generated by the <code>make regen-grammar</code> stage, stored in <code>Include/token.h</code>: </p>
3876 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pprint</span> <span class="k">import</span> <span class="n">pprint</span>
3877 <span class="gp">>>> </span><span class="kn">import</span> <span class="nn">parser</span>
3878 <span class="gp">>>> </span><span class="n">st</span> <span class="o">=</span> <span class="n">parser</span><span class="o">.</span><span class="n">expr</span><span class="p">(</span><span class="s1">'a + 1'</span><span class="p">)</span>
3879 <span class="gp">>>> </span><span class="n">pprint</span><span class="p">(</span><span class="n">parser</span><span class="o">.</span><span class="n">st2list</span><span class="p">(</span><span class="n">st</span><span class="p">))</span>
3880 <span class="go">[258,</span>
3881 <span class="go"> [332,</span>
3882 <span class="go"> [306,</span>
3883 <span class="go"> [310,</span>
3884 <span class="go"> [311,</span>
3885 <span class="go"> [312,</span>
3886 <span class="go"> [313,</span>
3887 <span class="go"> [316,</span>
3888 <span class="go"> [317,</span>
3889 <span class="go"> [318,</span>
3890 <span class="go"> [319,</span>
3891 <span class="go"> [320,</span>
3892 <span class="go"> [321, [322, [323, [324, [325, [1, 'a']]]]]],</span>
3893 <span class="go"> [14, '+'],</span>
3894 <span class="go"> [321, [322, [323, [324, [325, [2, '1']]]]]]]]]]]]]]]]],</span>
3895 <span class="go"> [4, ''],</span>
3896 <span class="go"> [0, '']]</span>
3897 </pre></div>
3898
3899 <p>To make it easier to understand, you can take all the numbers in the <code>symbol</code> and <code>token</code> modules, put them into a dictionary and recursively replace the values in the output of <code>parser.st2list()</code> with the names:</p>
3900 <div class="highlight python"><pre><span></span><span class="kn">import</span> <span class="nn">symbol</span>
3901 <span class="kn">import</span> <span class="nn">token</span>
3902 <span class="kn">import</span> <span class="nn">parser</span>
3903
3904 <span class="k">def</span> <span class="nf">lex</span><span class="p">(</span><span class="n">expression</span><span class="p">):</span>
3905 <span class="n">symbols</span> <span class="o">=</span> <span class="p">{</span><span class="n">v</span><span class="p">:</span> <span class="n">k</span> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">symbol</span><span class="o">.</span><span class="vm">__dict__</span><span class="o">.</span><span class="n">items</span><span class="p">()</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="nb">int</span><span class="p">)}</span>
3906 <span class="n">tokens</span> <span class="o">=</span> <span class="p">{</span><span class="n">v</span><span class="p">:</span> <span class="n">k</span> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">token</span><span class="o">.</span><span class="vm">__dict__</span><span class="o">.</span><span class="n">items</span><span class="p">()</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="nb">int</span><span class="p">)}</span>
3907 <span class="n">lexicon</span> <span class="o">=</span> <span class="p">{</span><span class="o">**</span><span class="n">symbols</span><span class="p">,</span> <span class="o">**</span><span class="n">tokens</span><span class="p">}</span>
3908 <span class="n">st</span> <span class="o">=</span> <span class="n">parser</span><span class="o">.</span><span class="n">expr</span><span class="p">(</span><span class="n">expression</span><span class="p">)</span>
3909 <span class="n">st_list</span> <span class="o">=</span> <span class="n">parser</span><span class="o">.</span><span class="n">st2list</span><span class="p">(</span><span class="n">st</span><span class="p">)</span>
3910
3911 <span class="k">def</span> <span class="nf">replace</span><span class="p">(</span><span class="n">l</span><span class="p">:</span> <span class="nb">list</span><span class="p">):</span>
3912 <span class="n">r</span> <span class="o">=</span> <span class="p">[]</span>
3913 <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">l</span><span class="p">:</span>
3914 <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="nb">list</span><span class="p">):</span>
3915 <span class="n">r</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">replace</span><span class="p">(</span><span class="n">i</span><span class="p">))</span>
3916 <span class="k">else</span><span class="p">:</span>
3917 <span class="k">if</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">lexicon</span><span class="p">:</span>
3918 <span class="n">r</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">lexicon</span><span class="p">[</span><span class="n">i</span><span class="p">])</span>
3919 <span class="k">else</span><span class="p">:</span>
3920 <span class="n">r</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">i</span><span class="p">)</span>
3921 <span class="k">return</span> <span class="n">r</span>
3922
3923 <span class="k">return</span> <span class="n">replace</span><span class="p">(</span><span class="n">st_list</span><span class="p">)</span>
3924 </pre></div>
3925
3926 <p>You can run <code>lex()</code> with a simple expression, like <code>a + 1</code> to see how this is represented as a parser-tree:</p>
3927 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pprint</span> <span class="k">import</span> <span class="n">pprint</span>
3928 <span class="gp">>>> </span><span class="n">pprint</span><span class="p">(</span><span class="n">lex</span><span class="p">(</span><span class="s1">'a + 1'</span><span class="p">))</span>
3929
3930 <span class="go">['eval_input',</span>
3931 <span class="go"> ['testlist',</span>
3932 <span class="go"> ['test',</span>
3933 <span class="go"> ['or_test',</span>
3934 <span class="go"> ['and_test',</span>
3935 <span class="go"> ['not_test',</span>
3936 <span class="go"> ['comparison',</span>
3937 <span class="go"> ['expr',</span>
3938 <span class="go"> ['xor_expr',</span>
3939 <span class="go"> ['and_expr',</span>
3940 <span class="go"> ['shift_expr',</span>
3941 <span class="go"> ['arith_expr',</span>
3942 <span class="go"> ['term',</span>
3943 <span class="go"> ['factor', ['power', ['atom_expr', ['atom', ['NAME', 'a']]]]]],</span>
3944 <span class="go"> ['PLUS', '+'],</span>
3945 <span class="go"> ['term',</span>
3946 <span class="go"> ['factor',</span>
3947 <span class="go"> ['power', ['atom_expr', ['atom', ['NUMBER', '1']]]]]]]]]]]]]]]]],</span>
3948 <span class="go"> ['NEWLINE', ''],</span>
3949 <span class="go"> ['ENDMARKER', '']]</span>
3950 </pre></div>
3951
3952 <p>In the output, you can see the symbols in lowercase, such as <code>'test'</code> and the tokens in uppercase, such as <code>'NUMBER'</code>.</p>
3953 <h3 id="abstract-syntax-trees">Abstract Syntax Trees</h3>
3954 <p>The next stage in the CPython interpreter is to convert the CST generated by the parser into something more logical that can be executed. The structure is a higher-level representation of the code, called an Abstract Syntax Tree (AST).</p>
3955 <p>ASTs are produced inline with the CPython interpreter process, but you can also generate them in both Python using the <code>ast</code> module in the Standard Library as well as through the C API.</p>
3956 <p>Before diving into the C implementation of the AST, it would be useful to understand what an AST looks like for a simple piece of Python code.</p>
3957 <p>To do this, here’s a simple app called <code>instaviz</code> for this tutorial. It displays the AST and bytecode instructions (which we’ll cover later) in a Web UI.</p>
3958 <p>To install <code>instaviz</code>:</p>
3959 <div class="highlight sh"><pre><span></span><span class="gp">$</span> pip install instaviz
3960 </pre></div>
3961
3962 <p>Then, open up a REPL by running <code>python</code> at the command line with no arguments:</p>
3963 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">instaviz</span>
3964 <span class="gp">>>> </span><span class="k">def</span> <span class="nf">example</span><span class="p">():</span>
3965 <span class="go"> a = 1</span>
3966 <span class="go"> b = a + 1</span>
3967 <span class="go"> return b</span>
3968
3969 <span class="gp">>>> </span><span class="n">instaviz</span><span class="o">.</span><span class="n">show</span><span class="p">(</span><span class="n">example</span><span class="p">)</span>
3970 </pre></div>
3971
3972 <p>You’ll see a notification on the command-line that a web server has started on port <code>8080</code>. If you were using that port for something else, you can change it by calling <code>instaviz.show(example, port=9090)</code> or another port number.</p>
3973 <p>In the web browser, you can see the detailed breakdown of your function:</p>
3974 <p><a href="https://files.realpython.com/media/screenshot.e148c89e3a9a.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/screenshot.e148c89e3a9a.png" width="4802" height="2566" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/screenshot.e148c89e3a9a.png&w=1200&sig=eeb7b21839b90e3726ea5db7bfa0aac05730b1fe 1200w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/screenshot.e148c89e3a9a.png&w=2401&sig=aed4aed922bbe4ef9c665a90c9244ebaa950e032 2401w, https://files.realpython.com/media/screenshot.e148c89e3a9a.png 4802w" sizes="75vw" alt="Instaviz screenshot"/></a></p>
3975 <p>The bottom left graph is the function you declared in REPL, represented as an Abstract Syntax Tree. Each node in the tree is an AST type. They are found in the <code>ast</code> module, and all inherit from <code>_ast.AST</code>. </p>
3976 <p>Some of the nodes have properties which link them to child nodes, unlike the CST, which has a generic child node property. </p>
3977 <p>For example, if you click on the Assign node in the center, this links to the line <code>b = a + 1</code>:</p>
3978 <p><a href="https://files.realpython.com/media/Screen_Shot_2019-03-19_at_1.24.17_pm.a5df8d873988.png" target="_blank"><img class="img-fluid mx-auto d-block border w-75" src="https://files.realpython.com/media/Screen_Shot_2019-03-19_at_1.24.17_pm.a5df8d873988.png" width="2226" height="1596" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screen_Shot_2019-03-19_at_1.24.17_pm.a5df8d873988.png&w=556&sig=6a6f30034c85f411bd6c159df8aef50e899dda9c 556w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screen_Shot_2019-03-19_at_1.24.17_pm.a5df8d873988.png&w=1113&sig=1b85cbf0bec4b114ef52d8a7bcdbfa08a1519237 1113w, https://files.realpython.com/media/Screen_Shot_2019-03-19_at_1.24.17_pm.a5df8d873988.png 2226w" sizes="75vw" alt="Instaviz screenshot 2"/></a></p>
3979 <p>It has two properties:</p>
3980 <ol>
3981 <li><strong><code>targets</code></strong> is a list of names to assign. It is a list because you can assign to multiple variables with a single expression using unpacking</li>
3982 <li><strong><code>value</code></strong> is the value to assign, which in this case is a <code>BinOp</code> statement, <code>a + 1</code>.</li>
3983 </ol>
3984 <p>If you click on the <code>BinOp</code> statement, it shows the properties of relevance:</p>
3985 <ul>
3986 <li><strong><code>left</code>:</strong> the node to the left of the operator</li>
3987 <li><strong><code>op</code>:</strong> the operator, in this case, an <code>Add</code> node (<code>+</code>) for addition</li>
3988 <li><strong><code>right</code>:</strong> the node to the right of the operator</li>
3989 </ul>
3990 <p><a href="https://files.realpython.com/media/Screen_Shot_2019-03-19_at_1.24.37_pm.21a11b49a820.png" target="_blank"><img class="img-fluid mx-auto d-block border w-75" src="https://files.realpython.com/media/Screen_Shot_2019-03-19_at_1.24.37_pm.21a11b49a820.png" width="1708" height="932" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screen_Shot_2019-03-19_at_1.24.37_pm.21a11b49a820.png&w=427&sig=381d8bee4cc98dc98031abc6bc34ec376fd452b7 427w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screen_Shot_2019-03-19_at_1.24.37_pm.21a11b49a820.png&w=854&sig=ccc0b88e9d762b2298a33ca2a03e826ee13f4193 854w, https://files.realpython.com/media/Screen_Shot_2019-03-19_at_1.24.37_pm.21a11b49a820.png 1708w" sizes="75vw" alt="Instaviz screenshot 3"/></a></p>
3991 <p>Compiling an AST in C is not a straightforward task, so the <code>Python/ast.c</code> module is over 5000 lines of code.</p>
3992 <p>There are a few entry points, forming part of the AST’s public API. In the last section on the lexer and parser, you stopped when you’d reached the call to <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/ast.c#L772"><code>PyAST_FromNodeObject()</code></a>. By this stage, the Python interpreter process had created a CST in the format of <code>node *</code> tree.</p>
3993 <p>Jumping then into <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/ast.c#L772"><code>PyAST_FromNodeObject()</code></a> inside <code>Python/ast.c</code>, you can see it receives the <code>node *</code> tree, the filename, compiler flags, and the <code>PyArena</code>.</p>
3994 <p>The return type from this function is <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Include/ast.h#L10"><code>mod_ty</code></a>, defined in <code>Include/Python-ast.h</code>. <code>mod_ty</code> is a container structure for one of the 5 module types in Python:</p>
3995 <ol>
3996 <li><code>Module</code> </li>
3997 <li><code>Interactive</code></li>
3998 <li><code>Expression</code></li>
3999 <li><code>FunctionType</code></li>
4000 <li><code>Suite</code></li>
4001 </ol>
4002 <p>In <code>Include/Python-ast.h</code> you can see that an <code>Expression</code> type requires a field <code>body</code>, which is an <code>expr_ty</code> type. The <code>expr_ty</code> type is also defined in <code>Include/Python-ast.h</code>:</p>
4003 <div class="highlight c"><pre><span></span><span class="k">enum</span> <span class="n">_mod_kind</span> <span class="p">{</span><span class="n">Module_kind</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">Interactive_kind</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">Expression_kind</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span>
4004 <span class="n">FunctionType_kind</span><span class="o">=</span><span class="mi">4</span><span class="p">,</span> <span class="n">Suite_kind</span><span class="o">=</span><span class="mi">5</span><span class="p">};</span>
4005 <span class="k">struct</span> <span class="n">_mod</span> <span class="p">{</span>
4006 <span class="k">enum</span> <span class="n">_mod_kind</span> <span class="n">kind</span><span class="p">;</span>
4007 <span class="k">union</span> <span class="p">{</span>
4008 <span class="k">struct</span> <span class="p">{</span>
4009 <span class="n">asdl_seq</span> <span class="o">*</span><span class="n">body</span><span class="p">;</span>
4010 <span class="n">asdl_seq</span> <span class="o">*</span><span class="n">type_ignores</span><span class="p">;</span>
4011 <span class="p">}</span> <span class="n">Module</span><span class="p">;</span>
4012
4013 <span class="k">struct</span> <span class="p">{</span>
4014 <span class="n">asdl_seq</span> <span class="o">*</span><span class="n">body</span><span class="p">;</span>
4015 <span class="p">}</span> <span class="n">Interactive</span><span class="p">;</span>
4016
4017 <span class="k">struct</span> <span class="p">{</span>
4018 <span class="n">expr_ty</span> <span class="n">body</span><span class="p">;</span>
4019 <span class="p">}</span> <span class="n">Expression</span><span class="p">;</span>
4020
4021 <span class="k">struct</span> <span class="p">{</span>
4022 <span class="n">asdl_seq</span> <span class="o">*</span><span class="n">argtypes</span><span class="p">;</span>
4023 <span class="n">expr_ty</span> <span class="n">returns</span><span class="p">;</span>
4024 <span class="p">}</span> <span class="n">FunctionType</span><span class="p">;</span>
4025
4026 <span class="k">struct</span> <span class="p">{</span>
4027 <span class="n">asdl_seq</span> <span class="o">*</span><span class="n">body</span><span class="p">;</span>
4028 <span class="p">}</span> <span class="n">Suite</span><span class="p">;</span>
4029
4030 <span class="p">}</span> <span class="n">v</span><span class="p">;</span>
4031 <span class="p">};</span>
4032 </pre></div>
4033
4034 <p>The AST types are all listed in <code>Parser/Python.asdl</code>. You will see the module types, statement types, expression types, operators, and comprehensions all listed. The names of the types in this document relate to the classes generated by the AST and the same classes named in the <code>ast</code> standard module library.</p>
4035 <p>The parameters and names in <code>Include/Python-ast.h</code> correlate directly to those specified in <code>Parser/Python.asdl</code>:</p>
4036 <div class="highlight text"><pre><span></span>-- ASDL's 5 builtin types are:
4037 -- identifier, int, string, object, constant
4038
4039 module Python
4040 {
4041 mod = Module(stmt* body, type_ignore *type_ignores)
4042 | Interactive(stmt* body)
4043 <span class="hll"> | Expression(expr body)
4044 </span> | FunctionType(expr* argtypes, expr returns)
4045 </pre></div>
4046
4047 <p>The C header file and structures are there so that the <code>Python/ast.c</code> program can quickly generate the structures with pointers to the relevant data.</p>
4048 <p>Looking at <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/ast.c#L772"><code>PyAST_FromNodeObject()</code></a> you can see that it is essentially a <code>switch</code> statement around the result from <code>TYPE(n)</code>. <code>TYPE()</code> is one of the core functions used by the AST to determine what type a node in the concrete syntax tree is. In the case of <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/ast.c#L772"><code>PyAST_FromNodeObject()</code></a> it’s just looking at the first node, so it can only be one of the module types defined as <code>Module</code>, <code>Interactive</code>, <code>Expression</code>, <code>FunctionType</code>.</p>
4049 <p>The result of <code>TYPE()</code> will be either a symbol or token type, which we’re very familiar with by this stage.</p>
4050 <p>For <code>file_input</code>, the results should be a <code>Module</code>. Modules are a series of statements, of which there are a few types. The logic to traverse the children of <code>n</code> and create statement nodes is within <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/ast.c#L4512"><code>ast_for_stmt()</code></a>. This function is called either once, if there is only 1 statement in the module, or in a loop if there are many. The resulting <code>Module</code> is then returned with the <code>PyArena</code>.</p>
4051 <p>For <code>eval_input</code>, the result should be an <code>Expression</code>. The result from <code>CHILD(n ,0)</code>, which is the first child of <code>n</code> is passed to <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/ast.c#L3246"><code>ast_for_testlist()</code></a> which returns an <code>expr_ty</code> type. This <code>expr_ty</code> is sent to <code>Expression()</code> with the PyArena to create an expression node, and then passed back as a result:</p>
4052 <div class="highlight c"><pre><span></span><span class="n">mod_ty</span>
4053 <span class="nf">PyAST_FromNodeObject</span><span class="p">(</span><span class="k">const</span> <span class="n">node</span> <span class="o">*</span><span class="n">n</span><span class="p">,</span> <span class="n">PyCompilerFlags</span> <span class="o">*</span><span class="n">flags</span><span class="p">,</span>
4054 <span class="n">PyObject</span> <span class="o">*</span><span class="n">filename</span><span class="p">,</span> <span class="n">PyArena</span> <span class="o">*</span><span class="n">arena</span><span class="p">)</span>
4055 <span class="p">{</span>
4056 <span class="p">...</span>
4057 <span class="k">switch</span> <span class="p">(</span><span class="n">TYPE</span><span class="p">(</span><span class="n">n</span><span class="p">))</span> <span class="p">{</span>
4058 <span class="k">case</span> <span class="nl">file_input</span><span class="p">:</span>
4059 <span class="n">stmts</span> <span class="o">=</span> <span class="n">_Py_asdl_seq_new</span><span class="p">(</span><span class="n">num_stmts</span><span class="p">(</span><span class="n">n</span><span class="p">),</span> <span class="n">arena</span><span class="p">);</span>
4060 <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">stmts</span><span class="p">)</span>
4061 <span class="k">goto</span> <span class="n">out</span><span class="p">;</span>
4062 <span class="k">for</span> <span class="p">(</span><span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">NCH</span><span class="p">(</span><span class="n">n</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
4063 <span class="n">ch</span> <span class="o">=</span> <span class="n">CHILD</span><span class="p">(</span><span class="n">n</span><span class="p">,</span> <span class="n">i</span><span class="p">);</span>
4064 <span class="k">if</span> <span class="p">(</span><span class="n">TYPE</span><span class="p">(</span><span class="n">ch</span><span class="p">)</span> <span class="o">==</span> <span class="n">NEWLINE</span><span class="p">)</span>
4065 <span class="k">continue</span><span class="p">;</span>
4066 <span class="n">REQ</span><span class="p">(</span><span class="n">ch</span><span class="p">,</span> <span class="n">stmt</span><span class="p">);</span>
4067 <span class="n">num</span> <span class="o">=</span> <span class="n">num_stmts</span><span class="p">(</span><span class="n">ch</span><span class="p">);</span>
4068 <span class="k">if</span> <span class="p">(</span><span class="n">num</span> <span class="o">==</span> <span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
4069 <span class="hll"> <span class="n">s</span> <span class="o">=</span> <span class="n">ast_for_stmt</span><span class="p">(</span><span class="o">&</span><span class="n">c</span><span class="p">,</span> <span class="n">ch</span><span class="p">);</span>
4070 </span> <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">s</span><span class="p">)</span>
4071 <span class="k">goto</span> <span class="n">out</span><span class="p">;</span>
4072 <span class="n">asdl_seq_SET</span><span class="p">(</span><span class="n">stmts</span><span class="p">,</span> <span class="n">k</span><span class="o">++</span><span class="p">,</span> <span class="n">s</span><span class="p">);</span>
4073 <span class="p">}</span>
4074 <span class="k">else</span> <span class="p">{</span>
4075 <span class="n">ch</span> <span class="o">=</span> <span class="n">CHILD</span><span class="p">(</span><span class="n">ch</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span>
4076 <span class="n">REQ</span><span class="p">(</span><span class="n">ch</span><span class="p">,</span> <span class="n">simple_stmt</span><span class="p">);</span>
4077 <span class="k">for</span> <span class="p">(</span><span class="n">j</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">j</span> <span class="o"><</span> <span class="n">num</span><span class="p">;</span> <span class="n">j</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
4078 <span class="hll"> <span class="n">s</span> <span class="o">=</span> <span class="n">ast_for_stmt</span><span class="p">(</span><span class="o">&</span><span class="n">c</span><span class="p">,</span> <span class="n">CHILD</span><span class="p">(</span><span class="n">ch</span><span class="p">,</span> <span class="n">j</span> <span class="o">*</span> <span class="mi">2</span><span class="p">));</span>
4079 </span> <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">s</span><span class="p">)</span>
4080 <span class="k">goto</span> <span class="n">out</span><span class="p">;</span>
4081 <span class="n">asdl_seq_SET</span><span class="p">(</span><span class="n">stmts</span><span class="p">,</span> <span class="n">k</span><span class="o">++</span><span class="p">,</span> <span class="n">s</span><span class="p">);</span>
4082 <span class="p">}</span>
4083 <span class="p">}</span>
4084 <span class="p">}</span>
4085
4086 <span class="cm">/* Type ignores are stored under the ENDMARKER in file_input. */</span>
4087 <span class="p">...</span>
4088
4089 <span class="hll"> <span class="n">res</span> <span class="o">=</span> <span class="n">Module</span><span class="p">(</span><span class="n">stmts</span><span class="p">,</span> <span class="n">type_ignores</span><span class="p">,</span> <span class="n">arena</span><span class="p">);</span>
4090 </span> <span class="k">break</span><span class="p">;</span>
4091 <span class="k">case</span> <span class="nl">eval_input</span><span class="p">:</span> <span class="p">{</span>
4092 <span class="n">expr_ty</span> <span class="n">testlist_ast</span><span class="p">;</span>
4093
4094 <span class="cm">/* XXX Why not comp_for here? */</span>
4095 <span class="hll"> <span class="n">testlist_ast</span> <span class="o">=</span> <span class="n">ast_for_testlist</span><span class="p">(</span><span class="o">&</span><span class="n">c</span><span class="p">,</span> <span class="n">CHILD</span><span class="p">(</span><span class="n">n</span><span class="p">,</span> <span class="mi">0</span><span class="p">));</span>
4096 </span> <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">testlist_ast</span><span class="p">)</span>
4097 <span class="k">goto</span> <span class="n">out</span><span class="p">;</span>
4098 <span class="hll"> <span class="n">res</span> <span class="o">=</span> <span class="n">Expression</span><span class="p">(</span><span class="n">testlist_ast</span><span class="p">,</span> <span class="n">arena</span><span class="p">);</span>
4099 </span> <span class="k">break</span><span class="p">;</span>
4100 <span class="p">}</span>
4101 <span class="k">case</span> <span class="nl">single_input</span><span class="p">:</span>
4102 <span class="p">...</span>
4103 <span class="k">break</span><span class="p">;</span>
4104 <span class="k">case</span> <span class="nl">func_type_input</span><span class="p">:</span>
4105 <span class="p">...</span>
4106 <span class="p">...</span>
4107 <span class="k">return</span> <span class="n">res</span><span class="p">;</span>
4108 <span class="p">}</span>
4109 </pre></div>
4110
4111 <p>Inside the <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/ast.c#L4512"><code>ast_for_stmt()</code></a> function, there is another <code>switch</code> statement for each possible statement type (<code>simple_stmt</code>, <code>compound_stmt</code>, and so on) and the code to determine the arguments to the node class.</p>
4112 <p>One of the simpler functions is for the power expression, i.e., <code>2**4</code> is 2 to the power of 4. This function starts by getting the <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/ast.c#L2770"><code>ast_for_atom_expr()</code></a>, which is the number <code>2</code> in our example, then if that has one child, it returns the atomic expression. If it has more than one child, it will get the right-hand (the number <code>4</code>) and return a <code>BinOp</code> (binary operation) with the operator as <code>Pow</code> (power), the left hand of <code>e</code> (2), and the right hand of <code>f</code> (4):</p>
4113 <div class="highlight c"><pre><span></span><span class="k">static</span> <span class="n">expr_ty</span>
4114 <span class="nf">ast_for_power</span><span class="p">(</span><span class="k">struct</span> <span class="n">compiling</span> <span class="o">*</span><span class="n">c</span><span class="p">,</span> <span class="k">const</span> <span class="n">node</span> <span class="o">*</span><span class="n">n</span><span class="p">)</span>
4115 <span class="p">{</span>
4116 <span class="cm">/* power: atom trailer* ('**' factor)*</span>
4117 <span class="cm"> */</span>
4118 <span class="n">expr_ty</span> <span class="n">e</span><span class="p">;</span>
4119 <span class="n">REQ</span><span class="p">(</span><span class="n">n</span><span class="p">,</span> <span class="n">power</span><span class="p">);</span>
4120 <span class="n">e</span> <span class="o">=</span> <span class="n">ast_for_atom_expr</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">CHILD</span><span class="p">(</span><span class="n">n</span><span class="p">,</span> <span class="mi">0</span><span class="p">));</span>
4121 <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">e</span><span class="p">)</span>
4122 <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
4123 <span class="k">if</span> <span class="p">(</span><span class="n">NCH</span><span class="p">(</span><span class="n">n</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span><span class="p">)</span>
4124 <span class="k">return</span> <span class="n">e</span><span class="p">;</span>
4125 <span class="k">if</span> <span class="p">(</span><span class="n">TYPE</span><span class="p">(</span><span class="n">CHILD</span><span class="p">(</span><span class="n">n</span><span class="p">,</span> <span class="n">NCH</span><span class="p">(</span><span class="n">n</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span><span class="p">))</span> <span class="o">==</span> <span class="n">factor</span><span class="p">)</span> <span class="p">{</span>
4126 <span class="n">expr_ty</span> <span class="n">f</span> <span class="o">=</span> <span class="n">ast_for_expr</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">CHILD</span><span class="p">(</span><span class="n">n</span><span class="p">,</span> <span class="n">NCH</span><span class="p">(</span><span class="n">n</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span><span class="p">));</span>
4127 <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">f</span><span class="p">)</span>
4128 <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
4129 <span class="n">e</span> <span class="o">=</span> <span class="n">BinOp</span><span class="p">(</span><span class="n">e</span><span class="p">,</span> <span class="n">Pow</span><span class="p">,</span> <span class="n">f</span><span class="p">,</span> <span class="n">LINENO</span><span class="p">(</span><span class="n">n</span><span class="p">),</span> <span class="n">n</span><span class="o">-></span><span class="n">n_col_offset</span><span class="p">,</span>
4130 <span class="n">n</span><span class="o">-></span><span class="n">n_end_lineno</span><span class="p">,</span> <span class="n">n</span><span class="o">-></span><span class="n">n_end_col_offset</span><span class="p">,</span> <span class="n">c</span><span class="o">-></span><span class="n">c_arena</span><span class="p">);</span>
4131 <span class="p">}</span>
4132 <span class="k">return</span> <span class="n">e</span><span class="p">;</span>
4133 <span class="p">}</span>
4134 </pre></div>
4135
4136 <p>You can see the result of this if you send a short function to the <code>instaviz</code> module:</p>
4137 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">def</span> <span class="nf">foo</span><span class="p">():</span>
4138 <span class="go"> 2**4</span>
4139 <span class="gp">>>> </span><span class="kn">import</span> <span class="nn">instaviz</span>
4140 <span class="gp">>>> </span><span class="n">instaviz</span><span class="o">.</span><span class="n">show</span><span class="p">(</span><span class="n">foo</span><span class="p">)</span>
4141 </pre></div>
4142
4143 <p><a href="https://files.realpython.com/media/Screen_Shot_2019-03-19_at_5.34.51_pm.c3a1e8d717f5.png" target="_blank"><img class="img-fluid mx-auto d-block border w-75" src="https://files.realpython.com/media/Screen_Shot_2019-03-19_at_5.34.51_pm.c3a1e8d717f5.png" width="1708" height="1094" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screen_Shot_2019-03-19_at_5.34.51_pm.c3a1e8d717f5.png&w=427&sig=68167794b71cec6447aa8fcb4d22ca738a2e2ce4 427w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screen_Shot_2019-03-19_at_5.34.51_pm.c3a1e8d717f5.png&w=854&sig=feeef6a7ce0b14d39c57956f77320125bcce43bf 854w, https://files.realpython.com/media/Screen_Shot_2019-03-19_at_5.34.51_pm.c3a1e8d717f5.png 1708w" sizes="75vw" alt="Instaviz screenshot 4"/></a></p>
4144 <p>In the UI you can also see the corresponding properties:</p>
4145 <p><a href="https://files.realpython.com/media/Screen_Shot_2019-03-19_at_5.36.34_pm.0067235460de.png" target="_blank"><img class="img-fluid mx-auto d-block border w-75" src="https://files.realpython.com/media/Screen_Shot_2019-03-19_at_5.36.34_pm.0067235460de.png" width="1708" height="630" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screen_Shot_2019-03-19_at_5.36.34_pm.0067235460de.png&w=427&sig=91663308ef854874e262756b2a28e598d263fff5 427w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screen_Shot_2019-03-19_at_5.36.34_pm.0067235460de.png&w=854&sig=fbfcdd33bcf685f91ebe0559b4135ead2b296d7e 854w, https://files.realpython.com/media/Screen_Shot_2019-03-19_at_5.36.34_pm.0067235460de.png 1708w" sizes="75vw" alt="Instaviz screenshot 5"/></a></p>
4146 <p>In summary, each statement type and expression has a corresponding <code>ast_for_*()</code> function to create it. The arguments are defined in <code>Parser/Python.asdl</code> and exposed via the <code>ast</code> module in the standard library. If an expression or statement has children, then it will call the corresponding <code>ast_for_*</code> child function in a depth-first traversal.</p>
4147 <h3 id="conclusion_1">Conclusion</h3>
4148 <p>CPython’s versatility and low-level execution API make it the ideal candidate for an embedded scripting engine. You will see CPython used in many UI applications, such as Game Design, 3D graphics and system automation. </p>
4149 <p>The interpreter process is flexible and efficient, and now you have an understanding of how it works you’re ready to understand the compiler.</p>
4150 <h2 h1="h1" id="part-3-the-cpython-compiler-and-execution-loop">Part 3: The CPython Compiler and Execution Loop</h2>
4151 <p>In Part 2, you saw how the CPython interpreter takes an input, such as a file or string, and converts it into a logical Abstract Syntax Tree. We’re still not at the stage where this code can be executed. Next, we have to go deeper to convert the Abstract Syntax Tree into a set of sequential commands that the CPU can understand. </p>
4152 <h3 id="compiling">Compiling</h3>
4153 <p>Now the interpreter has an AST with the properties required for each of the operations, functions, classes, and namespaces. It is the job of the compiler to turn the AST into something the CPU can understand.</p>
4154 <p>This compilation task is split into 2 parts:</p>
4155 <ol>
4156 <li>Traverse the tree and create a control-flow-graph, which represents the logical sequence for execution</li>
4157 <li>Convert the nodes in the CFG to smaller, executable statements, known as byte-code</li>
4158 </ol>
4159 <p>Earlier, we were looking at how files are executed, and the <code>PyRun_FileExFlags()</code> function in <code>Python/pythonrun.c</code>. Inside this function, we converted the <code>FILE</code> handle into a <code>mod</code>, of type <code>mod_ty</code>. This task was completed by <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pythonrun.c#L1369"><code>PyParser_ASTFromFileObject()</code></a>, which in turns calls the <code>tokenizer</code>, <code>parser-tokenizer</code> and then the AST:</p>
4160 <div class="highlight c"><pre><span></span><span class="n">PyObject</span> <span class="o">*</span>
4161 <span class="nf">PyRun_FileExFlags</span><span class="p">(</span><span class="kt">FILE</span> <span class="o">*</span><span class="n">fp</span><span class="p">,</span> <span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">filename_str</span><span class="p">,</span> <span class="kt">int</span> <span class="n">start</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">globals</span><span class="p">,</span>
4162 <span class="n">PyObject</span> <span class="o">*</span><span class="n">locals</span><span class="p">,</span> <span class="kt">int</span> <span class="n">closeit</span><span class="p">,</span> <span class="n">PyCompilerFlags</span> <span class="o">*</span><span class="n">flags</span><span class="p">)</span>
4163 <span class="p">{</span>
4164 <span class="p">...</span>
4165 <span class="hll"> <span class="n">mod</span> <span class="o">=</span> <span class="n">PyParser_ASTFromFileObject</span><span class="p">(</span><span class="n">fp</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="nb">NULL</span><span class="p">,</span> <span class="n">start</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span>
4166 </span> <span class="p">...</span>
4167 <span class="hll"> <span class="n">ret</span> <span class="o">=</span> <span class="n">run_mod</span><span class="p">(</span><span class="n">mod</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="n">globals</span><span class="p">,</span> <span class="n">locals</span><span class="p">,</span> <span class="n">flags</span><span class="p">,</span> <span class="n">arena</span><span class="p">);</span>
4168 </span><span class="p">}</span>
4169 </pre></div>
4170
4171 <p>The resulting module from the call to is sent to <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pythonrun.c#L1125"><code>run_mod()</code></a> still in <code>Python/pythonrun.c</code>. This is a small function that gets a <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Include/code.h#L69"><code>PyCodeObject</code></a> from <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/compile.c#L312"><code>PyAST_CompileObject()</code></a> and sends it on to <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pythonrun.c#L1094"><code>run_eval_code_obj()</code></a>. You will tackle <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pythonrun.c#L1094"><code>run_eval_code_obj()</code></a> in the next section:</p>
4172 <div class="highlight c"><pre><span></span><span class="k">static</span> <span class="n">PyObject</span> <span class="o">*</span>
4173 <span class="nf">run_mod</span><span class="p">(</span><span class="n">mod_ty</span> <span class="n">mod</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">filename</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">globals</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">locals</span><span class="p">,</span>
4174 <span class="n">PyCompilerFlags</span> <span class="o">*</span><span class="n">flags</span><span class="p">,</span> <span class="n">PyArena</span> <span class="o">*</span><span class="n">arena</span><span class="p">)</span>
4175 <span class="p">{</span>
4176 <span class="n">PyCodeObject</span> <span class="o">*</span><span class="n">co</span><span class="p">;</span>
4177 <span class="n">PyObject</span> <span class="o">*</span><span class="n">v</span><span class="p">;</span>
4178 <span class="hll"> <span class="n">co</span> <span class="o">=</span> <span class="n">PyAST_CompileObject</span><span class="p">(</span><span class="n">mod</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="n">flags</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="n">arena</span><span class="p">);</span>
4179 </span> <span class="k">if</span> <span class="p">(</span><span class="n">co</span> <span class="o">==</span> <span class="nb">NULL</span><span class="p">)</span>
4180 <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
4181
4182 <span class="k">if</span> <span class="p">(</span><span class="n">PySys_Audit</span><span class="p">(</span><span class="s">"exec"</span><span class="p">,</span> <span class="s">"O"</span><span class="p">,</span> <span class="n">co</span><span class="p">)</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
4183 <span class="n">Py_DECREF</span><span class="p">(</span><span class="n">co</span><span class="p">);</span>
4184 <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
4185 <span class="p">}</span>
4186
4187 <span class="hll"> <span class="n">v</span> <span class="o">=</span> <span class="n">run_eval_code_obj</span><span class="p">(</span><span class="n">co</span><span class="p">,</span> <span class="n">globals</span><span class="p">,</span> <span class="n">locals</span><span class="p">);</span>
4188 </span> <span class="n">Py_DECREF</span><span class="p">(</span><span class="n">co</span><span class="p">);</span>
4189 <span class="k">return</span> <span class="n">v</span><span class="p">;</span>
4190 <span class="p">}</span>
4191 </pre></div>
4192
4193 <p>The <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/compile.c#L312"><code>PyAST_CompileObject()</code></a> function is the main entry point to the CPython compiler. It takes a Python module as its primary argument, along with the name of the file, the globals, locals, and the <code>PyArena</code> all created earlier in the interpreter process.</p>
4194 <p>We’re starting to get into the guts of the CPython compiler now, with decades of development and Computer Science theory behind it. Don’t be put off by the language. Once we break down the compiler into logical steps, it’ll make sense.</p>
4195 <p>Before the compiler starts, a global compiler state is created. This type, <code>compiler</code> is defined in <code>Python/compile.c</code> and contains properties used by the compiler to remember the compiler flags, the stack, and the <code>PyArena</code>:</p>
4196 <div class="highlight c"><pre><span></span><span class="k">struct</span> <span class="n">compiler</span> <span class="p">{</span>
4197 <span class="n">PyObject</span> <span class="o">*</span><span class="n">c_filename</span><span class="p">;</span>
4198 <span class="k">struct</span> <span class="n">symtable</span> <span class="o">*</span><span class="n">c_st</span><span class="p">;</span>
4199 <span class="n">PyFutureFeatures</span> <span class="o">*</span><span class="n">c_future</span><span class="p">;</span> <span class="cm">/* pointer to module's __future__ */</span>
4200 <span class="n">PyCompilerFlags</span> <span class="o">*</span><span class="n">c_flags</span><span class="p">;</span>
4201
4202 <span class="kt">int</span> <span class="n">c_optimize</span><span class="p">;</span> <span class="cm">/* optimization level */</span>
4203 <span class="kt">int</span> <span class="n">c_interactive</span><span class="p">;</span> <span class="cm">/* true if in interactive mode */</span>
4204 <span class="kt">int</span> <span class="n">c_nestlevel</span><span class="p">;</span>
4205 <span class="kt">int</span> <span class="n">c_do_not_emit_bytecode</span><span class="p">;</span> <span class="cm">/* The compiler won't emit any bytecode</span>
4206 <span class="cm"> if this value is different from zero.</span>
4207 <span class="cm"> This can be used to temporarily visit</span>
4208 <span class="cm"> nodes without emitting bytecode to</span>
4209 <span class="cm"> check only errors. */</span>
4210
4211 <span class="n">PyObject</span> <span class="o">*</span><span class="n">c_const_cache</span><span class="p">;</span> <span class="cm">/* Python dict holding all constants,</span>
4212 <span class="cm"> including names tuple */</span>
4213 <span class="k">struct</span> <span class="n">compiler_unit</span> <span class="o">*</span><span class="n">u</span><span class="p">;</span> <span class="cm">/* compiler state for current block */</span>
4214 <span class="n">PyObject</span> <span class="o">*</span><span class="n">c_stack</span><span class="p">;</span> <span class="cm">/* Python list holding compiler_unit ptrs */</span>
4215 <span class="n">PyArena</span> <span class="o">*</span><span class="n">c_arena</span><span class="p">;</span> <span class="cm">/* pointer to memory allocation arena */</span>
4216 <span class="p">};</span>
4217 </pre></div>
4218
4219 <p>Inside <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/compile.c#L312"><code>PyAST_CompileObject()</code></a>, there are 11 main steps happening:</p>
4220 <ol>
4221 <li>Create an empty <code>__doc__</code> property to the module if it doesn’t exist.</li>
4222 <li>Create an empty <code>__annotations__</code> property to the module if it doesn’t exist.</li>
4223 <li>Set the filename of the global compiler state to the filename argument.</li>
4224 <li>Set the memory allocation arena for the compiler to the one used by the interpreter.</li>
4225 <li>Copy any <code>__future__</code> flags in the module to the future flags in the compiler.</li>
4226 <li>Merge runtime flags provided by the command-line or environment variables.</li>
4227 <li>Enable any <code>__future__</code> features in the compiler.</li>
4228 <li>Set the optimization level to the provided argument, or default.</li>
4229 <li>Build a symbol table from the module object.</li>
4230 <li>Run the compiler with the compiler state and return the code object.</li>
4231 <li>Free any allocated memory by the compiler.</li>
4232 </ol>
4233 <div class="highlight c"><pre><span></span><span class="n">PyCodeObject</span> <span class="o">*</span>
4234 <span class="nf">PyAST_CompileObject</span><span class="p">(</span><span class="n">mod_ty</span> <span class="n">mod</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">filename</span><span class="p">,</span> <span class="n">PyCompilerFlags</span> <span class="o">*</span><span class="n">flags</span><span class="p">,</span>
4235 <span class="kt">int</span> <span class="n">optimize</span><span class="p">,</span> <span class="n">PyArena</span> <span class="o">*</span><span class="n">arena</span><span class="p">)</span>
4236 <span class="p">{</span>
4237 <span class="k">struct</span> <span class="n">compiler</span> <span class="n">c</span><span class="p">;</span>
4238 <span class="n">PyCodeObject</span> <span class="o">*</span><span class="n">co</span> <span class="o">=</span> <span class="nb">NULL</span><span class="p">;</span>
4239 <span class="n">PyCompilerFlags</span> <span class="n">local_flags</span> <span class="o">=</span> <span class="n">_PyCompilerFlags_INIT</span><span class="p">;</span>
4240 <span class="kt">int</span> <span class="n">merged</span><span class="p">;</span>
4241 <span class="n">PyConfig</span> <span class="o">*</span><span class="n">config</span> <span class="o">=</span> <span class="o">&</span><span class="n">_PyInterpreterState_GET_UNSAFE</span><span class="p">()</span><span class="o">-></span><span class="n">config</span><span class="p">;</span>
4242 <span class="hll">
4243 </span> <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">__doc__</span><span class="p">)</span> <span class="p">{</span>
4244 <span class="n">__doc__</span> <span class="o">=</span> <span class="n">PyUnicode_InternFromString</span><span class="p">(</span><span class="s">"__doc__"</span><span class="p">);</span>
4245 <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">__doc__</span><span class="p">)</span>
4246 <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
4247 <span class="p">}</span>
4248 <span class="hll"> <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">__annotations__</span><span class="p">)</span> <span class="p">{</span>
4249 </span> <span class="n">__annotations__</span> <span class="o">=</span> <span class="n">PyUnicode_InternFromString</span><span class="p">(</span><span class="s">"__annotations__"</span><span class="p">);</span>
4250 <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">__annotations__</span><span class="p">)</span>
4251 <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
4252 <span class="p">}</span>
4253 <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">compiler_init</span><span class="p">(</span><span class="o">&</span><span class="n">c</span><span class="p">))</span>
4254 <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
4255 <span class="hll"> <span class="n">Py_INCREF</span><span class="p">(</span><span class="n">filename</span><span class="p">);</span>
4256 </span><span class="hll"> <span class="n">c</span><span class="p">.</span><span class="n">c_filename</span> <span class="o">=</span> <span class="n">filename</span><span class="p">;</span>
4257 </span><span class="hll"> <span class="n">c</span><span class="p">.</span><span class="n">c_arena</span> <span class="o">=</span> <span class="n">arena</span><span class="p">;</span>
4258 </span> <span class="n">c</span><span class="p">.</span><span class="n">c_future</span> <span class="o">=</span> <span class="n">PyFuture_FromASTObject</span><span class="p">(</span><span class="n">mod</span><span class="p">,</span> <span class="n">filename</span><span class="p">);</span>
4259 <span class="k">if</span> <span class="p">(</span><span class="n">c</span><span class="p">.</span><span class="n">c_future</span> <span class="o">==</span> <span class="nb">NULL</span><span class="p">)</span>
4260 <span class="k">goto</span> <span class="n">finally</span><span class="p">;</span>
4261 <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">flags</span><span class="p">)</span> <span class="p">{</span>
4262 <span class="n">flags</span> <span class="o">=</span> <span class="o">&</span><span class="n">local_flags</span><span class="p">;</span>
4263 <span class="hll"> <span class="p">}</span>
4264 </span><span class="hll"> <span class="n">merged</span> <span class="o">=</span> <span class="n">c</span><span class="p">.</span><span class="n">c_future</span><span class="o">-></span><span class="n">ff_features</span> <span class="o">|</span> <span class="n">flags</span><span class="o">-></span><span class="n">cf_flags</span><span class="p">;</span>
4265 </span> <span class="n">c</span><span class="p">.</span><span class="n">c_future</span><span class="o">-></span><span class="n">ff_features</span> <span class="o">=</span> <span class="n">merged</span><span class="p">;</span>
4266 <span class="n">flags</span><span class="o">-></span><span class="n">cf_flags</span> <span class="o">=</span> <span class="n">merged</span><span class="p">;</span>
4267 <span class="hll"> <span class="n">c</span><span class="p">.</span><span class="n">c_flags</span> <span class="o">=</span> <span class="n">flags</span><span class="p">;</span>
4268 </span> <span class="n">c</span><span class="p">.</span><span class="n">c_optimize</span> <span class="o">=</span> <span class="p">(</span><span class="n">optimize</span> <span class="o">==</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="o">?</span> <span class="n">config</span><span class="o">-></span><span class="nl">optimization_level</span> <span class="p">:</span> <span class="n">optimize</span><span class="p">;</span>
4269 <span class="n">c</span><span class="p">.</span><span class="n">c_nestlevel</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
4270 <span class="n">c</span><span class="p">.</span><span class="n">c_do_not_emit_bytecode</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
4271
4272 <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">_PyAST_Optimize</span><span class="p">(</span><span class="n">mod</span><span class="p">,</span> <span class="n">arena</span><span class="p">,</span> <span class="n">c</span><span class="p">.</span><span class="n">c_optimize</span><span class="p">))</span> <span class="p">{</span>
4273 <span class="k">goto</span> <span class="n">finally</span><span class="p">;</span>
4274 <span class="hll"> <span class="p">}</span>
4275 </span>
4276 <span class="n">c</span><span class="p">.</span><span class="n">c_st</span> <span class="o">=</span> <span class="n">PySymtable_BuildObject</span><span class="p">(</span><span class="n">mod</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="n">c</span><span class="p">.</span><span class="n">c_future</span><span class="p">);</span>
4277 <span class="k">if</span> <span class="p">(</span><span class="n">c</span><span class="p">.</span><span class="n">c_st</span> <span class="o">==</span> <span class="nb">NULL</span><span class="p">)</span> <span class="p">{</span>
4278 <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">PyErr_Occurred</span><span class="p">())</span>
4279 <span class="n">PyErr_SetString</span><span class="p">(</span><span class="n">PyExc_SystemError</span><span class="p">,</span> <span class="s">"no symtable"</span><span class="p">);</span>
4280 <span class="k">goto</span> <span class="n">finally</span><span class="p">;</span>
4281 <span class="hll"> <span class="p">}</span>
4282 </span>
4283 <span class="n">co</span> <span class="o">=</span> <span class="n">compiler_mod</span><span class="p">(</span><span class="o">&</span><span class="n">c</span><span class="p">,</span> <span class="n">mod</span><span class="p">);</span>
4284
4285 <span class="nl">finally</span><span class="p">:</span>
4286 <span class="n">compiler_free</span><span class="p">(</span><span class="o">&</span><span class="n">c</span><span class="p">);</span>
4287 <span class="n">assert</span><span class="p">(</span><span class="n">co</span> <span class="o">||</span> <span class="n">PyErr_Occurred</span><span class="p">());</span>
4288 <span class="k">return</span> <span class="n">co</span><span class="p">;</span>
4289 <span class="p">}</span>
4290 </pre></div>
4291
4292 <h4 id="future-flags-and-compiler-flags">Future Flags and Compiler Flags</h4>
4293 <p>Before the compiler runs, there are two types of flags to toggle the features inside the compiler. These come from two places:</p>
4294 <ol>
4295 <li>The interpreter state, which may have been command-line options, set in <code>pyconfig.h</code> or via environment variables</li>
4296 <li>The use of <code>__future__</code> statements inside the actual source code of the module</li>
4297 </ol>
4298 <p>To distinguish the two types of flags, think that the <code>__future__</code> flags are required because of the syntax or features in that specific module. For example, Python 3.7 introduced delayed evaluation of type hints through the <code>annotations</code> future flag:</p>
4299 <div class="highlight python"><pre><span></span><span class="kn">from</span> <span class="nn">__future__</span> <span class="k">import</span> <span class="n">annotations</span>
4300 </pre></div>
4301
4302 <p>The code after this statement might use unresolved type hints, so the <code>__future__</code> statement is required. Otherwise, the module wouldn’t import. It would be unmaintainable to manually request that the person importing the module enable this specific compiler flag.</p>
4303 <p>The other compiler flags are specific to the environment, so they might change the way the code executes or the way the compiler runs, but they shouldn’t link to the source in the same way that <code>__future__</code> statements do.</p>
4304 <p>One example of a compiler flag would be the <a href="https://docs.python.org/3/using/cmdline.html#cmdoption-o"><code>-O</code> flag for optimizing the use of <code>assert</code> statements</a>. This flag disables any <code>assert</code> statements, which may have been put in the code for <a href="https://realpython.com/python-debugging-pdb/">debugging purposes</a>.
4305 It can also be enabled with the <code>PYTHONOPTIMIZE=1</code> environment variable setting.</p>
4306 <h4 id="symbol-tables">Symbol Tables</h4>
4307 <p>In <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/compile.c#L312"><code>PyAST_CompileObject()</code></a> there was a reference to a <code>symtable</code> and a call to <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/symtable.c#L262"><code>PySymtable_BuildObject()</code></a> with the module to be executed.</p>
4308 <p>The purpose of the symbol table is to provide a list of namespaces, globals, and locals for the compiler to use for referencing and resolving scopes.</p>
4309 <p>The <code>symtable</code> structure in <code>Include/symtable.h</code> is well documented, so it’s clear what each of the fields is for. There should be one symtable instance for the compiler, so namespacing becomes essential. </p>
4310 <p>If you create a function called <code>resolve_names()</code> in one module and declare another function with the same name in another module, you want to be sure which one is called. The symtable serves this purpose, as well as ensuring that variables declared within a narrow scope don’t automatically become globals (after all, this isn’t JavaScript):</p>
4311 <div class="highlight c"><pre><span></span><span class="k">struct</span> <span class="n">symtable</span> <span class="p">{</span>
4312 <span class="n">PyObject</span> <span class="o">*</span><span class="n">st_filename</span><span class="p">;</span> <span class="cm">/* name of file being compiled,</span>
4313 <span class="cm"> decoded from the filesystem encoding */</span>
4314 <span class="k">struct</span> <span class="n">_symtable_entry</span> <span class="o">*</span><span class="n">st_cur</span><span class="p">;</span> <span class="cm">/* current symbol table entry */</span>
4315 <span class="k">struct</span> <span class="n">_symtable_entry</span> <span class="o">*</span><span class="n">st_top</span><span class="p">;</span> <span class="cm">/* symbol table entry for module */</span>
4316 <span class="n">PyObject</span> <span class="o">*</span><span class="n">st_blocks</span><span class="p">;</span> <span class="cm">/* dict: map AST node addresses</span>
4317 <span class="cm"> * to symbol table entries */</span>
4318 <span class="n">PyObject</span> <span class="o">*</span><span class="n">st_stack</span><span class="p">;</span> <span class="cm">/* list: stack of namespace info */</span>
4319 <span class="n">PyObject</span> <span class="o">*</span><span class="n">st_global</span><span class="p">;</span> <span class="cm">/* borrowed ref to st_top->ste_symbols */</span>
4320 <span class="kt">int</span> <span class="n">st_nblocks</span><span class="p">;</span> <span class="cm">/* number of blocks used. kept for</span>
4321 <span class="cm"> consistency with the corresponding</span>
4322 <span class="cm"> compiler structure */</span>
4323 <span class="n">PyObject</span> <span class="o">*</span><span class="n">st_private</span><span class="p">;</span> <span class="cm">/* name of current class or NULL */</span>
4324 <span class="n">PyFutureFeatures</span> <span class="o">*</span><span class="n">st_future</span><span class="p">;</span> <span class="cm">/* module's future features that affect</span>
4325 <span class="cm"> the symbol table */</span>
4326 <span class="kt">int</span> <span class="n">recursion_depth</span><span class="p">;</span> <span class="cm">/* current recursion depth */</span>
4327 <span class="kt">int</span> <span class="n">recursion_limit</span><span class="p">;</span> <span class="cm">/* recursion limit */</span>
4328 <span class="p">};</span>
4329 </pre></div>
4330
4331 <p>Some of the symbol table API is exposed via <a href="https://docs.python.org/3/library/symtable.html">the <code>symtable</code> module</a> in the standard library. You can provide an expression or a module an receive a <code>symtable.SymbolTable</code> instance.</p>
4332 <p>You can provide a string with a Python expression and the <code>compile_type</code> of <code>"eval"</code>, or a module, function or class, and the <code>compile_mode</code> of <code>"exec"</code> to get a symbol table.</p>
4333 <p>Looping over the elements in the table we can see some of the public and private fields and their types:</p>
4334 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">symtable</span>
4335 <span class="gp">>>> </span><span class="n">s</span> <span class="o">=</span> <span class="n">symtable</span><span class="o">.</span><span class="n">symtable</span><span class="p">(</span><span class="s1">'b + 1'</span><span class="p">,</span> <span class="n">filename</span><span class="o">=</span><span class="s1">'test.py'</span><span class="p">,</span> <span class="n">compile_type</span><span class="o">=</span><span class="s1">'eval'</span><span class="p">)</span>
4336 <span class="gp">>>> </span><span class="p">[</span><span class="n">symbol</span><span class="o">.</span><span class="vm">__dict__</span> <span class="k">for</span> <span class="n">symbol</span> <span class="ow">in</span> <span class="n">s</span><span class="o">.</span><span class="n">get_symbols</span><span class="p">()]</span>
4337 <span class="go">[{'_Symbol__name': 'b', '_Symbol__flags': 6160, '_Symbol__scope': 3, '_Symbol__namespaces': ()}]</span>
4338 </pre></div>
4339
4340 <p>The C code behind this is all within <code>Python/symtable.c</code> and the primary interface is the <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/symtable.c#L262"><code>PySymtable_BuildObject()</code></a> function.</p>
4341 <p>Similar to the top-level AST function we covered earlier, the <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/symtable.c#L262"><code>PySymtable_BuildObject()</code></a> function switches between the <code>mod_ty</code> possible types (Module, Expression, Interactive, Suite, FunctionType), and visits each of the statements inside them.</p>
4342 <p>Remember, <code>mod_ty</code> is an AST instance, so the will now recursively explore the nodes and branches of the tree and add entries to the symtable:</p>
4343 <div class="highlight c"><pre><span></span><span class="k">struct</span> <span class="n">symtable</span> <span class="o">*</span>
4344 <span class="nf">PySymtable_BuildObject</span><span class="p">(</span><span class="n">mod_ty</span> <span class="n">mod</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">filename</span><span class="p">,</span> <span class="n">PyFutureFeatures</span> <span class="o">*</span><span class="n">future</span><span class="p">)</span>
4345 <span class="p">{</span>
4346 <span class="hll"> <span class="k">struct</span> <span class="n">symtable</span> <span class="o">*</span><span class="n">st</span> <span class="o">=</span> <span class="n">symtable_new</span><span class="p">();</span>
4347 </span> <span class="n">asdl_seq</span> <span class="o">*</span><span class="n">seq</span><span class="p">;</span>
4348 <span class="kt">int</span> <span class="n">i</span><span class="p">;</span>
4349 <span class="n">PyThreadState</span> <span class="o">*</span><span class="n">tstate</span><span class="p">;</span>
4350 <span class="kt">int</span> <span class="n">recursion_limit</span> <span class="o">=</span> <span class="n">Py_GetRecursionLimit</span><span class="p">();</span>
4351 <span class="p">...</span>
4352 <span class="n">st</span><span class="o">-></span><span class="n">st_top</span> <span class="o">=</span> <span class="n">st</span><span class="o">-></span><span class="n">st_cur</span><span class="p">;</span>
4353 <span class="k">switch</span> <span class="p">(</span><span class="n">mod</span><span class="o">-></span><span class="n">kind</span><span class="p">)</span> <span class="p">{</span>
4354 <span class="k">case</span> <span class="nl">Module_kind</span><span class="p">:</span>
4355 <span class="n">seq</span> <span class="o">=</span> <span class="n">mod</span><span class="o">-></span><span class="n">v</span><span class="p">.</span><span class="n">Module</span><span class="p">.</span><span class="n">body</span><span class="p">;</span>
4356 <span class="k">for</span> <span class="p">(</span><span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">asdl_seq_LEN</span><span class="p">(</span><span class="n">seq</span><span class="p">);</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span>
4357 <span class="hll"> <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">symtable_visit_stmt</span><span class="p">(</span><span class="n">st</span><span class="p">,</span>
4358 </span> <span class="p">(</span><span class="n">stmt_ty</span><span class="p">)</span><span class="n">asdl_seq_GET</span><span class="p">(</span><span class="n">seq</span><span class="p">,</span> <span class="n">i</span><span class="p">)))</span>
4359 <span class="k">goto</span> <span class="n">error</span><span class="p">;</span>
4360 <span class="k">break</span><span class="p">;</span>
4361 <span class="k">case</span> <span class="nl">Expression_kind</span><span class="p">:</span>
4362 <span class="p">...</span>
4363 <span class="k">case</span> <span class="nl">Interactive_kind</span><span class="p">:</span>
4364 <span class="p">...</span>
4365 <span class="k">case</span> <span class="nl">Suite_kind</span><span class="p">:</span>
4366 <span class="p">...</span>
4367 <span class="k">case</span> <span class="nl">FunctionType_kind</span><span class="p">:</span>
4368 <span class="p">...</span>
4369 <span class="p">}</span>
4370 <span class="p">...</span>
4371 <span class="p">}</span>
4372 </pre></div>
4373
4374 <p>So for a module, <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/symtable.c#L262"><code>PySymtable_BuildObject()</code></a> will loop through each statement in the module and call <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/symtable.c#L1176"><code>symtable_visit_stmt()</code></a>. The <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/symtable.c#L1176"><code>symtable_visit_stmt()</code></a> is a huge <code>switch</code> statement with a case for each statement type (defined in <code>Parser/Python.asdl</code>).</p>
4375 <p>For each statement type, there is specific logic to that statement type. For example, a function definition has particular logic for:</p>
4376 <ol>
4377 <li>If the recursion depth is beyond the limit, raise a recursion depth error</li>
4378 <li>The name of the function to be added as a local variable</li>
4379 <li>The default values for sequential arguments to be resolved</li>
4380 <li>The default values for keyword arguments to be resolved </li>
4381 <li>Any annotations for the arguments or the return type are resolved</li>
4382 <li>Any function decorators are resolved</li>
4383 <li>The code block with the contents of the function is visited in <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/symtable.c#L973"><code>symtable_enter_block()</code></a></li>
4384 <li>The arguments are visited</li>
4385 <li>The body of the function is visited</li>
4386 </ol>
4387 <div class="alert alert-primary" role="alert">
4388 <p><strong>Note:</strong> If you’ve ever wondered why Python’s default arguments are mutable, the reason is in this function. You can see they are a pointer to the variable in the symtable. No extra work is done to copy any values to an immutable type.</p>
4389 </div>
4390 <div class="highlight c"><pre><span></span><span class="k">static</span> <span class="kt">int</span>
4391 <span class="nf">symtable_visit_stmt</span><span class="p">(</span><span class="k">struct</span> <span class="n">symtable</span> <span class="o">*</span><span class="n">st</span><span class="p">,</span> <span class="n">stmt_ty</span> <span class="n">s</span><span class="p">)</span>
4392 <span class="p">{</span>
4393 <span class="hll"> <span class="k">if</span> <span class="p">(</span><span class="o">++</span><span class="n">st</span><span class="o">-></span><span class="n">recursion_depth</span> <span class="o">></span> <span class="n">st</span><span class="o">-></span><span class="n">recursion_limit</span><span class="p">)</span> <span class="p">{</span> <span class="c1">// 1.</span>
4394 </span> <span class="n">PyErr_SetString</span><span class="p">(</span><span class="n">PyExc_RecursionError</span><span class="p">,</span>
4395 <span class="s">"maximum recursion depth exceeded during compilation"</span><span class="p">);</span>
4396 <span class="n">VISIT_QUIT</span><span class="p">(</span><span class="n">st</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span>
4397 <span class="p">}</span>
4398 <span class="k">switch</span> <span class="p">(</span><span class="n">s</span><span class="o">-></span><span class="n">kind</span><span class="p">)</span> <span class="p">{</span>
4399 <span class="k">case</span> <span class="nl">FunctionDef_kind</span><span class="p">:</span>
4400 <span class="hll"> <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">symtable_add_def</span><span class="p">(</span><span class="n">st</span><span class="p">,</span> <span class="n">s</span><span class="o">-></span><span class="n">v</span><span class="p">.</span><span class="n">FunctionDef</span><span class="p">.</span><span class="n">name</span><span class="p">,</span> <span class="n">DEF_LOCAL</span><span class="p">))</span> <span class="c1">// 2.</span>
4401 </span> <span class="n">VISIT_QUIT</span><span class="p">(</span><span class="n">st</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span>
4402 <span class="hll"> <span class="k">if</span> <span class="p">(</span><span class="n">s</span><span class="o">-></span><span class="n">v</span><span class="p">.</span><span class="n">FunctionDef</span><span class="p">.</span><span class="n">args</span><span class="o">-></span><span class="n">defaults</span><span class="p">)</span> <span class="c1">// 3.</span>
4403 </span> <span class="n">VISIT_SEQ</span><span class="p">(</span><span class="n">st</span><span class="p">,</span> <span class="n">expr</span><span class="p">,</span> <span class="n">s</span><span class="o">-></span><span class="n">v</span><span class="p">.</span><span class="n">FunctionDef</span><span class="p">.</span><span class="n">args</span><span class="o">-></span><span class="n">defaults</span><span class="p">);</span>
4404 <span class="hll"> <span class="k">if</span> <span class="p">(</span><span class="n">s</span><span class="o">-></span><span class="n">v</span><span class="p">.</span><span class="n">FunctionDef</span><span class="p">.</span><span class="n">args</span><span class="o">-></span><span class="n">kw_defaults</span><span class="p">)</span> <span class="c1">// 4.</span>
4405 </span> <span class="n">VISIT_SEQ_WITH_NULL</span><span class="p">(</span><span class="n">st</span><span class="p">,</span> <span class="n">expr</span><span class="p">,</span> <span class="n">s</span><span class="o">-></span><span class="n">v</span><span class="p">.</span><span class="n">FunctionDef</span><span class="p">.</span><span class="n">args</span><span class="o">-></span><span class="n">kw_defaults</span><span class="p">);</span>
4406 <span class="hll"> <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">symtable_visit_annotations</span><span class="p">(</span><span class="n">st</span><span class="p">,</span> <span class="n">s</span><span class="p">,</span> <span class="n">s</span><span class="o">-></span><span class="n">v</span><span class="p">.</span><span class="n">FunctionDef</span><span class="p">.</span><span class="n">args</span><span class="p">,</span> <span class="c1">// 5.</span>
4407 </span> <span class="n">s</span><span class="o">-></span><span class="n">v</span><span class="p">.</span><span class="n">FunctionDef</span><span class="p">.</span><span class="n">returns</span><span class="p">))</span>
4408 <span class="n">VISIT_QUIT</span><span class="p">(</span><span class="n">st</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span>
4409 <span class="hll"> <span class="k">if</span> <span class="p">(</span><span class="n">s</span><span class="o">-></span><span class="n">v</span><span class="p">.</span><span class="n">FunctionDef</span><span class="p">.</span><span class="n">decorator_list</span><span class="p">)</span> <span class="c1">// 6.</span>
4410 </span> <span class="n">VISIT_SEQ</span><span class="p">(</span><span class="n">st</span><span class="p">,</span> <span class="n">expr</span><span class="p">,</span> <span class="n">s</span><span class="o">-></span><span class="n">v</span><span class="p">.</span><span class="n">FunctionDef</span><span class="p">.</span><span class="n">decorator_list</span><span class="p">);</span>
4411 <span class="hll"> <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">symtable_enter_block</span><span class="p">(</span><span class="n">st</span><span class="p">,</span> <span class="n">s</span><span class="o">-></span><span class="n">v</span><span class="p">.</span><span class="n">FunctionDef</span><span class="p">.</span><span class="n">name</span><span class="p">,</span> <span class="c1">// 7.</span>
4412 </span> <span class="n">FunctionBlock</span><span class="p">,</span> <span class="p">(</span><span class="kt">void</span> <span class="o">*</span><span class="p">)</span><span class="n">s</span><span class="p">,</span> <span class="n">s</span><span class="o">-></span><span class="n">lineno</span><span class="p">,</span>
4413 <span class="n">s</span><span class="o">-></span><span class="n">col_offset</span><span class="p">))</span>
4414 <span class="n">VISIT_QUIT</span><span class="p">(</span><span class="n">st</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span>
4415 <span class="hll"> <span class="n">VISIT</span><span class="p">(</span><span class="n">st</span><span class="p">,</span> <span class="n">arguments</span><span class="p">,</span> <span class="n">s</span><span class="o">-></span><span class="n">v</span><span class="p">.</span><span class="n">FunctionDef</span><span class="p">.</span><span class="n">args</span><span class="p">);</span> <span class="c1">// 8.</span>
4416 </span><span class="hll"> <span class="n">VISIT_SEQ</span><span class="p">(</span><span class="n">st</span><span class="p">,</span> <span class="n">stmt</span><span class="p">,</span> <span class="n">s</span><span class="o">-></span><span class="n">v</span><span class="p">.</span><span class="n">FunctionDef</span><span class="p">.</span><span class="n">body</span><span class="p">);</span> <span class="c1">// 9.</span>
4417 </span> <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">symtable_exit_block</span><span class="p">(</span><span class="n">st</span><span class="p">,</span> <span class="n">s</span><span class="p">))</span>
4418 <span class="n">VISIT_QUIT</span><span class="p">(</span><span class="n">st</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span>
4419 <span class="k">break</span><span class="p">;</span>
4420 <span class="k">case</span> <span class="nl">ClassDef_kind</span><span class="p">:</span> <span class="p">{</span>
4421 <span class="p">...</span>
4422 <span class="p">}</span>
4423 <span class="k">case</span> <span class="nl">Return_kind</span><span class="p">:</span>
4424 <span class="p">...</span>
4425 <span class="k">case</span> <span class="nl">Delete_kind</span><span class="p">:</span>
4426 <span class="p">...</span>
4427 <span class="k">case</span> <span class="nl">Assign_kind</span><span class="p">:</span>
4428 <span class="p">...</span>
4429 <span class="k">case</span> <span class="nl">AnnAssign_kind</span><span class="p">:</span>
4430 <span class="p">...</span>
4431 </pre></div>
4432
4433 <p>Once the resulting symtable has been created, it is sent back to be used for the compiler.</p>
4434 <h4 id="core-compilation-process">Core Compilation Process</h4>
4435 <p>Now that the <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/compile.c#L312"><code>PyAST_CompileObject()</code></a> has a compiler state, a symtable, and a module in the form of the AST, the actual compilation can begin.</p>
4436 <p>The purpose of the core compiler is to:</p>
4437 <ul>
4438 <li>Convert the state, symtable, and AST into a <a href="https://en.wikipedia.org/wiki/Control-flow_graph">Control-Flow-Graph (CFG)</a></li>
4439 <li>Protect the execution stage from runtime exceptions by catching any logic and code errors and raising them here</li>
4440 </ul>
4441 <p>You can call the CPython compiler in Python code by calling the built-in function <code>compile()</code>. It returns a <code>code object</code> instance:</p>
4442 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="nb">compile</span><span class="p">(</span><span class="s1">'b+1'</span><span class="p">,</span> <span class="s1">'test.py'</span><span class="p">,</span> <span class="n">mode</span><span class="o">=</span><span class="s1">'eval'</span><span class="p">)</span>
4443 <span class="go"><code object <module> at 0x10f222780, file "test.py", line 1></span>
4444 </pre></div>
4445
4446 <p>The same as with the <code>symtable()</code> function, a simple expression should have a mode of <code>'eval'</code> and a module, function, or class should have a mode of <code>'exec'</code>.</p>
4447 <p>The compiled code can be found in the <code>co_code</code> property of the code object:</p>
4448 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">co</span><span class="o">.</span><span class="n">co_code</span>
4449 <span class="go">b'e\x00d\x00\x17\x00S\x00'</span>
4450 </pre></div>
4451
4452 <p>There is also a <code>dis</code> module in the standard library, which disassembles the bytecode instructions and can print them on the screen or give you a list of <code>Instruction</code> instances.</p>
4453 <p>If you import <code>dis</code> and give the <code>dis()</code> function the code object’s <code>co_code</code> property it disassembles it and prints the instructions on the REPL:</p>
4454 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">dis</span>
4455 <span class="gp">>>> </span><span class="n">dis</span><span class="o">.</span><span class="n">dis</span><span class="p">(</span><span class="n">co</span><span class="o">.</span><span class="n">co_code</span><span class="p">)</span>
4456 <span class="go"> 0 LOAD_NAME 0 (0)</span>
4457 <span class="go"> 2 LOAD_CONST 0 (0)</span>
4458 <span class="go"> 4 BINARY_ADD</span>
4459 <span class="go"> 6 RETURN_VALUE</span>
4460 </pre></div>
4461
4462 <p><code>LOAD_NAME</code>, <code>LOAD_CONST</code>, <code>BINARY_ADD</code>, and <code>RETURN_VALUE</code> are all bytecode instructions. They’re called bytecode because, in binary form, they were a byte long. However, since Python 3.6 the storage format was changed to a <code>word</code>, so now they’re technically wordcode, not bytecode.</p>
4463 <p>The <a href="https://docs.python.org/3/library/dis.html#python-bytecode-instructions">full list of bytecode instructions</a> is available for each version of Python, and it does change between versions. For example, in Python 3.7, some new bytecode instructions were introduced to speed up execution of specific method calls.</p>
4464 <p>In an earlier section, we explored the <code>instaviz</code> package. This included a visualization of the code object type by running the compiler. It also displays the Bytecode operations inside the code objects.</p>
4465 <p>Execute instaviz again to see the code object and bytecode for a function defined on the REPL:</p>
4466 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">instaviz</span>
4467 <span class="gp">>>> </span><span class="k">def</span> <span class="nf">example</span><span class="p">():</span>
4468 <span class="go"> a = 1</span>
4469 <span class="go"> b = a + 1</span>
4470 <span class="go"> return b</span>
4471 <span class="gp">>>> </span><span class="n">instaviz</span><span class="o">.</span><span class="n">show</span><span class="p">(</span><span class="n">example</span><span class="p">)</span>
4472 </pre></div>
4473
4474 <p>If we now jump into <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/compile.c#L1782"><code>compiler_mod()</code></a>, a function used to switch to different compiler functions depending on the module type. We’ll assume that <code>mod</code> is a <code>Module</code>. The module is compiled into the compiler state and then <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/compile.c#L5971"><code>assemble()</code></a> is run to create a <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Include/code.h#L69"><code>PyCodeObject</code></a>.</p>
4475 <p>The new code object is returned back to <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/compile.c#L312"><code>PyAST_CompileObject()</code></a> and sent on for execution:</p>
4476 <div class="highlight c"><pre><span></span><span class="k">static</span> <span class="n">PyCodeObject</span> <span class="o">*</span>
4477 <span class="nf">compiler_mod</span><span class="p">(</span><span class="k">struct</span> <span class="n">compiler</span> <span class="o">*</span><span class="n">c</span><span class="p">,</span> <span class="n">mod_ty</span> <span class="n">mod</span><span class="p">)</span>
4478 <span class="p">{</span>
4479 <span class="hll"> <span class="n">PyCodeObject</span> <span class="o">*</span><span class="n">co</span><span class="p">;</span>
4480 </span> <span class="kt">int</span> <span class="n">addNone</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
4481 <span class="k">static</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">module</span><span class="p">;</span>
4482 <span class="p">...</span>
4483 <span class="k">switch</span> <span class="p">(</span><span class="n">mod</span><span class="o">-></span><span class="n">kind</span><span class="p">)</span> <span class="p">{</span>
4484 <span class="k">case</span> <span class="nl">Module_kind</span><span class="p">:</span>
4485 <span class="hll"> <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">compiler_body</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">mod</span><span class="o">-></span><span class="n">v</span><span class="p">.</span><span class="n">Module</span><span class="p">.</span><span class="n">body</span><span class="p">))</span> <span class="p">{</span>
4486 </span> <span class="n">compiler_exit_scope</span><span class="p">(</span><span class="n">c</span><span class="p">);</span>
4487 <span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
4488 <span class="p">}</span>
4489 <span class="k">break</span><span class="p">;</span>
4490 <span class="k">case</span> <span class="nl">Interactive_kind</span><span class="p">:</span>
4491 <span class="p">...</span>
4492 <span class="k">case</span> <span class="nl">Expression_kind</span><span class="p">:</span>
4493 <span class="p">...</span>
4494 <span class="k">case</span> <span class="nl">Suite_kind</span><span class="p">:</span>
4495 <span class="p">...</span>
4496 <span class="p">...</span>
4497 <span class="hll"> <span class="n">co</span> <span class="o">=</span> <span class="n">assemble</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">addNone</span><span class="p">);</span>
4498 </span> <span class="n">compiler_exit_scope</span><span class="p">(</span><span class="n">c</span><span class="p">);</span>
4499 <span class="hll"> <span class="k">return</span> <span class="n">co</span><span class="p">;</span>
4500 </span><span class="p">}</span>
4501 </pre></div>
4502
4503 <p>The <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/compile.c#L1743"><code>compiler_body()</code></a> function has some optimization flags and then loops over each statement in the module and visits it, similar to how the <code>symtable</code> functions worked:</p>
4504 <div class="highlight c"><pre><span></span><span class="k">static</span> <span class="kt">int</span>
4505 <span class="nf">compiler_body</span><span class="p">(</span><span class="k">struct</span> <span class="n">compiler</span> <span class="o">*</span><span class="n">c</span><span class="p">,</span> <span class="n">asdl_seq</span> <span class="o">*</span><span class="n">stmts</span><span class="p">)</span>
4506 <span class="p">{</span>
4507 <span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
4508 <span class="n">stmt_ty</span> <span class="n">st</span><span class="p">;</span>
4509 <span class="n">PyObject</span> <span class="o">*</span><span class="n">docstring</span><span class="p">;</span>
4510 <span class="p">...</span>
4511 <span class="hll"> <span class="k">for</span> <span class="p">(;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">asdl_seq_LEN</span><span class="p">(</span><span class="n">stmts</span><span class="p">);</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span>
4512 </span><span class="hll"> <span class="n">VISIT</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">stmt</span><span class="p">,</span> <span class="p">(</span><span class="n">stmt_ty</span><span class="p">)</span><span class="n">asdl_seq_GET</span><span class="p">(</span><span class="n">stmts</span><span class="p">,</span> <span class="n">i</span><span class="p">));</span>
4513 </span> <span class="k">return</span> <span class="mi">1</span><span class="p">;</span>
4514 <span class="p">}</span>
4515 </pre></div>
4516
4517 <p>The statement type is determined through a call to the <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Include/asdl.h#L32"><code>asdl_seq_GET()</code></a> function, which looks at the AST node’s type.</p>
4518 <p>Through some smart macros, <code>VISIT</code> calls a function in <code>Python/compile.c</code> for each statement type:</p>
4519 <div class="highlight c"><pre><span></span><span class="cp">#define VISIT(C, TYPE, V) {\</span>
4520 <span class="cp"> if (!compiler_visit_ ## TYPE((C), (V))) \</span>
4521 <span class="cp"> return 0; \</span>
4522 <span class="cp">}</span>
4523 </pre></div>
4524
4525 <p>For a <code>stmt</code> (the category for a statement) the compiler will then drop into <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/compile.c#L3310"><code>compiler_visit_stmt()</code></a> and switch through all of the potential statement types found in <code>Parser/Python.asdl</code>:</p>
4526 <div class="highlight c"><pre><span></span><span class="k">static</span> <span class="kt">int</span>
4527 <span class="nf">compiler_visit_stmt</span><span class="p">(</span><span class="k">struct</span> <span class="n">compiler</span> <span class="o">*</span><span class="n">c</span><span class="p">,</span> <span class="n">stmt_ty</span> <span class="n">s</span><span class="p">)</span>
4528 <span class="p">{</span>
4529 <span class="n">Py_ssize_t</span> <span class="n">i</span><span class="p">,</span> <span class="n">n</span><span class="p">;</span>
4530
4531 <span class="cm">/* Always assign a lineno to the next instruction for a stmt. */</span>
4532 <span class="n">c</span><span class="o">-></span><span class="n">u</span><span class="o">-></span><span class="n">u_lineno</span> <span class="o">=</span> <span class="n">s</span><span class="o">-></span><span class="n">lineno</span><span class="p">;</span>
4533 <span class="n">c</span><span class="o">-></span><span class="n">u</span><span class="o">-></span><span class="n">u_col_offset</span> <span class="o">=</span> <span class="n">s</span><span class="o">-></span><span class="n">col_offset</span><span class="p">;</span>
4534 <span class="n">c</span><span class="o">-></span><span class="n">u</span><span class="o">-></span><span class="n">u_lineno_set</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
4535
4536 <span class="k">switch</span> <span class="p">(</span><span class="n">s</span><span class="o">-></span><span class="n">kind</span><span class="p">)</span> <span class="p">{</span>
4537 <span class="k">case</span> <span class="nl">FunctionDef_kind</span><span class="p">:</span>
4538 <span class="k">return</span> <span class="n">compiler_function</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">s</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span>
4539 <span class="k">case</span> <span class="nl">ClassDef_kind</span><span class="p">:</span>
4540 <span class="k">return</span> <span class="n">compiler_class</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">s</span><span class="p">);</span>
4541 <span class="p">...</span>
4542 <span class="k">case</span> <span class="nl">For_kind</span><span class="p">:</span>
4543 <span class="k">return</span> <span class="n">compiler_for</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">s</span><span class="p">);</span>
4544 <span class="p">...</span>
4545 <span class="p">}</span>
4546
4547 <span class="k">return</span> <span class="mi">1</span><span class="p">;</span>
4548 <span class="p">}</span>
4549 </pre></div>
4550
4551 <p>As an example, let’s focus on the <code>For</code> statement, in Python is the:</p>
4552 <div class="highlight python"><pre><span></span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">iterable</span><span class="p">:</span>
4553 <span class="c1"># block</span>
4554 <span class="k">else</span><span class="p">:</span> <span class="c1"># optional if iterable is False</span>
4555 <span class="c1"># block</span>
4556 </pre></div>
4557
4558 <p>If the statement is a <code>For</code> type, it calls <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/compile.c#L2651"><code>compiler_for()</code></a>. There is an equivalent <code>compiler_*()</code> function for all of the statement and expression types. The more straightforward types create the bytecode instructions inline, some of the more complex statement types call other functions.</p>
4559 <p>Many of the statements can have sub-statements. A <code>for</code> loop has a body, but you can also have complex expressions in the assignment and the iterator.</p>
4560 <p>The compiler’s <code>compiler_</code> statements sends blocks to the compiler state. These blocks contain instructions, the instruction data structure in <code>Python/compile.c</code> has the opcode, any arguments, and the target block (if this is a jump instruction), it also contains the line number.</p>
4561 <p>For jump statements, they can either be absolute or relative jump statements. Jump statements are used to “jump” from one operation to another. Absolute jump statements specify the exact operation number in the compiled code object, whereas relative jump statements specify the jump target relative to another operation:</p>
4562 <div class="highlight c"><pre><span></span><span class="k">struct</span> <span class="n">instr</span> <span class="p">{</span>
4563 <span class="kt">unsigned</span> <span class="nl">i_jabs</span> <span class="p">:</span> <span class="mi">1</span><span class="p">;</span>
4564 <span class="kt">unsigned</span> <span class="nl">i_jrel</span> <span class="p">:</span> <span class="mi">1</span><span class="p">;</span>
4565 <span class="kt">unsigned</span> <span class="kt">char</span> <span class="n">i_opcode</span><span class="p">;</span>
4566 <span class="kt">int</span> <span class="n">i_oparg</span><span class="p">;</span>
4567 <span class="k">struct</span> <span class="n">basicblock_</span> <span class="o">*</span><span class="n">i_target</span><span class="p">;</span> <span class="cm">/* target block (if jump instruction) */</span>
4568 <span class="kt">int</span> <span class="n">i_lineno</span><span class="p">;</span>
4569 <span class="p">};</span>
4570 </pre></div>
4571
4572 <p>So a frame block (of type <code>basicblock</code>), contains the following fields:</p>
4573 <ul>
4574 <li>A <code>b_list</code> pointer, the link to a list of blocks for the compiler state</li>
4575 <li>A list of instructions <code>b_instr</code>, with both the allocated list size <code>b_ialloc</code>, and the number used <code>b_iused</code></li>
4576 <li>The next block after this one <code>b_next</code></li>
4577 <li>Whether the block has been “seen” by the assembler when traversing depth-first</li>
4578 <li>If this block has a <code>RETURN_VALUE</code> opcode (<code>b_return</code>)</li>
4579 <li>The depth of the stack when this block was entered (<code>b_startdepth</code>)</li>
4580 <li>The instruction offset for the assembler</li>
4581 </ul>
4582 <div class="highlight c"><pre><span></span><span class="k">typedef</span> <span class="k">struct</span> <span class="n">basicblock_</span> <span class="p">{</span>
4583 <span class="cm">/* Each basicblock in a compilation unit is linked via b_list in the</span>
4584 <span class="cm"> reverse order that the block are allocated. b_list points to the next</span>
4585 <span class="cm"> block, not to be confused with b_next, which is next by control flow. */</span>
4586 <span class="k">struct</span> <span class="n">basicblock_</span> <span class="o">*</span><span class="n">b_list</span><span class="p">;</span>
4587 <span class="cm">/* number of instructions used */</span>
4588 <span class="kt">int</span> <span class="n">b_iused</span><span class="p">;</span>
4589 <span class="cm">/* length of instruction array (b_instr) */</span>
4590 <span class="kt">int</span> <span class="n">b_ialloc</span><span class="p">;</span>
4591 <span class="cm">/* pointer to an array of instructions, initially NULL */</span>
4592 <span class="k">struct</span> <span class="n">instr</span> <span class="o">*</span><span class="n">b_instr</span><span class="p">;</span>
4593 <span class="cm">/* If b_next is non-NULL, it is a pointer to the next</span>
4594 <span class="cm"> block reached by normal control flow. */</span>
4595 <span class="k">struct</span> <span class="n">basicblock_</span> <span class="o">*</span><span class="n">b_next</span><span class="p">;</span>
4596 <span class="cm">/* b_seen is used to perform a DFS of basicblocks. */</span>
4597 <span class="kt">unsigned</span> <span class="nl">b_seen</span> <span class="p">:</span> <span class="mi">1</span><span class="p">;</span>
4598 <span class="cm">/* b_return is true if a RETURN_VALUE opcode is inserted. */</span>
4599 <span class="kt">unsigned</span> <span class="nl">b_return</span> <span class="p">:</span> <span class="mi">1</span><span class="p">;</span>
4600 <span class="cm">/* depth of stack upon entry of block, computed by stackdepth() */</span>
4601 <span class="kt">int</span> <span class="n">b_startdepth</span><span class="p">;</span>
4602 <span class="cm">/* instruction offset for block, computed by assemble_jump_offsets() */</span>
4603 <span class="kt">int</span> <span class="n">b_offset</span><span class="p">;</span>
4604 <span class="p">}</span> <span class="n">basicblock</span><span class="p">;</span>
4605 </pre></div>
4606
4607 <p>The <code>For</code> statement is somewhere in the middle in terms of complexity. There are 15 steps in the compilation of a <code>For</code> statement with the <code>for <target> in <iterator>:</code> syntax:</p>
4608 <ol>
4609 <li>Create a new code block called <code>start</code>, this allocates memory and creates a <code>basicblock</code> pointer</li>
4610 <li>Create a new code block called <code>cleanup</code></li>
4611 <li>Create a new code block called <code>end</code></li>
4612 <li>Push a frame block of type <code>FOR_LOOP</code> to the stack with <code>start</code> as the entry block and <code>end</code> as the exit block</li>
4613 <li>Visit the iterator expression, which adds any operations for the iterator</li>
4614 <li>Add the <code>GET_ITER</code> operation to the compiler state</li>
4615 <li>Switch to the <code>start</code> block</li>
4616 <li>Call <code>ADDOP_JREL</code> which calls <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/compile.c#L1413"><code>compiler_addop_j()</code></a> to add the <code>FOR_ITER</code> operation with an argument of the <code>cleanup</code> block</li>
4617 <li>Visit the <code>target</code> and add any special code, like tuple unpacking, to the <code>start</code> block</li>
4618 <li>Visit each statement in the body of the for loop</li>
4619 <li>Call <code>ADDOP_JABS</code> which calls <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/compile.c#L1413"><code>compiler_addop_j()</code></a> to add the <code>JUMP_ABSOLUTE</code> operation which indicates after the body is executed, jumps back to the start of the loop</li>
4620 <li>Move to the <code>cleanup</code> block</li>
4621 <li>Pop the <code>FOR_LOOP</code> frame block off the stack</li>
4622 <li>Visit the statements inside the <code>else</code> section of the for loop</li>
4623 <li>Use the <code>end</code> block</li>
4624 </ol>
4625 <p>Referring back to the <code>basicblock</code> structure. You can see how in the compilation of the for statement, the various blocks are created and pushed into the compiler’s frame block and stack:</p>
4626 <div class="highlight c"><pre><span></span><span class="k">static</span> <span class="kt">int</span>
4627 <span class="nf">compiler_for</span><span class="p">(</span><span class="k">struct</span> <span class="n">compiler</span> <span class="o">*</span><span class="n">c</span><span class="p">,</span> <span class="n">stmt_ty</span> <span class="n">s</span><span class="p">)</span>
4628 <span class="p">{</span>
4629 <span class="n">basicblock</span> <span class="o">*</span><span class="n">start</span><span class="p">,</span> <span class="o">*</span><span class="n">cleanup</span><span class="p">,</span> <span class="o">*</span><span class="n">end</span><span class="p">;</span>
4630
4631 <span class="hll"> <span class="n">start</span> <span class="o">=</span> <span class="n">compiler_new_block</span><span class="p">(</span><span class="n">c</span><span class="p">);</span> <span class="c1">// 1.</span>
4632 </span><span class="hll"> <span class="n">cleanup</span> <span class="o">=</span> <span class="n">compiler_new_block</span><span class="p">(</span><span class="n">c</span><span class="p">);</span> <span class="c1">// 2.</span>
4633 </span><span class="hll"> <span class="n">end</span> <span class="o">=</span> <span class="n">compiler_new_block</span><span class="p">(</span><span class="n">c</span><span class="p">);</span> <span class="c1">// 3.</span>
4634 </span> <span class="k">if</span> <span class="p">(</span><span class="n">start</span> <span class="o">==</span> <span class="nb">NULL</span> <span class="o">||</span> <span class="n">end</span> <span class="o">==</span> <span class="nb">NULL</span> <span class="o">||</span> <span class="n">cleanup</span> <span class="o">==</span> <span class="nb">NULL</span><span class="p">)</span>
4635 <span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
4636
4637 <span class="hll"> <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">compiler_push_fblock</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">FOR_LOOP</span><span class="p">,</span> <span class="n">start</span><span class="p">,</span> <span class="n">end</span><span class="p">))</span> <span class="c1">// 4.</span>
4638 </span> <span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
4639
4640 <span class="hll"> <span class="n">VISIT</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">expr</span><span class="p">,</span> <span class="n">s</span><span class="o">-></span><span class="n">v</span><span class="p">.</span><span class="n">For</span><span class="p">.</span><span class="n">iter</span><span class="p">);</span> <span class="c1">// 5.</span>
4641 </span><span class="hll"> <span class="n">ADDOP</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">GET_ITER</span><span class="p">);</span> <span class="c1">// 6.</span>
4642 </span><span class="hll"> <span class="n">compiler_use_next_block</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">start</span><span class="p">);</span> <span class="c1">// 7.</span>
4643 </span><span class="hll"> <span class="n">ADDOP_JREL</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">FOR_ITER</span><span class="p">,</span> <span class="n">cleanup</span><span class="p">);</span> <span class="c1">// 8.</span>
4644 </span><span class="hll"> <span class="n">VISIT</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">expr</span><span class="p">,</span> <span class="n">s</span><span class="o">-></span><span class="n">v</span><span class="p">.</span><span class="n">For</span><span class="p">.</span><span class="n">target</span><span class="p">);</span> <span class="c1">// 9.</span>
4645 </span><span class="hll"> <span class="n">VISIT_SEQ</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">stmt</span><span class="p">,</span> <span class="n">s</span><span class="o">-></span><span class="n">v</span><span class="p">.</span><span class="n">For</span><span class="p">.</span><span class="n">body</span><span class="p">);</span> <span class="c1">// 10.</span>
4646 </span><span class="hll"> <span class="n">ADDOP_JABS</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">JUMP_ABSOLUTE</span><span class="p">,</span> <span class="n">start</span><span class="p">);</span> <span class="c1">// 11.</span>
4647 </span><span class="hll"> <span class="n">compiler_use_next_block</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">cleanup</span><span class="p">);</span> <span class="c1">// 12.</span>
4648 </span>
4649 <span class="hll"> <span class="n">compiler_pop_fblock</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">FOR_LOOP</span><span class="p">,</span> <span class="n">start</span><span class="p">);</span> <span class="c1">// 13.</span>
4650 </span>
4651 <span class="hll"> <span class="n">VISIT_SEQ</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">stmt</span><span class="p">,</span> <span class="n">s</span><span class="o">-></span><span class="n">v</span><span class="p">.</span><span class="n">For</span><span class="p">.</span><span class="n">orelse</span><span class="p">);</span> <span class="c1">// 14.</span>
4652 </span><span class="hll"> <span class="n">compiler_use_next_block</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">end</span><span class="p">);</span> <span class="c1">// 15.</span>
4653 </span> <span class="k">return</span> <span class="mi">1</span><span class="p">;</span>
4654 <span class="p">}</span>
4655 </pre></div>
4656
4657 <p>Depending on the type of operation, there are different arguments required. For example, we used <code>ADDOP_JABS</code> and <code>ADDOP_JREL</code> here, which refer to “<strong>ADD</strong> <strong>O</strong>peration with <strong>J</strong>ump to a <strong>REL</strong>ative position” and “<strong>ADD</strong> <strong>O</strong>peration with <strong>J</strong>ump to an <strong>ABS</strong>olute position”. This is referring to the <code>APPOP_JREL</code> and <code>ADDOP_JABS</code> macros which call <code>compiler_addop_j(struct compiler *c, int opcode, basicblock *b, int absolute)</code> and set the <code>absolute</code> argument to 0 and 1 respectively.</p>
4658 <p>There are some other macros, like <code>ADDOP_I</code> calls <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/compile.c#L1383"><code>compiler_addop_i()</code></a> which add an operation with an integer argument, or <code>ADDOP_O</code> calls <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/compile.c#L1345"><code>compiler_addop_o()</code></a> which adds an operation with a <code>PyObject</code> argument. </p>
4659 <p>Once these stages have completed, the compiler has a list of frame blocks, each containing a list of instructions and a pointer to the next block.</p>
4660 <h4 id="assembly">Assembly</h4>
4661 <p>With the compiler state, the assembler performs a “depth-first-search” of the blocks and merge the instructions into a single bytecode sequence. The assembler state is declared in <code>Python/compile.c</code>:</p>
4662 <div class="highlight c"><pre><span></span><span class="k">struct</span> <span class="n">assembler</span> <span class="p">{</span>
4663 <span class="n">PyObject</span> <span class="o">*</span><span class="n">a_bytecode</span><span class="p">;</span> <span class="cm">/* string containing bytecode */</span>
4664 <span class="kt">int</span> <span class="n">a_offset</span><span class="p">;</span> <span class="cm">/* offset into bytecode */</span>
4665 <span class="kt">int</span> <span class="n">a_nblocks</span><span class="p">;</span> <span class="cm">/* number of reachable blocks */</span>
4666 <span class="n">basicblock</span> <span class="o">**</span><span class="n">a_postorder</span><span class="p">;</span> <span class="cm">/* list of blocks in dfs postorder */</span>
4667 <span class="n">PyObject</span> <span class="o">*</span><span class="n">a_lnotab</span><span class="p">;</span> <span class="cm">/* string containing lnotab */</span>
4668 <span class="kt">int</span> <span class="n">a_lnotab_off</span><span class="p">;</span> <span class="cm">/* offset into lnotab */</span>
4669 <span class="kt">int</span> <span class="n">a_lineno</span><span class="p">;</span> <span class="cm">/* last lineno of emitted instruction */</span>
4670 <span class="kt">int</span> <span class="n">a_lineno_off</span><span class="p">;</span> <span class="cm">/* bytecode offset of last lineno */</span>
4671 <span class="p">};</span>
4672 </pre></div>
4673
4674 <p>The <code>assemble()</code> function has a few tasks:</p>
4675 <ul>
4676 <li>Calculate the number of blocks for memory allocation</li>
4677 <li>Ensure that every block that falls off the end returns <code>None</code>, this is why every function returns <code>None</code>, whether or not a <code>return</code> statement exists</li>
4678 <li>Resolve any jump statements offsets that were marked as relative</li>
4679 <li>Call <code>dfs()</code> to perform a depth-first-search of the blocks</li>
4680 <li>Emit all the instructions to the compiler</li>
4681 <li>Call <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/compile.c#L5854"><code>makecode()</code></a> with the compiler state to generate the <code>PyCodeObject</code></li>
4682 </ul>
4683 <div class="highlight c"><pre><span></span><span class="k">static</span> <span class="n">PyCodeObject</span> <span class="o">*</span>
4684 <span class="nf">assemble</span><span class="p">(</span><span class="k">struct</span> <span class="n">compiler</span> <span class="o">*</span><span class="n">c</span><span class="p">,</span> <span class="kt">int</span> <span class="n">addNone</span><span class="p">)</span>
4685 <span class="p">{</span>
4686 <span class="n">basicblock</span> <span class="o">*</span><span class="n">b</span><span class="p">,</span> <span class="o">*</span><span class="n">entryblock</span><span class="p">;</span>
4687 <span class="k">struct</span> <span class="n">assembler</span> <span class="n">a</span><span class="p">;</span>
4688 <span class="kt">int</span> <span class="n">i</span><span class="p">,</span> <span class="n">j</span><span class="p">,</span> <span class="n">nblocks</span><span class="p">;</span>
4689 <span class="n">PyCodeObject</span> <span class="o">*</span><span class="n">co</span> <span class="o">=</span> <span class="nb">NULL</span><span class="p">;</span>
4690
4691 <span class="cm">/* Make sure every block that falls off the end returns None.</span>
4692 <span class="cm"> XXX NEXT_BLOCK() isn't quite right, because if the last</span>
4693 <span class="cm"> block ends with a jump or return b_next shouldn't set.</span>
4694 <span class="cm"> */</span>
4695 <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">c</span><span class="o">-></span><span class="n">u</span><span class="o">-></span><span class="n">u_curblock</span><span class="o">-></span><span class="n">b_return</span><span class="p">)</span> <span class="p">{</span>
4696 <span class="n">NEXT_BLOCK</span><span class="p">(</span><span class="n">c</span><span class="p">);</span>
4697 <span class="hll"> <span class="k">if</span> <span class="p">(</span><span class="n">addNone</span><span class="p">)</span>
4698 </span><span class="hll"> <span class="n">ADDOP_LOAD_CONST</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">Py_None</span><span class="p">);</span>
4699 </span><span class="hll"> <span class="n">ADDOP</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">RETURN_VALUE</span><span class="p">);</span>
4700 </span> <span class="p">}</span>
4701 <span class="p">...</span>
4702 <span class="hll"> <span class="n">dfs</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">entryblock</span><span class="p">,</span> <span class="o">&</span><span class="n">a</span><span class="p">,</span> <span class="n">nblocks</span><span class="p">);</span>
4703 </span>
4704 <span class="cm">/* Can't modify the bytecode after computing jump offsets. */</span>
4705 <span class="hll"> <span class="n">assemble_jump_offsets</span><span class="p">(</span><span class="o">&</span><span class="n">a</span><span class="p">,</span> <span class="n">c</span><span class="p">);</span>
4706 </span>
4707 <span class="cm">/* Emit code in reverse postorder from dfs. */</span>
4708 <span class="hll"> <span class="k">for</span> <span class="p">(</span><span class="n">i</span> <span class="o">=</span> <span class="n">a</span><span class="p">.</span><span class="n">a_nblocks</span> <span class="o">-</span> <span class="mi">1</span><span class="p">;</span> <span class="n">i</span> <span class="o">>=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span><span class="o">--</span><span class="p">)</span> <span class="p">{</span>
4709 </span><span class="hll"> <span class="n">b</span> <span class="o">=</span> <span class="n">a</span><span class="p">.</span><span class="n">a_postorder</span><span class="p">[</span><span class="n">i</span><span class="p">];</span>
4710 </span><span class="hll"> <span class="k">for</span> <span class="p">(</span><span class="n">j</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">j</span> <span class="o"><</span> <span class="n">b</span><span class="o">-></span><span class="n">b_iused</span><span class="p">;</span> <span class="n">j</span><span class="o">++</span><span class="p">)</span>
4711 </span><span class="hll"> <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">assemble_emit</span><span class="p">(</span><span class="o">&</span><span class="n">a</span><span class="p">,</span> <span class="o">&</span><span class="n">b</span><span class="o">-></span><span class="n">b_instr</span><span class="p">[</span><span class="n">j</span><span class="p">]))</span>
4712 </span><span class="hll"> <span class="k">goto</span> <span class="n">error</span><span class="p">;</span>
4713 </span><span class="hll"> <span class="p">}</span>
4714 </span> <span class="p">...</span>
4715
4716 <span class="n">co</span> <span class="o">=</span> <span class="n">makecode</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="o">&</span><span class="n">a</span><span class="p">);</span>
4717 <span class="hll"> <span class="nl">error</span><span class="p">:</span>
4718 </span> <span class="n">assemble_free</span><span class="p">(</span><span class="o">&</span><span class="n">a</span><span class="p">);</span>
4719 <span class="k">return</span> <span class="n">co</span><span class="p">;</span>
4720 <span class="p">}</span>
4721 </pre></div>
4722
4723 <p>The depth-first-search is performed by the <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/compile.c#L5397"><code>dfs()</code></a> function in <code>Python/compile.c</code>, which follows the the <code>b_next</code> pointers in each of the blocks, marks them as seen by toggling <code>b_seen</code> and then adds them to the assemblers <code>**a_postorder</code> list in reverse order.</p>
4724 <p>The function loops back over the assembler’s post-order list and for each block, if it has a jump operation, recursively call <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/compile.c#L5397"><code>dfs()</code></a> for that jump:</p>
4725 <div class="highlight c"><pre><span></span><span class="k">static</span> <span class="kt">void</span>
4726 <span class="nf">dfs</span><span class="p">(</span><span class="k">struct</span> <span class="n">compiler</span> <span class="o">*</span><span class="n">c</span><span class="p">,</span> <span class="n">basicblock</span> <span class="o">*</span><span class="n">b</span><span class="p">,</span> <span class="k">struct</span> <span class="n">assembler</span> <span class="o">*</span><span class="n">a</span><span class="p">,</span> <span class="kt">int</span> <span class="n">end</span><span class="p">)</span>
4727 <span class="p">{</span>
4728 <span class="kt">int</span> <span class="n">i</span><span class="p">,</span> <span class="n">j</span><span class="p">;</span>
4729
4730 <span class="cm">/* Get rid of recursion for normal control flow.</span>
4731 <span class="cm"> Since the number of blocks is limited, unused space in a_postorder</span>
4732 <span class="cm"> (from a_nblocks to end) can be used as a stack for still not ordered</span>
4733 <span class="cm"> blocks. */</span>
4734 <span class="k">for</span> <span class="p">(</span><span class="n">j</span> <span class="o">=</span> <span class="n">end</span><span class="p">;</span> <span class="n">b</span> <span class="o">&&</span> <span class="o">!</span><span class="n">b</span><span class="o">-></span><span class="n">b_seen</span><span class="p">;</span> <span class="n">b</span> <span class="o">=</span> <span class="n">b</span><span class="o">-></span><span class="n">b_next</span><span class="p">)</span> <span class="p">{</span>
4735 <span class="n">b</span><span class="o">-></span><span class="n">b_seen</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
4736 <span class="n">assert</span><span class="p">(</span><span class="n">a</span><span class="o">-></span><span class="n">a_nblocks</span> <span class="o"><</span> <span class="n">j</span><span class="p">);</span>
4737 <span class="n">a</span><span class="o">-></span><span class="n">a_postorder</span><span class="p">[</span><span class="o">--</span><span class="n">j</span><span class="p">]</span> <span class="o">=</span> <span class="n">b</span><span class="p">;</span>
4738 <span class="p">}</span>
4739 <span class="k">while</span> <span class="p">(</span><span class="n">j</span> <span class="o"><</span> <span class="n">end</span><span class="p">)</span> <span class="p">{</span>
4740 <span class="n">b</span> <span class="o">=</span> <span class="n">a</span><span class="o">-></span><span class="n">a_postorder</span><span class="p">[</span><span class="n">j</span><span class="o">++</span><span class="p">];</span>
4741 <span class="k">for</span> <span class="p">(</span><span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">b</span><span class="o">-></span><span class="n">b_iused</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
4742 <span class="k">struct</span> <span class="n">instr</span> <span class="o">*</span><span class="n">instr</span> <span class="o">=</span> <span class="o">&</span><span class="n">b</span><span class="o">-></span><span class="n">b_instr</span><span class="p">[</span><span class="n">i</span><span class="p">];</span>
4743 <span class="k">if</span> <span class="p">(</span><span class="n">instr</span><span class="o">-></span><span class="n">i_jrel</span> <span class="o">||</span> <span class="n">instr</span><span class="o">-></span><span class="n">i_jabs</span><span class="p">)</span>
4744 <span class="n">dfs</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">instr</span><span class="o">-></span><span class="n">i_target</span><span class="p">,</span> <span class="n">a</span><span class="p">,</span> <span class="n">j</span><span class="p">);</span>
4745 <span class="p">}</span>
4746 <span class="n">assert</span><span class="p">(</span><span class="n">a</span><span class="o">-></span><span class="n">a_nblocks</span> <span class="o"><</span> <span class="n">j</span><span class="p">);</span>
4747 <span class="n">a</span><span class="o">-></span><span class="n">a_postorder</span><span class="p">[</span><span class="n">a</span><span class="o">-></span><span class="n">a_nblocks</span><span class="o">++</span><span class="p">]</span> <span class="o">=</span> <span class="n">b</span><span class="p">;</span>
4748 <span class="p">}</span>
4749 <span class="p">}</span>
4750 </pre></div>
4751
4752 <h4 id="creating-a-code-object">Creating a Code Object</h4>
4753 <p>The task of <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/compile.c#L5854"><code>makecode()</code></a> is to go through the compiler state, some of the assembler’s properties and to put these into a <code>PyCodeObject</code> by calling <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Objects/codeobject.c#L246"><code>PyCode_New()</code></a>:</p>
4754 <p><a href="https://files.realpython.com/media/codeobject.9c054576627c.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/codeobject.9c054576627c.png" width="201" height="550" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/codeobject.9c054576627c.png&w=50&sig=9d1c4ff65adb0d6d578b775ca93a88843a3742c1 50w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/codeobject.9c054576627c.png&w=100&sig=076765eedb49d9e4e944629435b5a0fc10942c4c 100w, https://files.realpython.com/media/codeobject.9c054576627c.png 201w" sizes="75vw" alt="PyCodeObject structure"/></a></p>
4755 <p>The variable names, constants are put as properties to the code object:</p>
4756 <div class="highlight c"><pre><span></span><span class="k">static</span> <span class="n">PyCodeObject</span> <span class="o">*</span>
4757 <span class="nf">makecode</span><span class="p">(</span><span class="k">struct</span> <span class="n">compiler</span> <span class="o">*</span><span class="n">c</span><span class="p">,</span> <span class="k">struct</span> <span class="n">assembler</span> <span class="o">*</span><span class="n">a</span><span class="p">)</span>
4758 <span class="p">{</span>
4759 <span class="p">...</span>
4760
4761 <span class="n">consts</span> <span class="o">=</span> <span class="n">consts_dict_keys_inorder</span><span class="p">(</span><span class="n">c</span><span class="o">-></span><span class="n">u</span><span class="o">-></span><span class="n">u_consts</span><span class="p">);</span>
4762 <span class="n">names</span> <span class="o">=</span> <span class="n">dict_keys_inorder</span><span class="p">(</span><span class="n">c</span><span class="o">-></span><span class="n">u</span><span class="o">-></span><span class="n">u_names</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span>
4763 <span class="n">varnames</span> <span class="o">=</span> <span class="n">dict_keys_inorder</span><span class="p">(</span><span class="n">c</span><span class="o">-></span><span class="n">u</span><span class="o">-></span><span class="n">u_varnames</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span>
4764 <span class="p">...</span>
4765 <span class="n">cellvars</span> <span class="o">=</span> <span class="n">dict_keys_inorder</span><span class="p">(</span><span class="n">c</span><span class="o">-></span><span class="n">u</span><span class="o">-></span><span class="n">u_cellvars</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span>
4766 <span class="p">...</span>
4767 <span class="n">freevars</span> <span class="o">=</span> <span class="n">dict_keys_inorder</span><span class="p">(</span><span class="n">c</span><span class="o">-></span><span class="n">u</span><span class="o">-></span><span class="n">u_freevars</span><span class="p">,</span> <span class="n">PyTuple_GET_SIZE</span><span class="p">(</span><span class="n">cellvars</span><span class="p">));</span>
4768 <span class="p">...</span>
4769 <span class="n">flags</span> <span class="o">=</span> <span class="n">compute_code_flags</span><span class="p">(</span><span class="n">c</span><span class="p">);</span>
4770 <span class="k">if</span> <span class="p">(</span><span class="n">flags</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span>
4771 <span class="k">goto</span> <span class="n">error</span><span class="p">;</span>
4772
4773 <span class="n">bytecode</span> <span class="o">=</span> <span class="n">PyCode_Optimize</span><span class="p">(</span><span class="n">a</span><span class="o">-></span><span class="n">a_bytecode</span><span class="p">,</span> <span class="n">consts</span><span class="p">,</span> <span class="n">names</span><span class="p">,</span> <span class="n">a</span><span class="o">-></span><span class="n">a_lnotab</span><span class="p">);</span>
4774 <span class="p">...</span>
4775 <span class="n">co</span> <span class="o">=</span> <span class="n">PyCode_NewWithPosOnlyArgs</span><span class="p">(</span><span class="n">posonlyargcount</span><span class="o">+</span><span class="n">posorkeywordargcount</span><span class="p">,</span>
4776 <span class="n">posonlyargcount</span><span class="p">,</span> <span class="n">kwonlyargcount</span><span class="p">,</span> <span class="n">nlocals_int</span><span class="p">,</span>
4777 <span class="n">maxdepth</span><span class="p">,</span> <span class="n">flags</span><span class="p">,</span> <span class="n">bytecode</span><span class="p">,</span> <span class="n">consts</span><span class="p">,</span> <span class="n">names</span><span class="p">,</span>
4778 <span class="n">varnames</span><span class="p">,</span> <span class="n">freevars</span><span class="p">,</span> <span class="n">cellvars</span><span class="p">,</span> <span class="n">c</span><span class="o">-></span><span class="n">c_filename</span><span class="p">,</span>
4779 <span class="n">c</span><span class="o">-></span><span class="n">u</span><span class="o">-></span><span class="n">u_name</span><span class="p">,</span> <span class="n">c</span><span class="o">-></span><span class="n">u</span><span class="o">-></span><span class="n">u_firstlineno</span><span class="p">,</span> <span class="n">a</span><span class="o">-></span><span class="n">a_lnotab</span><span class="p">);</span>
4780 <span class="p">...</span>
4781 <span class="k">return</span> <span class="n">co</span><span class="p">;</span>
4782 <span class="p">}</span>
4783 </pre></div>
4784
4785 <p>You may also notice that the bytecode is sent to <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/peephole.c#L230"><code>PyCode_Optimize()</code></a> before it is sent to <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Objects/codeobject.c#L106"><code>PyCode_NewWithPosOnlyArgs()</code></a>. This function is part of the bytecode optimization process in <code>Python/peephole.c</code>.</p>
4786 <p>The peephole optimizer goes through the bytecode instructions and in certain scenarios, replace them with other instructions. For example, there is an optimizer called “constant unfolding”, so if you put the following statement into your script:</p>
4787 <div class="highlight python"><pre><span></span><span class="n">a</span> <span class="o">=</span> <span class="mi">1</span> <span class="o">+</span> <span class="mi">5</span>
4788 </pre></div>
4789
4790 <p>It optimizes that to:</p>
4791 <div class="highlight python"><pre><span></span><span class="n">a</span> <span class="o">=</span> <span class="mi">6</span>
4792 </pre></div>
4793
4794 <p>Because 1 and 5 are constant values, so the result should always be the same.</p>
4795 <h4 id="conclusion_2">Conclusion</h4>
4796 <p>We can pull together all of these stages with the instaviz module:</p>
4797 <div class="highlight python"><pre><span></span><span class="kn">import</span> <span class="nn">instaviz</span>
4798
4799 <span class="k">def</span> <span class="nf">foo</span><span class="p">():</span>
4800 <span class="n">a</span> <span class="o">=</span> <span class="mi">2</span><span class="o">**</span><span class="mi">4</span>
4801 <span class="n">b</span> <span class="o">=</span> <span class="mi">1</span> <span class="o">+</span> <span class="mi">5</span>
4802 <span class="n">c</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">6</span><span class="p">]</span>
4803 <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">c</span><span class="p">:</span>
4804 <span class="nb">print</span><span class="p">(</span><span class="n">i</span><span class="p">)</span>
4805 <span class="k">else</span><span class="p">:</span>
4806 <span class="nb">print</span><span class="p">(</span><span class="n">a</span><span class="p">)</span>
4807 <span class="k">return</span> <span class="n">c</span>
4808
4809
4810 <span class="n">instaviz</span><span class="o">.</span><span class="n">show</span><span class="p">(</span><span class="n">foo</span><span class="p">)</span>
4811 </pre></div>
4812
4813 <p>Will produce an AST graph:</p>
4814 <p><a href="https://files.realpython.com/media/Screen_Shot_2019-03-20_at_3.18.32_pm.4d9a0ea827ff.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/Screen_Shot_2019-03-20_at_3.18.32_pm.4d9a0ea827ff.png" width="2788" height="1554" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screen_Shot_2019-03-20_at_3.18.32_pm.4d9a0ea827ff.png&w=697&sig=9106a1bc23c5cc07f2ef159968c1ba2155a6562d 697w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screen_Shot_2019-03-20_at_3.18.32_pm.4d9a0ea827ff.png&w=1394&sig=244bebb56c46d0b1eef97b8332afb6bfaa5e3648 1394w, https://files.realpython.com/media/Screen_Shot_2019-03-20_at_3.18.32_pm.4d9a0ea827ff.png 2788w" sizes="75vw" alt="Instaviz screenshot 6"/></a></p>
4815 <p>With bytecode instructions in sequence:</p>
4816 <p><a href="https://files.realpython.com/media/Screen_Shot_2019-03-20_at_3.17.54_pm.6ea8ea532015.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/Screen_Shot_2019-03-20_at_3.17.54_pm.6ea8ea532015.png" width="2536" height="1592" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screen_Shot_2019-03-20_at_3.17.54_pm.6ea8ea532015.png&w=634&sig=f499889305c84679bdf07256f978975bbbc98c03 634w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screen_Shot_2019-03-20_at_3.17.54_pm.6ea8ea532015.png&w=1268&sig=8d86595bce0f05c8dc45c3db3c0bff9b2d9cb0a9 1268w, https://files.realpython.com/media/Screen_Shot_2019-03-20_at_3.17.54_pm.6ea8ea532015.png 2536w" sizes="75vw" alt="Instaviz screenshot 7"/></a></p>
4817 <p>Also, the code object with the variable names, constants, and binary <code>co_code</code>:</p>
4818 <p><a href="https://files.realpython.com/media/Screen_Shot_2019-03-20_at_3.17.41_pm.231a0678f142.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/Screen_Shot_2019-03-20_at_3.17.41_pm.231a0678f142.png" width="2098" height="940" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screen_Shot_2019-03-20_at_3.17.41_pm.231a0678f142.png&w=524&sig=6daa3f3b9841eabbbf87d73886b81d346cdb33b3 524w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screen_Shot_2019-03-20_at_3.17.41_pm.231a0678f142.png&w=1049&sig=7b074cc948021547f3da60e6bf0be3747585c6c8 1049w, https://files.realpython.com/media/Screen_Shot_2019-03-20_at_3.17.41_pm.231a0678f142.png 2098w" sizes="75vw" alt="Instaviz screenshot 8"/></a></p>
4819 <h3 id="execution">Execution</h3>
4820 <p>In <code>Python/pythonrun.c</code> we broke out just before the call to <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pythonrun.c#L1094"><code>run_eval_code_obj()</code></a>.</p>
4821 <p>This call takes a code object, either fetched from the marshaled <code>.pyc</code> file, or compiled through the AST and compiler stages.</p>
4822 <p><a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/pythonrun.c#L1094"><code>run_eval_code_obj()</code></a> will pass the globals, locals, <code>PyArena</code>, and compiled <code>PyCodeObject</code> to <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/ceval.c#L716"><code>PyEval_EvalCode()</code></a> in <code>Python/ceval.c</code>.</p>
4823 <p>This stage forms the execution component of CPython. Each of the bytecode operations is taken and executed using a <a href="http://www.cs.uwm.edu/classes/cs315/Bacon/Lecture/HTML/ch10s07.html">“Stack Frame” based system</a>.</p>
4824 <div class="alert alert-primary" role="alert">
4825 <p><strong>What is a Stack Frame?</strong></p>
4826 <p>Stack Frames are a data type used by many runtimes, not just Python, that allows functions to be called and variables to be returned between functions. Stack Frames also contain arguments, local variables, and other state information.</p>
4827 <p>Typically, a Stack Frame exists for every function call, and they are stacked in sequence. You can see CPython’s frame stack anytime an exception is unhandled and the stack is printed on the screen.</p>
4828 </div>
4829 <p><a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/ceval.c#L716"><code>PyEval_EvalCode()</code></a> is the public API for evaluating a code object. The logic for evaluation is split between <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/ceval.c#L4045"><code>_PyEval_EvalCodeWithName()</code></a> and <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/ceval.c#L745"><code>_PyEval_EvalFrameDefault()</code></a>, which are both in <code>ceval.c</code>.</p>
4830 <p>The public API <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/ceval.c#L716"><code>PyEval_EvalCode()</code></a> will construct an execution frame from the top of the stack by calling <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/ceval.c#L4045"><code>_PyEval_EvalCodeWithName()</code></a>.</p>
4831 <p>The construction of the first execution frame has many steps:</p>
4832 <ol>
4833 <li>Keyword and positional arguments are resolved.</li>
4834 <li>The use of <code>*args</code> and <code>**kwargs</code> in function definitions are resolved.</li>
4835 <li>Arguments are added as local variables to the scope.</li>
4836 <li>Co-routines and <a href="https://realpython.com/introduction-to-python-generators/">Generators</a> are created, including the Asynchronous Generators.</li>
4837 </ol>
4838 <p>The frame object looks like this:</p>
4839 <p><a href="https://files.realpython.com/media/PyFrameObject.8616eee0503e.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/PyFrameObject.8616eee0503e.png" width="161" height="408" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/PyFrameObject.8616eee0503e.png&w=40&sig=5c85bcc7939e61d207a19cf82d23cab3f73ec760 40w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/PyFrameObject.8616eee0503e.png&w=80&sig=25333bbe22791650facbac4288bb9f49065fb014 80w, https://files.realpython.com/media/PyFrameObject.8616eee0503e.png 161w" sizes="75vw" alt="PyFrameObject structure"/></a></p>
4840 <p>Let’s step through those sequences.</p>
4841 <h4 id="1-constructing-thread-state">1. Constructing Thread State</h4>
4842 <p>Before a frame can be executed, it needs to be referenced from a thread. CPython can have many threads running at any one time within a single interpreter. An Interpreter state includes a list of those threads as a linked list. The thread structure is called <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Include/pystate.h#L23"><code>PyThreadState</code></a>, and there are many references throughout <code>ceval.c</code>.</p>
4843 <p>Here is the structure of the thread state object:</p>
4844 <p><a href="https://files.realpython.com/media/PyThreadState.20467f3689b7.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/PyThreadState.20467f3689b7.png" width="201" height="208" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/PyThreadState.20467f3689b7.png&w=50&sig=90efd8d98ffa8ad9ed8b233c1e73fa469e4db4ac 50w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/PyThreadState.20467f3689b7.png&w=100&sig=e5d4a21dbc5cce9c1017a6d1805cf9eedf03ac9c 100w, https://files.realpython.com/media/PyThreadState.20467f3689b7.png 201w" sizes="75vw" alt="PyThreadState structure"/></a></p>
4845 <h4 id="2-constructing-frames">2. Constructing Frames</h4>
4846 <p>The input to <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/ceval.c#L716"><code>PyEval_EvalCode()</code></a> and therefore <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/ceval.c#L4045"><code>_PyEval_EvalCodeWithName()</code></a> has arguments for:</p>
4847 <ul>
4848 <li><strong><code>_co</code>:</strong> a <code>PyCodeObject</code></li>
4849 <li><strong><code>globals</code>:</strong> a <code>PyDict</code> with variable names as keys and their values</li>
4850 <li><strong><code>locals</code>:</strong> a <code>PyDict</code> with variable names as keys and their values</li>
4851 </ul>
4852 <p>The other arguments are optional, and not used for the basic API:</p>
4853 <ul>
4854 <li><strong><code>args</code>:</strong> a <code>PyTuple</code> with positional argument values in order, and <code>argcount</code> for the number of values</li>
4855 <li><strong><code>kwnames</code>:</strong> a list of keyword argument names</li>
4856 <li><strong><code>kwargs</code>:</strong> a list of keyword argument values, and <code>kwcount</code> for the number of them</li>
4857 <li><strong><code>defs</code>:</strong> a list of default values for positional arguments, and <code>defcount</code> for the length</li>
4858 <li><strong><code>kwdefs</code>:</strong> a dictionary with the default values for keyword arguments</li>
4859 <li><strong><code>closure</code>:</strong> a tuple with strings to merge into the code objects <code>co_freevars</code> field</li>
4860 <li><strong><code>name</code>:</strong> the name for this evaluation statement as a string</li>
4861 <li><strong><code>qualname</code>:</strong> the qualified name for this evaluation statement as a string</li>
4862 </ul>
4863 <div class="highlight c"><pre><span></span><span class="n">PyObject</span> <span class="o">*</span>
4864 <span class="nf">_PyEval_EvalCodeWithName</span><span class="p">(</span><span class="n">PyObject</span> <span class="o">*</span><span class="n">_co</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">globals</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">locals</span><span class="p">,</span>
4865 <span class="n">PyObject</span> <span class="o">*</span><span class="k">const</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="n">Py_ssize_t</span> <span class="n">argcount</span><span class="p">,</span>
4866 <span class="n">PyObject</span> <span class="o">*</span><span class="k">const</span> <span class="o">*</span><span class="n">kwnames</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="k">const</span> <span class="o">*</span><span class="n">kwargs</span><span class="p">,</span>
4867 <span class="n">Py_ssize_t</span> <span class="n">kwcount</span><span class="p">,</span> <span class="kt">int</span> <span class="n">kwstep</span><span class="p">,</span>
4868 <span class="n">PyObject</span> <span class="o">*</span><span class="k">const</span> <span class="o">*</span><span class="n">defs</span><span class="p">,</span> <span class="n">Py_ssize_t</span> <span class="n">defcount</span><span class="p">,</span>
4869 <span class="n">PyObject</span> <span class="o">*</span><span class="n">kwdefs</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">closure</span><span class="p">,</span>
4870 <span class="n">PyObject</span> <span class="o">*</span><span class="n">name</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">qualname</span><span class="p">)</span>
4871 <span class="p">{</span>
4872 <span class="p">...</span>
4873
4874 <span class="n">PyThreadState</span> <span class="o">*</span><span class="n">tstate</span> <span class="o">=</span> <span class="n">_PyThreadState_GET</span><span class="p">();</span>
4875 <span class="n">assert</span><span class="p">(</span><span class="n">tstate</span> <span class="o">!=</span> <span class="nb">NULL</span><span class="p">);</span>
4876
4877 <span class="k">if</span> <span class="p">(</span><span class="n">globals</span> <span class="o">==</span> <span class="nb">NULL</span><span class="p">)</span> <span class="p">{</span>
4878 <span class="n">_PyErr_SetString</span><span class="p">(</span><span class="n">tstate</span><span class="p">,</span> <span class="n">PyExc_SystemError</span><span class="p">,</span>
4879 <span class="s">"PyEval_EvalCodeEx: NULL globals"</span><span class="p">);</span>
4880 <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
4881 <span class="p">}</span>
4882
4883 <span class="cm">/* Create the frame */</span>
4884 <span class="n">f</span> <span class="o">=</span> <span class="n">_PyFrame_New_NoTrack</span><span class="p">(</span><span class="n">tstate</span><span class="p">,</span> <span class="n">co</span><span class="p">,</span> <span class="n">globals</span><span class="p">,</span> <span class="n">locals</span><span class="p">);</span>
4885 <span class="k">if</span> <span class="p">(</span><span class="n">f</span> <span class="o">==</span> <span class="nb">NULL</span><span class="p">)</span> <span class="p">{</span>
4886 <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
4887 <span class="p">}</span>
4888 <span class="n">fastlocals</span> <span class="o">=</span> <span class="n">f</span><span class="o">-></span><span class="n">f_localsplus</span><span class="p">;</span>
4889 <span class="n">freevars</span> <span class="o">=</span> <span class="n">f</span><span class="o">-></span><span class="n">f_localsplus</span> <span class="o">+</span> <span class="n">co</span><span class="o">-></span><span class="n">co_nlocals</span><span class="p">;</span>
4890 </pre></div>
4891
4892 <h4 id="3-converting-keyword-parameters-to-a-dictionary">3. Converting Keyword Parameters to a Dictionary</h4>
4893 <p>If the function definition contained a <code>**kwargs</code> style catch-all for keyword arguments, then a new dictionary is created, and the values are copied across. The <code>kwargs</code> name is then set as a variable, like in this example:</p>
4894 <div class="highlight python"><pre><span></span><span class="k">def</span> <span class="nf">example</span><span class="p">(</span><span class="n">arg</span><span class="p">,</span> <span class="n">arg2</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
4895 <span class="nb">print</span><span class="p">(</span><span class="n">kwargs</span><span class="p">[</span><span class="s1">'extra'</span><span class="p">])</span> <span class="c1"># this would resolve to a dictionary key</span>
4896 </pre></div>
4897
4898 <p>The logic for creating a keyword argument dictionary is in the next part of <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/ceval.c#L4045">_PyEval_EvalCodeWithName()</a>:</p>
4899 <div class="highlight c"><pre><span></span> <span class="cm">/* Create a dictionary for keyword parameters (**kwargs) */</span>
4900 <span class="k">if</span> <span class="p">(</span><span class="n">co</span><span class="o">-></span><span class="n">co_flags</span> <span class="o">&</span> <span class="n">CO_VARKEYWORDS</span><span class="p">)</span> <span class="p">{</span>
4901 <span class="n">kwdict</span> <span class="o">=</span> <span class="n">PyDict_New</span><span class="p">();</span>
4902 <span class="k">if</span> <span class="p">(</span><span class="n">kwdict</span> <span class="o">==</span> <span class="nb">NULL</span><span class="p">)</span>
4903 <span class="k">goto</span> <span class="n">fail</span><span class="p">;</span>
4904 <span class="n">i</span> <span class="o">=</span> <span class="n">total_args</span><span class="p">;</span>
4905 <span class="k">if</span> <span class="p">(</span><span class="n">co</span><span class="o">-></span><span class="n">co_flags</span> <span class="o">&</span> <span class="n">CO_VARARGS</span><span class="p">)</span> <span class="p">{</span>
4906 <span class="n">i</span><span class="o">++</span><span class="p">;</span>
4907 <span class="p">}</span>
4908 <span class="n">SETLOCAL</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">kwdict</span><span class="p">);</span>
4909 <span class="p">}</span>
4910 <span class="k">else</span> <span class="p">{</span>
4911 <span class="n">kwdict</span> <span class="o">=</span> <span class="nb">NULL</span><span class="p">;</span>
4912 <span class="p">}</span>
4913 </pre></div>
4914
4915 <p>The <code>kwdict</code> variable will reference a <code>PyDictObject</code> if any keyword arguments were found.</p>
4916 <h4 id="4-converting-positional-arguments-into-variables">4. Converting Positional Arguments Into Variables</h4>
4917 <p>Next, each of the positional arguments (if provided) are set as local variables:</p>
4918 <div class="highlight c"><pre><span></span> <span class="cm">/* Copy all positional arguments into local variables */</span>
4919 <span class="k">if</span> <span class="p">(</span><span class="n">argcount</span> <span class="o">></span> <span class="n">co</span><span class="o">-></span><span class="n">co_argcount</span><span class="p">)</span> <span class="p">{</span>
4920 <span class="n">n</span> <span class="o">=</span> <span class="n">co</span><span class="o">-></span><span class="n">co_argcount</span><span class="p">;</span>
4921 <span class="p">}</span>
4922 <span class="k">else</span> <span class="p">{</span>
4923 <span class="n">n</span> <span class="o">=</span> <span class="n">argcount</span><span class="p">;</span>
4924 <span class="p">}</span>
4925 <span class="k">for</span> <span class="p">(</span><span class="n">j</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">j</span> <span class="o"><</span> <span class="n">n</span><span class="p">;</span> <span class="n">j</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
4926 <span class="n">x</span> <span class="o">=</span> <span class="n">args</span><span class="p">[</span><span class="n">j</span><span class="p">];</span>
4927 <span class="n">Py_INCREF</span><span class="p">(</span><span class="n">x</span><span class="p">);</span>
4928 <span class="n">SETLOCAL</span><span class="p">(</span><span class="n">j</span><span class="p">,</span> <span class="n">x</span><span class="p">);</span>
4929 <span class="p">}</span>
4930 </pre></div>
4931
4932 <p>At the end of the loop, you’ll see a call to <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/ceval.c#L1005"><code>SETLOCAL()</code></a> with the value, so if a positional argument is defined with a value, that is available within this scope: </p>
4933 <div class="highlight python"><pre><span></span><span class="k">def</span> <span class="nf">example</span><span class="p">(</span><span class="n">arg1</span><span class="p">,</span> <span class="n">arg2</span><span class="p">):</span>
4934 <span class="nb">print</span><span class="p">(</span><span class="n">arg1</span><span class="p">,</span> <span class="n">arg2</span><span class="p">)</span> <span class="c1"># both args are already local variables.</span>
4935 </pre></div>
4936
4937 <p>Also, the reference counter for those variables is incremented, so the garbage collector won’t remove them until the frame has evaluated.</p>
4938 <h4 id="5-packing-positional-arguments-into-args">5. Packing Positional Arguments Into <code>*args</code></h4>
4939 <p>Similar to <code>**kwargs</code>, a function argument prepended with a <code>*</code> can be set to catch all remaining positional arguments. This argument is a tuple and the <code>*args</code> name is set as a local variable: </p>
4940 <div class="highlight c"><pre><span></span> <span class="cm">/* Pack other positional arguments into the *args argument */</span>
4941 <span class="k">if</span> <span class="p">(</span><span class="n">co</span><span class="o">-></span><span class="n">co_flags</span> <span class="o">&</span> <span class="n">CO_VARARGS</span><span class="p">)</span> <span class="p">{</span>
4942 <span class="n">u</span> <span class="o">=</span> <span class="n">_PyTuple_FromArray</span><span class="p">(</span><span class="n">args</span> <span class="o">+</span> <span class="n">n</span><span class="p">,</span> <span class="n">argcount</span> <span class="o">-</span> <span class="n">n</span><span class="p">);</span>
4943 <span class="k">if</span> <span class="p">(</span><span class="n">u</span> <span class="o">==</span> <span class="nb">NULL</span><span class="p">)</span> <span class="p">{</span>
4944 <span class="k">goto</span> <span class="n">fail</span><span class="p">;</span>
4945 <span class="p">}</span>
4946 <span class="n">SETLOCAL</span><span class="p">(</span><span class="n">total_args</span><span class="p">,</span> <span class="n">u</span><span class="p">);</span>
4947 <span class="p">}</span>
4948 </pre></div>
4949
4950 <h4 id="6-loading-keyword-arguments">6. Loading Keyword Arguments</h4>
4951 <p>If the function was called with keyword arguments and values, the <code>kwdict</code> dictionary created in step 4 is now filled with any remaining keyword arguments passed by the caller that doesn’t resolve to named arguments or positional arguments.</p>
4952 <p>For example, the <code>e</code> argument was neither positional or named, so it is added to <code>**remaining</code>:</p>
4953 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">def</span> <span class="nf">my_function</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="n">c</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">d</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="o">**</span><span class="n">remaining</span><span class="p">):</span>
4954 <span class="go"> print(a, b, c, d, remaining)</span>
4955
4956 <span class="gp">>>> </span><span class="n">my_function</span><span class="p">(</span><span class="n">a</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">b</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">c</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span> <span class="n">d</span><span class="o">=</span><span class="mi">4</span><span class="p">,</span> <span class="n">e</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span>
4957 <span class="go">(1, 2, 3, 4, {'e': 5})</span>
4958 </pre></div>
4959
4960 <div class="alert alert-primary" role="alert">
4961 <p><strong>Positional-only arguments</strong> is a new feature in Python 3.8. Introduced in <a href="https://www.python.org/dev/peps/pep-0570/">PEP570</a>, positional-only arguments are a way of stopping users of your API from using positional arguments with a keyword syntax.</p>
4962 <p>For example, this simple function converts Farenheit to Celcius. Note, the use of <code>/</code> as a special argument seperates positional-only arguments from the other arguments.</p>
4963 <div class="highlight python"><pre><span></span><span class="k">def</span> <span class="nf">to_celcius</span><span class="p">(</span><span class="n">farenheit</span><span class="p">,</span> <span class="o">/</span><span class="p">,</span> <span class="n">options</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
4964 <span class="k">return</span> <span class="p">(</span><span class="n">farenheit</span><span class="o">-</span><span class="mi">31</span><span class="p">)</span><span class="o">*</span><span class="mi">5</span><span class="o">/</span><span class="mi">9</span>
4965 </pre></div>
4966
4967 <p>All arguments to the left of <code>/</code> must be called only as a positional argument, and arguments to the right can be called as either positional or keyword arguments:</p>
4968 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">to_celcius</span><span class="p">(</span><span class="mi">110</span><span class="p">)</span>
4969 </pre></div>
4970
4971 <p>Calling the function using a keyword argument to a positional-only argument will raise a <code>TypeError</code>:</p>
4972 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">to_celcius</span><span class="p">(</span><span class="n">farenheit</span><span class="o">=</span><span class="mi">110</span><span class="p">)</span>
4973 <span class="gt">Traceback (most recent call last):</span>
4974 File <span class="nb">"<stdin>"</span>, line <span class="m">1</span>, in <span class="n"><module></span>
4975 <span class="gr">TypeError</span>: <span class="n">to_celcius() got some positional-only arguments passed as keyword arguments: 'farenheit'</span>
4976 </pre></div>
4977
4978 </div>
4979 <p>The resolution of the keyword argument dictionary values comes after the unpacking of all other arguments. The PEP570 positional-only arguments are shown by starting the keyword-argument loop at <code>co_posonlyargcount</code>. If the <code>/</code> symbol was used on the 3rd argument, the value of <code>co_posonlyargcount</code> would be <code>2</code>.
4980 <code>PyDict_SetItem()</code> is called for each remaining argument to add it to the <code>locals</code> dictionary, so when executing, each of the keyword arguments are scoped local variables:</p>
4981 <div class="highlight c"><pre><span></span> <span class="k">for</span> <span class="p">(</span><span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">kwcount</span><span class="p">;</span> <span class="n">i</span> <span class="o">+=</span> <span class="n">kwstep</span><span class="p">)</span> <span class="p">{</span>
4982 <span class="n">PyObject</span> <span class="o">**</span><span class="n">co_varnames</span><span class="p">;</span>
4983 <span class="n">PyObject</span> <span class="o">*</span><span class="n">keyword</span> <span class="o">=</span> <span class="n">kwnames</span><span class="p">[</span><span class="n">i</span><span class="p">];</span>
4984 <span class="n">PyObject</span> <span class="o">*</span><span class="n">value</span> <span class="o">=</span> <span class="n">kwargs</span><span class="p">[</span><span class="n">i</span><span class="p">];</span>
4985 <span class="p">...</span>
4986
4987 <span class="cm">/* Speed hack: do raw pointer compares. As names are</span>
4988 <span class="cm"> normally interned this should almost always hit. */</span>
4989 <span class="n">co_varnames</span> <span class="o">=</span> <span class="p">((</span><span class="n">PyTupleObject</span> <span class="o">*</span><span class="p">)(</span><span class="n">co</span><span class="o">-></span><span class="n">co_varnames</span><span class="p">))</span><span class="o">-></span><span class="n">ob_item</span><span class="p">;</span>
4990 <span class="hll"> <span class="k">for</span> <span class="p">(</span><span class="n">j</span> <span class="o">=</span> <span class="n">co</span><span class="o">-></span><span class="n">co_posonlyargcount</span><span class="p">;</span> <span class="n">j</span> <span class="o"><</span> <span class="n">total_args</span><span class="p">;</span> <span class="n">j</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
4991 </span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">name</span> <span class="o">=</span> <span class="n">co_varnames</span><span class="p">[</span><span class="n">j</span><span class="p">];</span>
4992 <span class="k">if</span> <span class="p">(</span><span class="n">name</span> <span class="o">==</span> <span class="n">keyword</span><span class="p">)</span> <span class="p">{</span>
4993 <span class="k">goto</span> <span class="n">kw_found</span><span class="p">;</span>
4994 <span class="p">}</span>
4995 <span class="p">}</span>
4996
4997 <span class="k">if</span> <span class="p">(</span><span class="n">kwdict</span> <span class="o">==</span> <span class="nb">NULL</span><span class="p">)</span> <span class="p">{</span>
4998
4999 <span class="k">if</span> <span class="p">(</span><span class="n">co</span><span class="o">-></span><span class="n">co_posonlyargcount</span>
5000 <span class="o">&&</span> <span class="n">positional_only_passed_as_keyword</span><span class="p">(</span><span class="n">tstate</span><span class="p">,</span> <span class="n">co</span><span class="p">,</span>
5001 <span class="n">kwcount</span><span class="p">,</span> <span class="n">kwnames</span><span class="p">))</span>
5002 <span class="p">{</span>
5003 <span class="k">goto</span> <span class="n">fail</span><span class="p">;</span>
5004 <span class="p">}</span>
5005
5006 <span class="n">_PyErr_Format</span><span class="p">(</span><span class="n">tstate</span><span class="p">,</span> <span class="n">PyExc_TypeError</span><span class="p">,</span>
5007 <span class="s">"%U() got an unexpected keyword argument '%S'"</span><span class="p">,</span>
5008 <span class="n">co</span><span class="o">-></span><span class="n">co_name</span><span class="p">,</span> <span class="n">keyword</span><span class="p">);</span>
5009 <span class="k">goto</span> <span class="n">fail</span><span class="p">;</span>
5010 <span class="p">}</span>
5011
5012 <span class="k">if</span> <span class="p">(</span><span class="n">PyDict_SetItem</span><span class="p">(</span><span class="n">kwdict</span><span class="p">,</span> <span class="n">keyword</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span> <span class="o">==</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
5013 <span class="k">goto</span> <span class="n">fail</span><span class="p">;</span>
5014 <span class="p">}</span>
5015 <span class="k">continue</span><span class="p">;</span>
5016
5017 <span class="nl">kw_found</span><span class="p">:</span>
5018 <span class="p">...</span>
5019 <span class="n">Py_INCREF</span><span class="p">(</span><span class="n">value</span><span class="p">);</span>
5020 <span class="n">SETLOCAL</span><span class="p">(</span><span class="n">j</span><span class="p">,</span> <span class="n">value</span><span class="p">);</span>
5021 <span class="p">}</span>
5022 <span class="p">...</span>
5023 </pre></div>
5024
5025 <p>At the end of the loop, you’ll see a call to <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/ceval.c#L1005"><code>SETLOCAL()</code></a> with the value. If a keyword argument is defined with a value, that is available within this scope:</p>
5026 <div class="highlight python"><pre><span></span><span class="k">def</span> <span class="nf">example</span><span class="p">(</span><span class="n">arg1</span><span class="p">,</span> <span class="n">arg2</span><span class="p">,</span> <span class="n">example_kwarg</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
5027 <span class="nb">print</span><span class="p">(</span><span class="n">example_kwarg</span><span class="p">)</span> <span class="c1"># example_kwarg is already a local variable.</span>
5028 </pre></div>
5029
5030 <h4 id="7-adding-missing-positional-arguments">7. Adding Missing Positional Arguments</h4>
5031 <p>Any positional arguments provided to a function call that are not in the list of positional arguments are added to a <code>*args</code> tuple if this tuple does not exist, a failure is raised:</p>
5032 <div class="highlight c"><pre><span></span> <span class="cm">/* Add missing positional arguments (copy default values from defs) */</span>
5033 <span class="k">if</span> <span class="p">(</span><span class="n">argcount</span> <span class="o"><</span> <span class="n">co</span><span class="o">-></span><span class="n">co_argcount</span><span class="p">)</span> <span class="p">{</span>
5034 <span class="n">Py_ssize_t</span> <span class="n">m</span> <span class="o">=</span> <span class="n">co</span><span class="o">-></span><span class="n">co_argcount</span> <span class="o">-</span> <span class="n">defcount</span><span class="p">;</span>
5035 <span class="n">Py_ssize_t</span> <span class="n">missing</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
5036 <span class="k">for</span> <span class="p">(</span><span class="n">i</span> <span class="o">=</span> <span class="n">argcount</span><span class="p">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">m</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
5037 <span class="k">if</span> <span class="p">(</span><span class="n">GETLOCAL</span><span class="p">(</span><span class="n">i</span><span class="p">)</span> <span class="o">==</span> <span class="nb">NULL</span><span class="p">)</span> <span class="p">{</span>
5038 <span class="n">missing</span><span class="o">++</span><span class="p">;</span>
5039 <span class="p">}</span>
5040 <span class="p">}</span>
5041 <span class="k">if</span> <span class="p">(</span><span class="n">missing</span><span class="p">)</span> <span class="p">{</span>
5042 <span class="n">missing_arguments</span><span class="p">(</span><span class="n">co</span><span class="p">,</span> <span class="n">missing</span><span class="p">,</span> <span class="n">defcount</span><span class="p">,</span> <span class="n">fastlocals</span><span class="p">);</span>
5043 <span class="k">goto</span> <span class="n">fail</span><span class="p">;</span>
5044 <span class="p">}</span>
5045 <span class="k">if</span> <span class="p">(</span><span class="n">n</span> <span class="o">></span> <span class="n">m</span><span class="p">)</span>
5046 <span class="n">i</span> <span class="o">=</span> <span class="n">n</span> <span class="o">-</span> <span class="n">m</span><span class="p">;</span>
5047 <span class="k">else</span>
5048 <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
5049 <span class="k">for</span> <span class="p">(;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">defcount</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
5050 <span class="k">if</span> <span class="p">(</span><span class="n">GETLOCAL</span><span class="p">(</span><span class="n">m</span><span class="o">+</span><span class="n">i</span><span class="p">)</span> <span class="o">==</span> <span class="nb">NULL</span><span class="p">)</span> <span class="p">{</span>
5051 <span class="n">PyObject</span> <span class="o">*</span><span class="n">def</span> <span class="o">=</span> <span class="n">defs</span><span class="p">[</span><span class="n">i</span><span class="p">];</span>
5052 <span class="n">Py_INCREF</span><span class="p">(</span><span class="n">def</span><span class="p">);</span>
5053 <span class="n">SETLOCAL</span><span class="p">(</span><span class="n">m</span><span class="o">+</span><span class="n">i</span><span class="p">,</span> <span class="n">def</span><span class="p">);</span>
5054 <span class="p">}</span>
5055 <span class="p">}</span>
5056 <span class="p">}</span>
5057 </pre></div>
5058
5059 <h4 id="8-adding-missing-keyword-arguments">8. Adding Missing Keyword Arguments</h4>
5060 <p>Any keyword arguments provided to a function call that are not in the list of named keyword arguments are added to a <code>**kwargs</code> dictionary if this dictionary does not exist, a failure is raised:</p>
5061 <div class="highlight c"><pre><span></span> <span class="cm">/* Add missing keyword arguments (copy default values from kwdefs) */</span>
5062 <span class="k">if</span> <span class="p">(</span><span class="n">co</span><span class="o">-></span><span class="n">co_kwonlyargcount</span> <span class="o">></span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
5063 <span class="n">Py_ssize_t</span> <span class="n">missing</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
5064 <span class="k">for</span> <span class="p">(</span><span class="n">i</span> <span class="o">=</span> <span class="n">co</span><span class="o">-></span><span class="n">co_argcount</span><span class="p">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">total_args</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
5065 <span class="n">PyObject</span> <span class="o">*</span><span class="n">name</span><span class="p">;</span>
5066 <span class="k">if</span> <span class="p">(</span><span class="n">GETLOCAL</span><span class="p">(</span><span class="n">i</span><span class="p">)</span> <span class="o">!=</span> <span class="nb">NULL</span><span class="p">)</span>
5067 <span class="k">continue</span><span class="p">;</span>
5068 <span class="n">name</span> <span class="o">=</span> <span class="n">PyTuple_GET_ITEM</span><span class="p">(</span><span class="n">co</span><span class="o">-></span><span class="n">co_varnames</span><span class="p">,</span> <span class="n">i</span><span class="p">);</span>
5069 <span class="k">if</span> <span class="p">(</span><span class="n">kwdefs</span> <span class="o">!=</span> <span class="nb">NULL</span><span class="p">)</span> <span class="p">{</span>
5070 <span class="n">PyObject</span> <span class="o">*</span><span class="n">def</span> <span class="o">=</span> <span class="n">PyDict_GetItemWithError</span><span class="p">(</span><span class="n">kwdefs</span><span class="p">,</span> <span class="n">name</span><span class="p">);</span>
5071 <span class="p">...</span>
5072 <span class="p">}</span>
5073 <span class="n">missing</span><span class="o">++</span><span class="p">;</span>
5074 <span class="p">}</span>
5075 <span class="p">...</span>
5076 <span class="p">}</span>
5077 </pre></div>
5078
5079 <h4 id="9-collapsing-closures">9. Collapsing Closures</h4>
5080 <p>Any closure names are added to the code object’s list of free variable names:</p>
5081 <div class="highlight c"><pre><span></span> <span class="cm">/* Copy closure variables to free variables */</span>
5082 <span class="k">for</span> <span class="p">(</span><span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">PyTuple_GET_SIZE</span><span class="p">(</span><span class="n">co</span><span class="o">-></span><span class="n">co_freevars</span><span class="p">);</span> <span class="o">++</span><span class="n">i</span><span class="p">)</span> <span class="p">{</span>
5083 <span class="n">PyObject</span> <span class="o">*</span><span class="n">o</span> <span class="o">=</span> <span class="n">PyTuple_GET_ITEM</span><span class="p">(</span><span class="n">closure</span><span class="p">,</span> <span class="n">i</span><span class="p">);</span>
5084 <span class="n">Py_INCREF</span><span class="p">(</span><span class="n">o</span><span class="p">);</span>
5085 <span class="n">freevars</span><span class="p">[</span><span class="n">PyTuple_GET_SIZE</span><span class="p">(</span><span class="n">co</span><span class="o">-></span><span class="n">co_cellvars</span><span class="p">)</span> <span class="o">+</span> <span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="n">o</span><span class="p">;</span>
5086 <span class="p">}</span>
5087 </pre></div>
5088
5089 <h4 id="10-creating-generators-coroutines-and-asynchronous-generators">10. Creating Generators, Coroutines, and Asynchronous Generators</h4>
5090 <p>If the evaluated code object has a flag that it is a generator, coroutine or async generator, then a new frame is created using one of the unique methods in the Generator, Coroutine or Async libraries and the current frame is added as a property.</p>
5091 <p>The new frame is then returned, and the original frame is not evaluated. The frame is only evaluated when the generator/coroutine/async method is called on to execute its target:</p>
5092 <div class="highlight c"><pre><span></span> <span class="cm">/* Handle generator/coroutine/asynchronous generator */</span>
5093 <span class="k">if</span> <span class="p">(</span><span class="n">co</span><span class="o">-></span><span class="n">co_flags</span> <span class="o">&</span> <span class="p">(</span><span class="n">CO_GENERATOR</span> <span class="o">|</span> <span class="n">CO_COROUTINE</span> <span class="o">|</span> <span class="n">CO_ASYNC_GENERATOR</span><span class="p">))</span> <span class="p">{</span>
5094 <span class="p">...</span>
5095
5096 <span class="cm">/* Create a new generator that owns the ready to run frame</span>
5097 <span class="cm"> * and return that as the value. */</span>
5098 <span class="k">if</span> <span class="p">(</span><span class="n">is_coro</span><span class="p">)</span> <span class="p">{</span>
5099 <span class="n">gen</span> <span class="o">=</span> <span class="n">PyCoro_New</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">qualname</span><span class="p">);</span>
5100 <span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="n">co</span><span class="o">-></span><span class="n">co_flags</span> <span class="o">&</span> <span class="n">CO_ASYNC_GENERATOR</span><span class="p">)</span> <span class="p">{</span>
5101 <span class="n">gen</span> <span class="o">=</span> <span class="n">PyAsyncGen_New</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">qualname</span><span class="p">);</span>
5102 <span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
5103 <span class="n">gen</span> <span class="o">=</span> <span class="n">PyGen_NewWithQualName</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">qualname</span><span class="p">);</span>
5104 <span class="p">}</span>
5105 <span class="p">...</span>
5106
5107 <span class="k">return</span> <span class="n">gen</span><span class="p">;</span>
5108 <span class="p">}</span>
5109 </pre></div>
5110
5111 <p>Lastly, <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/ceval.c#L738"><code>PyEval_EvalFrameEx()</code></a> is called with the new frame:</p>
5112 <div class="highlight c"><pre><span></span> <span class="n">retval</span> <span class="o">=</span> <span class="n">PyEval_EvalFrameEx</span><span class="p">(</span><span class="n">f</span><span class="p">,</span><span class="mi">0</span><span class="p">);</span>
5113 <span class="p">...</span>
5114 <span class="p">}</span>
5115 </pre></div>
5116
5117 <h4 id="frame-execution">Frame Execution</h4>
5118 <p>As covered earlier in the compiler and AST chapters, the code object contains a binary encoding of the bytecode to be executed. It also contains a list of variables and a symbol table.</p>
5119 <p>The local and global variables are determined at runtime based on how that function, module, or block was called. This information is added to the frame by the <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/ceval.c#L4045"><code>_PyEval_EvalCodeWithName()</code></a> function. There are other usages of frames, like the coroutine decorator, which dynamically generates a frame with the target as a variable.</p>
5120 <p>The public API, <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/ceval.c#L738"><code>PyEval_EvalFrameEx()</code></a> calls the interpreter’s configured frame evaluation function in the <code>eval_frame</code> property. Frame evaluation was <a href="https://www.python.org/dev/peps/pep-0523/">made pluggable in Python 3.7 with PEP 523</a>.</p>
5121 <p><a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/ceval.c#L745"><code>_PyEval_EvalFrameDefault()</code></a> is the default function, and it is unusual to use anything other than this. </p>
5122 <p>Frames are executed in the main execution loop inside <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/ceval.c#L745"><code>_PyEval_EvalFrameDefault()</code></a>. This function is central function that brings everything together and brings your code to life. It contains decades of optimization since even a single line of code can have a significant impact on performance for the whole of CPython.</p>
5123 <p>Everything that gets executed in CPython goes through this function.</p>
5124 <div class="alert alert-primary" role="alert">
5125 <p><strong>Note:</strong> Something you might notice when reading <code>ceval.c</code>, is how many times C macros have been used. C Macros are a way of having DRY-compliant code without the overhead of making function calls. The compiler converts the macros into C code and then compile the generated code. </p>
5126 <p>If you want to see the expanded code, you can run <code>gcc -E</code> on Linux and macOS:</p>
5127 <div class="highlight sh"><pre><span></span><span class="gp">$</span> gcc -E Python/ceval.c
5128 </pre></div>
5129
5130 <p>Alternatively, <a href="https://realpython.com/python-development-visual-studio-code/">Visual Studio Code</a> can do inline macro expansion once you have installed the official C/C++ extension:</p>
5131 <p><a href="https://files.realpython.com/media/Screen_Shot_2019-03-25_at_3.33.40_pm.c240b1a46e99.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/Screen_Shot_2019-03-25_at_3.33.40_pm.c240b1a46e99.png" width="1866" height="622" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screen_Shot_2019-03-25_at_3.33.40_pm.c240b1a46e99.png&w=466&sig=28bc3a2f62e499a92fdac8fbd951c6ab5a18d361 466w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screen_Shot_2019-03-25_at_3.33.40_pm.c240b1a46e99.png&w=933&sig=ec297187398d1a9b390bdadedd2c96855b2dfacc 933w, https://files.realpython.com/media/Screen_Shot_2019-03-25_at_3.33.40_pm.c240b1a46e99.png 1866w" sizes="75vw" alt="C Macro expansion with VScode"/></a>
5132 </p>
5133 </div>
5134 <p>We can step through frame execution in Python 3.7 and beyond by enabling the tracing attribute on the current thread.</p>
5135 <p>This code example sets the global tracing function to a function called <code>trace()</code> that gets the stack from the current frame, prints the disassembled opcodes to the screen, and some extra information for debugging:</p>
5136 <div class="highlight python"><pre><span></span><span class="kn">import</span> <span class="nn">sys</span>
5137 <span class="kn">import</span> <span class="nn">dis</span>
5138 <span class="kn">import</span> <span class="nn">traceback</span>
5139 <span class="kn">import</span> <span class="nn">io</span>
5140
5141 <span class="k">def</span> <span class="nf">trace</span><span class="p">(</span><span class="n">frame</span><span class="p">,</span> <span class="n">event</span><span class="p">,</span> <span class="n">args</span><span class="p">):</span>
5142 <span class="n">frame</span><span class="o">.</span><span class="n">f_trace_opcodes</span> <span class="o">=</span> <span class="kc">True</span>
5143 <span class="n">stack</span> <span class="o">=</span> <span class="n">traceback</span><span class="o">.</span><span class="n">extract_stack</span><span class="p">(</span><span class="n">frame</span><span class="p">)</span>
5144 <span class="n">pad</span> <span class="o">=</span> <span class="s2">" "</span><span class="o">*</span><span class="nb">len</span><span class="p">(</span><span class="n">stack</span><span class="p">)</span> <span class="o">+</span> <span class="s2">"|"</span>
5145 <span class="k">if</span> <span class="n">event</span> <span class="o">==</span> <span class="s1">'opcode'</span><span class="p">:</span>
5146 <span class="k">with</span> <span class="n">io</span><span class="o">.</span><span class="n">StringIO</span><span class="p">()</span> <span class="k">as</span> <span class="n">out</span><span class="p">:</span>
5147 <span class="n">dis</span><span class="o">.</span><span class="n">disco</span><span class="p">(</span><span class="n">frame</span><span class="o">.</span><span class="n">f_code</span><span class="p">,</span> <span class="n">frame</span><span class="o">.</span><span class="n">f_lasti</span><span class="p">,</span> <span class="n">file</span><span class="o">=</span><span class="n">out</span><span class="p">)</span>
5148 <span class="n">lines</span> <span class="o">=</span> <span class="n">out</span><span class="o">.</span><span class="n">getvalue</span><span class="p">()</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">)</span>
5149 <span class="p">[</span><span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s2">"</span><span class="si">{pad}{l}</span><span class="s2">"</span><span class="p">)</span> <span class="k">for</span> <span class="n">l</span> <span class="ow">in</span> <span class="n">lines</span><span class="p">]</span>
5150 <span class="k">elif</span> <span class="n">event</span> <span class="o">==</span> <span class="s1">'call'</span><span class="p">:</span>
5151 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s2">"</span><span class="si">{pad}</span><span class="s2">Calling </span><span class="si">{frame.f_code}</span><span class="s2">"</span><span class="p">)</span>
5152 <span class="k">elif</span> <span class="n">event</span> <span class="o">==</span> <span class="s1">'return'</span><span class="p">:</span>
5153 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s2">"</span><span class="si">{pad}</span><span class="s2">Returning </span><span class="si">{args}</span><span class="s2">"</span><span class="p">)</span>
5154 <span class="k">elif</span> <span class="n">event</span> <span class="o">==</span> <span class="s1">'line'</span><span class="p">:</span>
5155 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s2">"</span><span class="si">{pad}</span><span class="s2">Changing line to </span><span class="si">{frame.f_lineno}</span><span class="s2">"</span><span class="p">)</span>
5156 <span class="k">else</span><span class="p">:</span>
5157 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s2">"</span><span class="si">{pad}{frame}</span><span class="s2"> (</span><span class="si">{event}</span><span class="s2"> - </span><span class="si">{args}</span><span class="s2">)"</span><span class="p">)</span>
5158 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s2">"</span><span class="si">{pad}</span><span class="s2">----------------------------------"</span><span class="p">)</span>
5159 <span class="k">return</span> <span class="n">trace</span>
5160 <span class="n">sys</span><span class="o">.</span><span class="n">settrace</span><span class="p">(</span><span class="n">trace</span><span class="p">)</span>
5161
5162 <span class="c1"># Run some code for a demo</span>
5163 <span class="nb">eval</span><span class="p">(</span><span class="s1">'"-".join([letter for letter in "hello"])'</span><span class="p">)</span>
5164 </pre></div>
5165
5166 <p>This prints the code within each stack and point to the next operation before it is executed. When a frame returns a value, the return statement is printed:</p>
5167 <p><a href="https://files.realpython.com/media/Screen_Shot_2019-03-25_at_1.21.07_pm.7b03e9032f62.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/Screen_Shot_2019-03-25_at_1.21.07_pm.7b03e9032f62.png" width="2572" height="1686" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screen_Shot_2019-03-25_at_1.21.07_pm.7b03e9032f62.png&w=643&sig=6d6106abaf9fb5f3d7e45e09320c7575b3a0e77c 643w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Screen_Shot_2019-03-25_at_1.21.07_pm.7b03e9032f62.png&w=1286&sig=c41bf26bd15641fded6a31125b0cb5b630496647 1286w, https://files.realpython.com/media/Screen_Shot_2019-03-25_at_1.21.07_pm.7b03e9032f62.png 2572w" sizes="75vw" alt="Evaluating frame with tracing"/></a></p>
5168 <p>The full list of instructions is available on the <a href="https://docs.python.org/3/library/dis.html#python-bytecode-instructions"><code>dis</code> module</a> documentation. </p>
5169 <h4 id="the-value-stack">The Value Stack</h4>
5170 <p>Inside the core evaluation loop, a value stack is created. This stack is a list of pointers to sequential <code>PyObject</code> instances.</p>
5171 <p>One way to think of the value stack is like a wooden peg on which you can stack cylinders. You would only add or remove one item at a time. This is done using the <code>PUSH(a)</code> macro, where <code>a</code> is a pointer to a <code>PyObject</code>.</p>
5172 <p>For example, if you created a <code>PyLong</code> with the value 10 and pushed it onto the value stack:</p>
5173 <div class="highlight c"><pre><span></span><span class="n">PyObject</span> <span class="o">*</span><span class="n">a</span> <span class="o">=</span> <span class="n">PyLong_FromLong</span><span class="p">(</span><span class="mi">10</span><span class="p">);</span>
5174 <span class="n">PUSH</span><span class="p">(</span><span class="n">a</span><span class="p">);</span>
5175 </pre></div>
5176
5177 <p>This action would have the following effect:</p>
5178 <p><a href="https://files.realpython.com/media/stacks_push.0c755d83b347.png" target="_blank"><img class="img-fluid " src="https://files.realpython.com/media/stacks_push.0c755d83b347.png" width="822" height="279" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/stacks_push.0c755d83b347.png&w=205&sig=d3ce00e768e8c7f89ef321ea039b159131b39d8a 205w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/stacks_push.0c755d83b347.png&w=411&sig=65cf700de41fd59781082f03c5d347fd891ac5a9 411w, https://files.realpython.com/media/stacks_push.0c755d83b347.png 822w" sizes="75vw" alt="PUSH()"/></a></p>
5179 <p>In the next operation, to fetch that value, you would use the <code>POP()</code> macro to take the top value from the stack:</p>
5180 <div class="highlight c"><pre><span></span><span class="n">PyObject</span> <span class="o">*</span><span class="n">a</span> <span class="o">=</span> <span class="n">POP</span><span class="p">();</span> <span class="c1">// a is PyLongObject with a value of 10</span>
5181 </pre></div>
5182
5183 <p>This action would return the top value and end up with an empty value stack:</p>
5184 <p><a href="https://files.realpython.com/media/stacks_pop.85872baaa8c3.png" target="_blank"><img class="img-fluid " src="https://files.realpython.com/media/stacks_pop.85872baaa8c3.png" width="778" height="226" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/stacks_pop.85872baaa8c3.png&w=194&sig=887719d9b65d3ef671e2310aae15904e04acba28 194w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/stacks_pop.85872baaa8c3.png&w=389&sig=6950fcd6370022df61559d12b74c6b685a61e238 389w, https://files.realpython.com/media/stacks_pop.85872baaa8c3.png 778w" sizes="75vw" alt="POP()"/></a></p>
5185 <p>If you were to add 2 values to the stack:</p>
5186 <div class="highlight c"><pre><span></span><span class="n">PyObject</span> <span class="o">*</span><span class="n">a</span> <span class="o">=</span> <span class="n">PyLong_FromLong</span><span class="p">(</span><span class="mi">10</span><span class="p">);</span>
5187 <span class="n">PyObject</span> <span class="o">*</span><span class="n">b</span> <span class="o">=</span> <span class="n">PyLong_FromLong</span><span class="p">(</span><span class="mi">20</span><span class="p">);</span>
5188 <span class="n">PUSH</span><span class="p">(</span><span class="n">a</span><span class="p">);</span>
5189 <span class="n">PUSH</span><span class="p">(</span><span class="n">b</span><span class="p">);</span>
5190 </pre></div>
5191
5192 <p>They would end up in the order in which they were added, so <code>a</code> would be pushed to the second position in the stack:</p>
5193 <p><a href="https://files.realpython.com/media/stacks_pushpush.e052996db029.png" target="_blank"><img class="img-fluid " src="https://files.realpython.com/media/stacks_pushpush.e052996db029.png" width="783" height="226" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/stacks_pushpush.e052996db029.png&w=195&sig=47cc1b6b30f314f1589eb9c8188efcd3eb0ef79e 195w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/stacks_pushpush.e052996db029.png&w=391&sig=a0813bcc81d314c3f2f0eb146634ba187d447634 391w, https://files.realpython.com/media/stacks_pushpush.e052996db029.png 783w" sizes="75vw" alt="PUSH();PUSH()"/></a></p>
5194 <p>If you were to fetch the top value in the stack, you would get a pointer to <code>b</code> because it is at the top:</p>
5195 <p><a href="https://files.realpython.com/media/stacks_pop2.af5068718f92.png" target="_blank"><img class="img-fluid " src="https://files.realpython.com/media/stacks_pop2.af5068718f92.png" width="789" height="226" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/stacks_pop2.af5068718f92.png&w=197&sig=1856b32d5f95f58eb2ad60a549165f85bf2b4d7a 197w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/stacks_pop2.af5068718f92.png&w=394&sig=ce3ba993ee2fa8cb037bb84144c73f95450954ac 394w, https://files.realpython.com/media/stacks_pop2.af5068718f92.png 789w" sizes="75vw" alt="POP();"/></a></p>
5196 <p>If you need to fetch the pointer to the top value in the stack without popping it, you can use the <code>PEEK(v)</code> operation, where <code>v</code> is the stack position:</p>
5197 <div class="highlight c"><pre><span></span><span class="n">PyObject</span> <span class="o">*</span><span class="n">first</span> <span class="o">=</span> <span class="n">PEEK</span><span class="p">(</span><span class="mi">0</span><span class="p">);</span>
5198 </pre></div>
5199
5200 <p>0 represents the top of the stack, 1 would be the second position:</p>
5201 <p><a href="https://files.realpython.com/media/stacks_peek.b00bde86bc7b.png" target="_blank"><img class="img-fluid " src="https://files.realpython.com/media/stacks_peek.b00bde86bc7b.png" width="802" height="174" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/stacks_peek.b00bde86bc7b.png&w=200&sig=fac4fef886154a979f9abe7316e9468179c14c71 200w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/stacks_peek.b00bde86bc7b.png&w=401&sig=463aeb044f65a4c446a37c71cd86ae71564a2ec6 401w, https://files.realpython.com/media/stacks_peek.b00bde86bc7b.png 802w" sizes="75vw" alt="PEEK()"/></a></p>
5202 <p>To clone the value at the top of the stack, the <code>DUP_TWO()</code> macro can be used, or by using the <code>DUP_TWO</code> opcode:</p>
5203 <div class="highlight c"><pre><span></span><span class="n">DUP_TOP</span><span class="p">();</span>
5204 </pre></div>
5205
5206 <p>This action would copy the value at the top to form 2 pointers to the same object:</p>
5207 <p><a href="https://files.realpython.com/media/stacks_duptop.9b2eafe67375.png" target="_blank"><img class="img-fluid " src="https://files.realpython.com/media/stacks_duptop.9b2eafe67375.png" width="799" height="177" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/stacks_duptop.9b2eafe67375.png&w=199&sig=3d40e1422f0aa5f3b50b8b174d02f0fe5df54364 199w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/stacks_duptop.9b2eafe67375.png&w=399&sig=11e2d5a69b3dd385c9201d7ab4eeb902f8fa20c9 399w, https://files.realpython.com/media/stacks_duptop.9b2eafe67375.png 799w" sizes="75vw" alt="DUP_TOP()"/></a></p>
5208 <p>There is a rotation macro <code>ROT_TWO</code> that swaps the first and second values:</p>
5209 <p><a href="https://files.realpython.com/media/stacks_rottwo.2990d9b3ecc7.png" target="_blank"><img class="img-fluid " src="https://files.realpython.com/media/stacks_rottwo.2990d9b3ecc7.png" width="803" height="175" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/stacks_rottwo.2990d9b3ecc7.png&w=200&sig=e3e88549e9801f463358ad1e8b5635ba56b77049 200w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/stacks_rottwo.2990d9b3ecc7.png&w=401&sig=d2c11f545336f13fa57bc1588f57b7e7f8dc049f 401w, https://files.realpython.com/media/stacks_rottwo.2990d9b3ecc7.png 803w" sizes="75vw" alt="ROT_TWO()"/></a></p>
5210 <p>Each of the opcodes have a predefined “stack effect,” calculated by the <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/compile.c#L878"><code>stack_effect()</code></a> function inside <code>Python/compile.c</code>. This function returns the delta in the number of values inside the stack for each opcode.</p>
5211 <h4 id="example-adding-an-item-to-a-list">Example: Adding an Item to a List</h4>
5212 <p>In Python, when you create a list, the <code>.append()</code> method is available on the list object:</p>
5213 <div class="highlight python"><pre><span></span><span class="n">my_list</span> <span class="o">=</span> <span class="p">[]</span>
5214 <span class="n">my_list</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">obj</span><span class="p">)</span>
5215 </pre></div>
5216
5217 <p>Where <code>obj</code> is an object, you want to append to the end of the list.</p>
5218 <p>There are 2 operations involved in this operation. <code>LOAD_FAST</code>, to load the object <code>obj</code> to the top of the value stack from the list of <code>locals</code> in the frame, and <code>LIST_APPEND</code> to add the object.</p>
5219 <p>First exploring <code>LOAD_FAST</code>, there are 5 steps:</p>
5220 <ol>
5221 <li>
5222 <p>The pointer to <code>obj</code> is loaded from <code>GETLOCAL()</code>, where the variable to load is the operation argument. The list of variable pointers is stored in <code>fastlocals</code>, which is a copy of the PyFrame attribute <code>f_localsplus</code>. The operation argument is a number, pointing to the index in the <code>fastlocals</code> array pointer. This means that the loading of a local is simply a copy of the pointer instead of having to look up the variable name.</p>
5223 </li>
5224 <li>
5225 <p>If variable no longer exists, an unbound local variable error is raised.</p>
5226 </li>
5227 <li>
5228 <p>The reference counter for <code>value</code> (in our case, <code>obj</code>) is increased by 1.</p>
5229 </li>
5230 <li>
5231 <p>The pointer to <code>obj</code> is pushed to the top of the value stack.</p>
5232 </li>
5233 <li>
5234 <p>The <code>FAST_DISPATCH</code> macro is called, if tracing is enabled, the loop goes over again (with all the tracing), if tracing is not enabled, a <code>goto</code> is called to <code>fast_next_opcode</code>, which jumps back to the top of the loop for the next instruction.</p>
5235 </li>
5236 </ol>
5237 <div class="highlight c"><pre><span></span> <span class="p">...</span>
5238 <span class="k">case</span> <span class="n">TARGET</span><span class="p">(</span><span class="n">LOAD_FAST</span><span class="p">)</span><span class="o">:</span> <span class="p">{</span>
5239 <span class="n">PyObject</span> <span class="o">*</span><span class="n">value</span> <span class="o">=</span> <span class="n">GETLOCAL</span><span class="p">(</span><span class="n">oparg</span><span class="p">);</span> <span class="c1">// 1.</span>
5240 <span class="k">if</span> <span class="p">(</span><span class="n">value</span> <span class="o">==</span> <span class="nb">NULL</span><span class="p">)</span> <span class="p">{</span>
5241 <span class="n">format_exc_check_arg</span><span class="p">(</span>
5242 <span class="n">PyExc_UnboundLocalError</span><span class="p">,</span>
5243 <span class="n">UNBOUNDLOCAL_ERROR_MSG</span><span class="p">,</span>
5244 <span class="n">PyTuple_GetItem</span><span class="p">(</span><span class="n">co</span><span class="o">-></span><span class="n">co_varnames</span><span class="p">,</span> <span class="n">oparg</span><span class="p">));</span>
5245 <span class="k">goto</span> <span class="n">error</span><span class="p">;</span> <span class="c1">// 2.</span>
5246 <span class="p">}</span>
5247 <span class="n">Py_INCREF</span><span class="p">(</span><span class="n">value</span><span class="p">);</span> <span class="c1">// 3.</span>
5248 <span class="n">PUSH</span><span class="p">(</span><span class="n">value</span><span class="p">);</span> <span class="c1">// 4.</span>
5249 <span class="n">FAST_DISPATCH</span><span class="p">();</span> <span class="c1">// 5.</span>
5250 <span class="p">}</span>
5251 <span class="p">...</span>
5252 </pre></div>
5253
5254 <p>Now the pointer to <code>obj</code> is at the top of the value stack. The next instruction <code>LIST_APPEND</code> is run.</p>
5255 <p>Many of the bytecode operations are referencing the base types, like PyUnicode, PyNumber. For example, <code>LIST_APPEND</code> appends an object to the end of a list. To achieve this, it pops the pointer from the value stack and returns the pointer to the last object in the stack. The macro is a shortcut for: </p>
5256 <div class="highlight c"><pre><span></span><span class="n">PyObject</span> <span class="o">*</span><span class="n">v</span> <span class="o">=</span> <span class="p">(</span><span class="o">*--</span><span class="n">stack_pointer</span><span class="p">);</span>
5257 </pre></div>
5258
5259 <p>Now the pointer to <code>obj</code> is stored as <code>v</code>. The list pointer is loaded from <code>PEEK(oparg)</code>.</p>
5260 <p>Then the C API for Python lists is called for <code>list</code> and <code>v</code>. The code for this is inside <code>Objects/listobject.c</code>, which we go into in the next chapter.</p>
5261 <p>A call to <code>PREDICT</code> is made, which guesses that the next operation will be <code>JUMP_ABSOLUTE</code>. The <code>PREDICT</code> macro has compiler-generated <code>goto</code> statements for each of the potential operations’ <code>case</code> statements. This means the CPU can jump to that instruction and not have to go through the loop again:</p>
5262 <div class="highlight c"><pre><span></span> <span class="p">...</span>
5263 <span class="k">case</span> <span class="n">TARGET</span><span class="p">(</span><span class="n">LIST_APPEND</span><span class="p">)</span><span class="o">:</span> <span class="p">{</span>
5264 <span class="n">PyObject</span> <span class="o">*</span><span class="n">v</span> <span class="o">=</span> <span class="n">POP</span><span class="p">();</span>
5265 <span class="n">PyObject</span> <span class="o">*</span><span class="n">list</span> <span class="o">=</span> <span class="n">PEEK</span><span class="p">(</span><span class="n">oparg</span><span class="p">);</span>
5266 <span class="kt">int</span> <span class="n">err</span><span class="p">;</span>
5267 <span class="hll"> <span class="n">err</span> <span class="o">=</span> <span class="n">PyList_Append</span><span class="p">(</span><span class="n">list</span><span class="p">,</span> <span class="n">v</span><span class="p">);</span>
5268 </span> <span class="n">Py_DECREF</span><span class="p">(</span><span class="n">v</span><span class="p">);</span>
5269 <span class="k">if</span> <span class="p">(</span><span class="n">err</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">)</span>
5270 <span class="k">goto</span> <span class="n">error</span><span class="p">;</span>
5271 <span class="n">PREDICT</span><span class="p">(</span><span class="n">JUMP_ABSOLUTE</span><span class="p">);</span>
5272 <span class="n">DISPATCH</span><span class="p">();</span>
5273 <span class="p">}</span>
5274 <span class="p">...</span>
5275 </pre></div>
5276
5277 <div class="alert alert-primary" role="alert">
5278 <p><strong>Opcode predictions:</strong>
5279 Some opcodes tend to come in pairs thus making it possible to predict the second code when the first is run. For example, <code>COMPARE_OP</code> is often followed by <code>POP_JUMP_IF_FALSE</code> or <code>POP_JUMP_IF_TRUE</code>.</p>
5280 <p>“Verifying the prediction costs a single high-speed test of a register variable against a constant. If the pairing was good, then the processor’s own internal branch predication has a high likelihood of success, resulting in a nearly zero-overhead transition to the next opcode. A successful prediction saves a trip through the eval-loop including its unpredictable switch-case branch. Combined with the processor’s internal branch prediction, a successful PREDICT has the effect of making the two opcodes run as if they were a single new opcode with the bodies combined.”</p>
5281 <p>If collecting opcode statistics, you have two choices:</p>
5282 <ol>
5283 <li>Keep the predictions turned-on and interpret the results as if some opcodes had been combined</li>
5284 <li>Turn off predictions so that the opcode frequency counter updates for both opcodes</li>
5285 </ol>
5286 <p>Opcode prediction is disabled with threaded code since the latter allows the CPU to record separate branch prediction information for each opcode.</p>
5287 </div>
5288 <p>Some of the operations, such as <code>CALL_FUNCTION</code>, <code>CALL_METHOD</code>, have an operation argument referencing another compiled function. In these cases, another frame is pushed to the frame stack in the thread, and the evaluation loop is run for that function until the function completes. Each time a new frame is created and pushed onto the stack, the value of the frame’s <code>f_back</code> is set to the current frame before the new one is created.</p>
5289 <p>This nesting of frames is clear when you see a stack trace, take this example script:</p>
5290 <div class="highlight python"><pre><span></span><span class="k">def</span> <span class="nf">function2</span><span class="p">():</span>
5291 <span class="k">raise</span> <span class="ne">RuntimeError</span>
5292
5293 <span class="k">def</span> <span class="nf">function1</span><span class="p">():</span>
5294 <span class="n">function2</span><span class="p">()</span>
5295
5296 <span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
5297 <span class="n">function1</span><span class="p">()</span>
5298 </pre></div>
5299
5300 <p>Calling this on the command line will give you:</p>
5301 <div class="highlight sh"><pre><span></span><span class="gp">$</span> ./python.exe example_stack.py
5302
5303 <span class="go">Traceback (most recent call last):</span>
5304 <span class="go"> File "example_stack.py", line 8, in <module></span>
5305 <span class="go"> function1()</span>
5306 <span class="go"> File "example_stack.py", line 5, in function1</span>
5307 <span class="go"> function2()</span>
5308 <span class="go"> File "example_stack.py", line 2, in function2</span>
5309 <span class="go"> raise RuntimeError</span>
5310 <span class="go">RuntimeError</span>
5311 </pre></div>
5312
5313 <p>In <code>traceback.py</code>, the <code>walk_stack()</code> function used to print trace backs:</p>
5314 <div class="highlight python"><pre><span></span><span class="k">def</span> <span class="nf">walk_stack</span><span class="p">(</span><span class="n">f</span><span class="p">):</span>
5315 <span class="sd">"""Walk a stack yielding the frame and line number for each frame.</span>
5316
5317 <span class="sd"> This will follow f.f_back from the given frame. If no frame is given, the</span>
5318 <span class="sd"> current stack is used. Usually used with StackSummary.extract.</span>
5319 <span class="sd"> """</span>
5320 <span class="k">if</span> <span class="n">f</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
5321 <span class="n">f</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">_getframe</span><span class="p">()</span><span class="o">.</span><span class="n">f_back</span><span class="o">.</span><span class="n">f_back</span>
5322 <span class="k">while</span> <span class="n">f</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
5323 <span class="k">yield</span> <span class="n">f</span><span class="p">,</span> <span class="n">f</span><span class="o">.</span><span class="n">f_lineno</span>
5324 <span class="n">f</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="n">f_back</span>
5325 </pre></div>
5326
5327 <p>Here you can see that the current frame, fetched by calling <code>sys._getframe()</code> and the parent’s parent is set as the frame, because you don’t want to see the call to <code>walk_stack()</code> or <code>print_trace()</code> in the trace back, so those function frames are skipped.</p>
5328 <p>Then the <code>f_back</code> pointer is followed to the top.</p>
5329 <p><code>sys._getframe()</code> is the Python API to get the <code>frame</code> attribute of the current thread.</p>
5330 <p>Here is how that frame stack would look visually, with 3 frames each with its code object and a thread state pointing to the current frame:</p>
5331 <p><a href="https://files.realpython.com/media/Frame_Stack_Example.20728854763c.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/Frame_Stack_Example.20728854763c.png" width="841" height="1556" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Frame_Stack_Example.20728854763c.png&w=210&sig=bb2e7a72a4363554566003b8412f81c843a8edc2 210w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/Frame_Stack_Example.20728854763c.png&w=420&sig=17eddfb3f22ccab1056545af3055470669c4ce3f 420w, https://files.realpython.com/media/Frame_Stack_Example.20728854763c.png 841w" sizes="75vw" alt="Example frame stack"/></a></p>
5332 <h3 id="conclusion_3">Conclusion</h3>
5333 <p>In this Part, you explored the most complex element of CPython: the compiler. The original author of Python, Guido van Rossum, made the statement that CPython’s compiler should be “dumb” so that people can understand it.</p>
5334 <p>By breaking down the compilation process into small, logical steps, it is far easier to understand.</p>
5335 <p>In the next chapter, we connect the compilation process with the basis of all Python code, the <code>object</code>.</p>
5336 <h2 h1="h1" id="part-4-objects-in-cpython">Part 4: Objects in CPython</h2>
5337 <p>CPython comes with a collection of basic types like strings, lists, tuples, dictionaries, and objects.</p>
5338 <p>All of these types are built-in. You don’t need to import any libraries, even from the standard library. Also, the instantiation of these built-in types has some handy shortcuts.</p>
5339 <p>For example, to create a new list, you can call:</p>
5340 <div class="highlight python"><pre><span></span><span class="n">lst</span> <span class="o">=</span> <span class="nb">list</span><span class="p">()</span>
5341 </pre></div>
5342
5343 <p>Or, you can use square brackets:</p>
5344 <div class="highlight python"><pre><span></span><span class="n">lst</span> <span class="o">=</span> <span class="p">[]</span>
5345 </pre></div>
5346
5347 <p>Strings can be instantiated from a string-literal by using either double or single quotes. We explored the grammar definitions earlier that cause the compiler to interpret double quotes as a string literal. </p>
5348 <p>All types in Python inherit from <code>object</code>, a built-in base type. Even strings, tuples, and list inherit from <code>object</code>. During the walk-through of the C code, you have read lots of references to <code>PyObject*</code>, the C-API structure for an <code>object</code>.</p>
5349 <p>Because C is not object-oriented <a href="https://realpython.com/python3-object-oriented-programming/">like Python</a>, objects in C don’t inherit from one another. <code>PyObject</code> is the data structure for the beginning of the Python object’s memory.</p>
5350 <p>Much of the base object API is declared in <code>Objects/object.c</code>, like the function <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Objects/object.c#L505"><code>PyObject_Repr</code></a>, which the built-in <code>repr()</code> function. You will also find <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Objects/object.c#L810"><code>PyObject_Hash()</code></a> and other APIs.</p>
5351 <p>All of these functions can be overridden in a custom object by implementing “dunder” methods on a Python object:</p>
5352 <div class="highlight python"><pre><span></span><span class="k">class</span> <span class="nc">MyObject</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
5353 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
5354 <span class="bp">self</span><span class="o">.</span><span class="n">id</span> <span class="o">=</span> <span class="nb">id</span>
5355 <span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">name</span>
5356
5357 <span class="k">def</span> <span class="nf">__repr__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
5358 <span class="k">return</span> <span class="s2">"<</span><span class="si">{0}</span><span class="s2"> id=</span><span class="si">{1}</span><span class="s2">>"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">id</span><span class="p">)</span>
5359 </pre></div>
5360
5361 <p>This code is implemented in <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Objects/object.c#L505"><code>PyObject_Repr()</code></a>, inside <code>Objects/object.c</code>. The type of the target object, <code>v</code> will be inferred through a call to <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Include/object.h#L122"><code>Py_TYPE()</code></a> and if the <code>tp_repr</code> field is set, then the function pointer is called.
5362 If the <code>tp_repr</code> field is not set, i.e. the object doesn’t declare a custom <code>__repr__</code> method, then the default behavior is run, which is to return <code>"<%s object at %p>"</code> with the type name and the ID:</p>
5363 <div class="highlight c"><pre><span></span><span class="n">PyObject</span> <span class="o">*</span>
5364 <span class="nf">PyObject_Repr</span><span class="p">(</span><span class="n">PyObject</span> <span class="o">*</span><span class="n">v</span><span class="p">)</span>
5365 <span class="p">{</span>
5366 <span class="n">PyObject</span> <span class="o">*</span><span class="n">res</span><span class="p">;</span>
5367 <span class="k">if</span> <span class="p">(</span><span class="n">PyErr_CheckSignals</span><span class="p">())</span>
5368 <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
5369 <span class="p">...</span>
5370 <span class="k">if</span> <span class="p">(</span><span class="n">v</span> <span class="o">==</span> <span class="nb">NULL</span><span class="p">)</span>
5371 <span class="k">return</span> <span class="n">PyUnicode_FromString</span><span class="p">(</span><span class="s">"<NULL>"</span><span class="p">);</span>
5372 <span class="hll"> <span class="k">if</span> <span class="p">(</span><span class="n">Py_TYPE</span><span class="p">(</span><span class="n">v</span><span class="p">)</span><span class="o">-></span><span class="n">tp_repr</span> <span class="o">==</span> <span class="nb">NULL</span><span class="p">)</span>
5373 </span> <span class="k">return</span> <span class="n">PyUnicode_FromFormat</span><span class="p">(</span><span class="s">"<%s object at %p>"</span><span class="p">,</span>
5374 <span class="n">v</span><span class="o">-></span><span class="n">ob_type</span><span class="o">-></span><span class="n">tp_name</span><span class="p">,</span> <span class="n">v</span><span class="p">);</span>
5375
5376 <span class="p">...</span>
5377 <span class="p">}</span>
5378 </pre></div>
5379
5380 <p>The ob_type field for a given <code>PyObject*</code> will point to the data structure <code>PyTypeObject</code>, defined in <code>Include/cpython/object.h</code>.
5381 This data-structure lists all the built-in functions, as fields and the arguments they should receive.</p>
5382 <p>Take <code>tp_repr</code> as an example:</p>
5383 <div class="highlight c"><pre><span></span><span class="k">typedef</span> <span class="k">struct</span> <span class="n">_typeobject</span> <span class="p">{</span>
5384 <span class="n">PyObject_VAR_HEAD</span>
5385 <span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">tp_name</span><span class="p">;</span> <span class="cm">/* For printing, in format "<module>.<name>" */</span>
5386 <span class="n">Py_ssize_t</span> <span class="n">tp_basicsize</span><span class="p">,</span> <span class="n">tp_itemsize</span><span class="p">;</span> <span class="cm">/* For allocation */</span>
5387
5388 <span class="cm">/* Methods to implement standard operations */</span>
5389 <span class="p">...</span>
5390 <span class="hll"> <span class="n">reprfunc</span> <span class="n">tp_repr</span><span class="p">;</span>
5391 </span></pre></div>
5392
5393 <p>Where <code>reprfunc</code> is a <code>typedef</code> for <code>PyObject *(*reprfunc)(PyObject *);</code>, a function that takes 1 pointer to <code>PyObject</code> (<code>self</code>).</p>
5394 <p>Some of the dunder APIs are optional, because they only apply to certain types, like numbers:</p>
5395 <div class="highlight c"><pre><span></span> <span class="cm">/* Method suites for standard classes */</span>
5396
5397 <span class="n">PyNumberMethods</span> <span class="o">*</span><span class="n">tp_as_number</span><span class="p">;</span>
5398 <span class="n">PySequenceMethods</span> <span class="o">*</span><span class="n">tp_as_sequence</span><span class="p">;</span>
5399 <span class="n">PyMappingMethods</span> <span class="o">*</span><span class="n">tp_as_mapping</span><span class="p">;</span>
5400 </pre></div>
5401
5402 <p>A sequence, like a list would implement the following methods:</p>
5403 <div class="highlight c"><pre><span></span><span class="k">typedef</span> <span class="k">struct</span> <span class="p">{</span>
5404 <span class="n">lenfunc</span> <span class="n">sq_length</span><span class="p">;</span> <span class="c1">// len(v)</span>
5405 <span class="n">binaryfunc</span> <span class="n">sq_concat</span><span class="p">;</span> <span class="c1">// v + x</span>
5406 <span class="n">ssizeargfunc</span> <span class="n">sq_repeat</span><span class="p">;</span> <span class="c1">// for x in v</span>
5407 <span class="n">ssizeargfunc</span> <span class="n">sq_item</span><span class="p">;</span> <span class="c1">// v[x]</span>
5408 <span class="kt">void</span> <span class="o">*</span><span class="n">was_sq_slice</span><span class="p">;</span> <span class="c1">// v[x:y:z]</span>
5409 <span class="n">ssizeobjargproc</span> <span class="n">sq_ass_item</span><span class="p">;</span> <span class="c1">// v[x] = z</span>
5410 <span class="kt">void</span> <span class="o">*</span><span class="n">was_sq_ass_slice</span><span class="p">;</span> <span class="c1">// v[x:y] = z</span>
5411 <span class="n">objobjproc</span> <span class="n">sq_contains</span><span class="p">;</span> <span class="c1">// x in v</span>
5412
5413 <span class="n">binaryfunc</span> <span class="n">sq_inplace_concat</span><span class="p">;</span>
5414 <span class="n">ssizeargfunc</span> <span class="n">sq_inplace_repeat</span><span class="p">;</span>
5415 <span class="p">}</span> <span class="n">PySequenceMethods</span><span class="p">;</span>
5416 </pre></div>
5417
5418 <p>All of these built-in functions are called the <a href="https://docs.python.org/3/reference/datamodel.html">Python Data Model</a>. One of the great resources for the Python Data Model is <a href="https://www.oreilly.com/library/view/fluent-python/9781491946237/">“Fluent Python” by Luciano Ramalho</a>.</p>
5419 <h3 id="base-object-type">Base Object Type</h3>
5420 <p>In <code>Objects/object.c</code>, the base implementation of <code>object</code> type is written as pure C code. There are some concrete implementations of basic logic, like shallow comparisons.</p>
5421 <p>Not all methods in a Python object are part of the Data Model, so that a Python object can contain attributes (either class or instance attributes) and methods.</p>
5422 <p>A simple way to think of a Python object is consisting of 2 things:</p>
5423 <ol>
5424 <li>The core data model, with pointers to compiled functions</li>
5425 <li>A dictionary with any custom attributes and methods</li>
5426 </ol>
5427 <p>The core data model is defined in the <code>PyTypeObject</code>, and the functions are defined in:</p>
5428 <ul>
5429 <li><code>Objects/object.c</code> for the built-in methods</li>
5430 <li><code>Objects/boolobject.c</code> for the <code>bool</code> type</li>
5431 <li><code>Objects/bytearrayobject.c</code> for the <code>byte[]</code> type</li>
5432 <li><code>Objects/bytesobjects.c</code> for the <code>bytes</code> type</li>
5433 <li><code>Objects/cellobject.c</code> for the <code>cell</code> type</li>
5434 <li><code>Objects/classobject.c</code> for the abstract <code>class</code> type, used in meta-programming</li>
5435 <li><code>Objects/codeobject.c</code> used for the built-in <code>code</code> object type</li>
5436 <li><code>Objects/complexobject.c</code> for a complex numeric type</li>
5437 <li><code>Objects/iterobject.c</code> for an iterator</li>
5438 <li><code>Objects/listobject.c</code> for the <code>list</code> type</li>
5439 <li><code>Objects/longobject.c</code> for the <code>long</code> numeric type</li>
5440 <li><code>Objects/memoryobject.c</code> for the base memory type</li>
5441 <li><code>Objects/methodobject.c</code> for the class method type</li>
5442 <li><code>Objects/moduleobject.c</code> for a module type</li>
5443 <li><code>Objects/namespaceobject.c</code> for a namespace type</li>
5444 <li><code>Objects/odictobject.c</code> for an ordered dictionary type</li>
5445 <li><code>Objects/rangeobject.c</code> for a range generator</li>
5446 <li><code>Objects/setobject.c</code> for a <code>set</code> type</li>
5447 <li><code>Objects/sliceobject.c</code> for a slice reference type</li>
5448 <li><code>Objects/structseq.c</code> for a <a href="https://docs.python.org/3/library/struct.html#struct.Struct"><code>struct.Struct</code></a> type</li>
5449 <li><code>Objects/tupleobject.c</code> for a <code>tuple</code> type</li>
5450 <li><code>Objects/typeobject.c</code> for a <code>type</code> type</li>
5451 <li><code>Objects/unicodeobject.c</code> for a <code>str</code> type</li>
5452 <li><code>Objects/weakrefobject.c</code> for a <a href="https://docs.python.org/3/library/weakref.html"><code>weakref</code> object</a></li>
5453 </ul>
5454 <p>We’re going to dive into 3 of these types: </p>
5455 <ol>
5456 <li>Booleans </li>
5457 <li>Integers</li>
5458 <li>Generators</li>
5459 </ol>
5460 <p>Booleans and Integers have a lot in common, so we’ll cover those first.</p>
5461 <h3 id="the-bool-and-long-integer-type">The Bool and Long Integer Type</h3>
5462 <p>The <code>bool</code> type is the most straightforward implementation of the built-in types. It inherits from <code>long</code> and has the predefined constants, <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Include/boolobject.h#L22"><code>Py_True</code></a> and <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Include/boolobject.h#L21"><code>Py_False</code></a>. These constants are immutable instances, created on the instantiation of the Python interpreter.</p>
5463 <p>Inside <code>Objects/boolobject.c</code>, you can see the helper function to create a <code>bool</code> instance from a number:</p>
5464 <div class="highlight c"><pre><span></span><span class="n">PyObject</span> <span class="o">*</span><span class="nf">PyBool_FromLong</span><span class="p">(</span><span class="kt">long</span> <span class="n">ok</span><span class="p">)</span>
5465 <span class="p">{</span>
5466 <span class="n">PyObject</span> <span class="o">*</span><span class="n">result</span><span class="p">;</span>
5467
5468 <span class="k">if</span> <span class="p">(</span><span class="n">ok</span><span class="p">)</span>
5469 <span class="n">result</span> <span class="o">=</span> <span class="n">Py_True</span><span class="p">;</span>
5470 <span class="k">else</span>
5471 <span class="n">result</span> <span class="o">=</span> <span class="n">Py_False</span><span class="p">;</span>
5472 <span class="n">Py_INCREF</span><span class="p">(</span><span class="n">result</span><span class="p">);</span>
5473 <span class="k">return</span> <span class="n">result</span><span class="p">;</span>
5474 <span class="p">}</span>
5475 </pre></div>
5476
5477 <p>This function uses the C evaluation of a numeric type to assign <code>Py_True</code> or <code>Py_False</code> to a result and increment the reference counters.</p>
5478 <p>The numeric functions for <code>and</code>, <code>xor</code>, and <code>or</code> are implemented, but addition, subtraction, and division are dereferenced from the base long type since it would make no sense to divide two boolean values.</p>
5479 <p>The implementation of <code>and</code> for a <code>bool</code> value checks if <code>a</code> and <code>b</code> are booleans, then check their references to <code>Py_True</code>, otherwise, are cast as numbers, and the <code>and</code> operation is run on the two numbers:</p>
5480 <div class="highlight c"><pre><span></span><span class="k">static</span> <span class="n">PyObject</span> <span class="o">*</span>
5481 <span class="nf">bool_and</span><span class="p">(</span><span class="n">PyObject</span> <span class="o">*</span><span class="n">a</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">b</span><span class="p">)</span>
5482 <span class="p">{</span>
5483 <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">PyBool_Check</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="o">||</span> <span class="o">!</span><span class="n">PyBool_Check</span><span class="p">(</span><span class="n">b</span><span class="p">))</span>
5484 <span class="k">return</span> <span class="n">PyLong_Type</span><span class="p">.</span><span class="n">tp_as_number</span><span class="o">-></span><span class="n">nb_and</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">);</span>
5485 <span class="k">return</span> <span class="n">PyBool_FromLong</span><span class="p">((</span><span class="n">a</span> <span class="o">==</span> <span class="n">Py_True</span><span class="p">)</span> <span class="o">&</span> <span class="p">(</span><span class="n">b</span> <span class="o">==</span> <span class="n">Py_True</span><span class="p">));</span>
5486 <span class="p">}</span>
5487 </pre></div>
5488
5489 <p>The <code>long</code> type is a bit more complex, as the memory requirements are expansive. In the transition from Python 2 to 3, CPython dropped support for the <code>int</code> type and instead used the <code>long</code> type as the primary integer type. Python’s <code>long</code> type is quite special in that it can store a variable-length number. The maximum length is set in the compiled binary.</p>
5490 <p>The data structure of a Python <code>long</code> consists of the <code>PyObject</code> header and a list of digits. The list of digits, <code>ob_digit</code> is initially set to have one digit, but it later expanded to a longer length when initialized:</p>
5491 <div class="highlight c"><pre><span></span><span class="k">struct</span> <span class="n">_longobject</span> <span class="p">{</span>
5492 <span class="n">PyObject_VAR_HEAD</span>
5493 <span class="n">digit</span> <span class="n">ob_digit</span><span class="p">[</span><span class="mi">1</span><span class="p">];</span>
5494 <span class="p">};</span>
5495 </pre></div>
5496
5497 <p>Memory is allocated to a new <code>long</code> through <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Objects/longobject.c#L262"><code>_PyLong_New()</code></a>. This function takes a fixed length and makes sure it is smaller than <code>MAX_LONG_DIGITS</code>. Then it reallocates the memory for <code>ob_digit</code> to match the length.</p>
5498 <p>To convert a C <code>long</code> type to a Python <code>long</code> type, the <code>long</code> is converted to a list of digits, the memory for the Python <code>long</code> is assigned, and then each of the digits is set.
5499 Because <code>long</code> is initialized with <code>ob_digit</code> already being at a length of 1, if the number is less than 10, then the value is set without the memory being allocated:</p>
5500 <div class="highlight c"><pre><span></span><span class="n">PyObject</span> <span class="o">*</span>
5501 <span class="nf">PyLong_FromLong</span><span class="p">(</span><span class="kt">long</span> <span class="n">ival</span><span class="p">)</span>
5502 <span class="p">{</span>
5503 <span class="n">PyLongObject</span> <span class="o">*</span><span class="n">v</span><span class="p">;</span>
5504 <span class="kt">unsigned</span> <span class="kt">long</span> <span class="n">abs_ival</span><span class="p">;</span>
5505 <span class="kt">unsigned</span> <span class="kt">long</span> <span class="n">t</span><span class="p">;</span> <span class="cm">/* unsigned so >> doesn't propagate sign bit */</span>
5506 <span class="kt">int</span> <span class="n">ndigits</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
5507 <span class="kt">int</span> <span class="n">sign</span><span class="p">;</span>
5508
5509 <span class="n">CHECK_SMALL_INT</span><span class="p">(</span><span class="n">ival</span><span class="p">);</span>
5510 <span class="p">...</span>
5511 <span class="cm">/* Fast path for single-digit ints */</span>
5512 <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="p">(</span><span class="n">abs_ival</span> <span class="o">>></span> <span class="n">PyLong_SHIFT</span><span class="p">))</span> <span class="p">{</span>
5513 <span class="n">v</span> <span class="o">=</span> <span class="n">_PyLong_New</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span>
5514 <span class="k">if</span> <span class="p">(</span><span class="n">v</span><span class="p">)</span> <span class="p">{</span>
5515 <span class="n">Py_SIZE</span><span class="p">(</span><span class="n">v</span><span class="p">)</span> <span class="o">=</span> <span class="n">sign</span><span class="p">;</span>
5516 <span class="n">v</span><span class="o">-></span><span class="n">ob_digit</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="n">Py_SAFE_DOWNCAST</span><span class="p">(</span>
5517 <span class="n">abs_ival</span><span class="p">,</span> <span class="kt">unsigned</span> <span class="kt">long</span><span class="p">,</span> <span class="n">digit</span><span class="p">);</span>
5518 <span class="p">}</span>
5519 <span class="k">return</span> <span class="p">(</span><span class="n">PyObject</span><span class="o">*</span><span class="p">)</span><span class="n">v</span><span class="p">;</span>
5520 <span class="p">}</span>
5521 <span class="p">...</span>
5522 <span class="cm">/* Larger numbers: loop to determine number of digits */</span>
5523 <span class="n">t</span> <span class="o">=</span> <span class="n">abs_ival</span><span class="p">;</span>
5524 <span class="k">while</span> <span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="p">{</span>
5525 <span class="o">++</span><span class="n">ndigits</span><span class="p">;</span>
5526 <span class="n">t</span> <span class="o">>>=</span> <span class="n">PyLong_SHIFT</span><span class="p">;</span>
5527 <span class="p">}</span>
5528 <span class="n">v</span> <span class="o">=</span> <span class="n">_PyLong_New</span><span class="p">(</span><span class="n">ndigits</span><span class="p">);</span>
5529 <span class="k">if</span> <span class="p">(</span><span class="n">v</span> <span class="o">!=</span> <span class="nb">NULL</span><span class="p">)</span> <span class="p">{</span>
5530 <span class="n">digit</span> <span class="o">*</span><span class="n">p</span> <span class="o">=</span> <span class="n">v</span><span class="o">-></span><span class="n">ob_digit</span><span class="p">;</span>
5531 <span class="n">Py_SIZE</span><span class="p">(</span><span class="n">v</span><span class="p">)</span> <span class="o">=</span> <span class="n">ndigits</span><span class="o">*</span><span class="n">sign</span><span class="p">;</span>
5532 <span class="n">t</span> <span class="o">=</span> <span class="n">abs_ival</span><span class="p">;</span>
5533 <span class="k">while</span> <span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="p">{</span>
5534 <span class="o">*</span><span class="n">p</span><span class="o">++</span> <span class="o">=</span> <span class="n">Py_SAFE_DOWNCAST</span><span class="p">(</span>
5535 <span class="n">t</span> <span class="o">&</span> <span class="n">PyLong_MASK</span><span class="p">,</span> <span class="kt">unsigned</span> <span class="kt">long</span><span class="p">,</span> <span class="n">digit</span><span class="p">);</span>
5536 <span class="n">t</span> <span class="o">>>=</span> <span class="n">PyLong_SHIFT</span><span class="p">;</span>
5537 <span class="p">}</span>
5538 <span class="p">}</span>
5539 <span class="k">return</span> <span class="p">(</span><span class="n">PyObject</span> <span class="o">*</span><span class="p">)</span><span class="n">v</span><span class="p">;</span>
5540 <span class="p">}</span>
5541 </pre></div>
5542
5543 <p>To convert a <a href="https://en.wikipedia.org/wiki/Double-precision_floating-point_format">double-point floating point</a> to a Python <code>long</code>, <code>PyLong_FromDouble()</code> does the math for you:</p>
5544 <div class="highlight c"><pre><span></span><span class="n">PyObject</span> <span class="o">*</span>
5545 <span class="nf">PyLong_FromDouble</span><span class="p">(</span><span class="kt">double</span> <span class="n">dval</span><span class="p">)</span>
5546 <span class="p">{</span>
5547 <span class="n">PyLongObject</span> <span class="o">*</span><span class="n">v</span><span class="p">;</span>
5548 <span class="kt">double</span> <span class="n">frac</span><span class="p">;</span>
5549 <span class="kt">int</span> <span class="n">i</span><span class="p">,</span> <span class="n">ndig</span><span class="p">,</span> <span class="n">expo</span><span class="p">,</span> <span class="n">neg</span><span class="p">;</span>
5550 <span class="n">neg</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
5551 <span class="k">if</span> <span class="p">(</span><span class="n">Py_IS_INFINITY</span><span class="p">(</span><span class="n">dval</span><span class="p">))</span> <span class="p">{</span>
5552 <span class="n">PyErr_SetString</span><span class="p">(</span><span class="n">PyExc_OverflowError</span><span class="p">,</span>
5553 <span class="s">"cannot convert float infinity to integer"</span><span class="p">);</span>
5554 <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
5555 <span class="p">}</span>
5556 <span class="k">if</span> <span class="p">(</span><span class="n">Py_IS_NAN</span><span class="p">(</span><span class="n">dval</span><span class="p">))</span> <span class="p">{</span>
5557 <span class="n">PyErr_SetString</span><span class="p">(</span><span class="n">PyExc_ValueError</span><span class="p">,</span>
5558 <span class="s">"cannot convert float NaN to integer"</span><span class="p">);</span>
5559 <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
5560 <span class="p">}</span>
5561 <span class="k">if</span> <span class="p">(</span><span class="n">dval</span> <span class="o"><</span> <span class="mf">0.0</span><span class="p">)</span> <span class="p">{</span>
5562 <span class="n">neg</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
5563 <span class="n">dval</span> <span class="o">=</span> <span class="o">-</span><span class="n">dval</span><span class="p">;</span>
5564 <span class="p">}</span>
5565 <span class="n">frac</span> <span class="o">=</span> <span class="n">frexp</span><span class="p">(</span><span class="n">dval</span><span class="p">,</span> <span class="o">&</span><span class="n">expo</span><span class="p">);</span> <span class="cm">/* dval = frac*2**expo; 0.0 <= frac < 1.0 */</span>
5566 <span class="k">if</span> <span class="p">(</span><span class="n">expo</span> <span class="o"><=</span> <span class="mi">0</span><span class="p">)</span>
5567 <span class="k">return</span> <span class="n">PyLong_FromLong</span><span class="p">(</span><span class="mi">0L</span><span class="p">);</span>
5568 <span class="n">ndig</span> <span class="o">=</span> <span class="p">(</span><span class="n">expo</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="o">/</span> <span class="n">PyLong_SHIFT</span> <span class="o">+</span> <span class="mi">1</span><span class="p">;</span> <span class="cm">/* Number of 'digits' in result */</span>
5569 <span class="n">v</span> <span class="o">=</span> <span class="n">_PyLong_New</span><span class="p">(</span><span class="n">ndig</span><span class="p">);</span>
5570 <span class="k">if</span> <span class="p">(</span><span class="n">v</span> <span class="o">==</span> <span class="nb">NULL</span><span class="p">)</span>
5571 <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
5572 <span class="n">frac</span> <span class="o">=</span> <span class="n">ldexp</span><span class="p">(</span><span class="n">frac</span><span class="p">,</span> <span class="p">(</span><span class="n">expo</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="o">%</span> <span class="n">PyLong_SHIFT</span> <span class="o">+</span> <span class="mi">1</span><span class="p">);</span>
5573 <span class="k">for</span> <span class="p">(</span><span class="n">i</span> <span class="o">=</span> <span class="n">ndig</span><span class="p">;</span> <span class="o">--</span><span class="n">i</span> <span class="o">>=</span> <span class="mi">0</span><span class="p">;</span> <span class="p">)</span> <span class="p">{</span>
5574 <span class="n">digit</span> <span class="n">bits</span> <span class="o">=</span> <span class="p">(</span><span class="n">digit</span><span class="p">)</span><span class="n">frac</span><span class="p">;</span>
5575 <span class="n">v</span><span class="o">-></span><span class="n">ob_digit</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="n">bits</span><span class="p">;</span>
5576 <span class="n">frac</span> <span class="o">=</span> <span class="n">frac</span> <span class="o">-</span> <span class="p">(</span><span class="kt">double</span><span class="p">)</span><span class="n">bits</span><span class="p">;</span>
5577 <span class="n">frac</span> <span class="o">=</span> <span class="n">ldexp</span><span class="p">(</span><span class="n">frac</span><span class="p">,</span> <span class="n">PyLong_SHIFT</span><span class="p">);</span>
5578 <span class="p">}</span>
5579 <span class="k">if</span> <span class="p">(</span><span class="n">neg</span><span class="p">)</span>
5580 <span class="n">Py_SIZE</span><span class="p">(</span><span class="n">v</span><span class="p">)</span> <span class="o">=</span> <span class="o">-</span><span class="p">(</span><span class="n">Py_SIZE</span><span class="p">(</span><span class="n">v</span><span class="p">));</span>
5581 <span class="k">return</span> <span class="p">(</span><span class="n">PyObject</span> <span class="o">*</span><span class="p">)</span><span class="n">v</span><span class="p">;</span>
5582 <span class="p">}</span>
5583 </pre></div>
5584
5585 <p>The remainder of the implementation functions in <code>longobject.c</code> have utilities, such as converting a Unicode string into a number with <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Objects/longobject.c#L2672"><code>PyLong_FromUnicodeObject()</code></a>.</p>
5586 <h3 id="a-review-of-the-generator-type">A Review of the Generator Type</h3>
5587 <p><a href="https://realpython.com/introduction-to-python-generators/">Python Generators</a> are functions which return a <code>yield</code> statement and can be called continually to generate further values.</p>
5588 <p>Commonly they are used as a more memory efficient way of looping through values in a large block of data, like a file, a database or over a network. </p>
5589 <p>Generator objects are returned in place of a value when <code>yield</code> is used instead of <code>return</code>. The generator object is created from the <code>yield</code> statement and returned to the caller.</p>
5590 <p>Let’s create a simple generator with a list of 4 constant values:</p>
5591 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">def</span> <span class="nf">example</span><span class="p">():</span>
5592 <span class="gp">... </span> <span class="n">lst</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">]</span>
5593 <span class="gp">... </span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">lst</span><span class="p">:</span>
5594 <span class="gp">... </span> <span class="k">yield</span> <span class="n">i</span>
5595 <span class="gp">... </span>
5596 <span class="gp">>>> </span><span class="n">gen</span> <span class="o">=</span> <span class="n">example</span><span class="p">()</span>
5597 <span class="gp">>>> </span><span class="n">gen</span>
5598 <span class="go"><generator object example at 0x100bcc480></span>
5599 </pre></div>
5600
5601 <p>If you explore the contents of the generator object, you can see some of the fields starting with <code>gi_</code>:</p>
5602 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="nb">dir</span><span class="p">(</span><span class="n">gen</span><span class="p">)</span>
5603 <span class="go">[ ...</span>
5604 <span class="go"> 'close', </span>
5605 <span class="go"> 'gi_code', </span>
5606 <span class="go"> 'gi_frame', </span>
5607 <span class="go"> 'gi_running', </span>
5608 <span class="go"> 'gi_yieldfrom', </span>
5609 <span class="go"> 'send', </span>
5610 <span class="go"> 'throw']</span>
5611 </pre></div>
5612
5613 <p>The <code>PyGenObject</code> type is defined in <code>Include/genobject.h</code> and there are 3 flavors:</p>
5614 <ol>
5615 <li>Generator objects</li>
5616 <li>Coroutine objects</li>
5617 <li>Async generator objects</li>
5618 </ol>
5619 <p>All 3 share the same subset of fields used in generators, and have similar behaviors:</p>
5620 <p><a href="https://files.realpython.com/media/generators.536b1404195a.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/generators.536b1404195a.png" width="612" height="264" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/generators.536b1404195a.png&w=153&sig=665718d32fbb9081b7983085eaeb6de7437ed6b6 153w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/generators.536b1404195a.png&w=306&sig=05fff378d8474cff821fbd585c9a57c5c5c51283 306w, https://files.realpython.com/media/generators.536b1404195a.png 612w" sizes="75vw" alt="Structure of generator types"/></a></p>
5621 <p>Focusing first on generators, you can see the fields:</p>
5622 <ul>
5623 <li><code>gi_frame</code> linking to a <code>PyFrameObject</code> for the generator, earlier in the execution chapter, we explored the use of locals and globals inside a frame’s value stack. This is how generators remember the last value of local variables since the frame is persistent between calls</li>
5624 <li><code>gi_running</code> set to 0 or 1 if the generator is currently running</li>
5625 <li><code>gi_code</code> linking to a <code>PyCodeObject</code> with the compiled function that yielded the generator so that it can be called again</li>
5626 <li><code>gi_weakreflist</code> linking to a list of weak references to objects inside the generator function</li>
5627 <li><code>gi_name</code> as the name of the generator</li>
5628 <li><code>gi_qualname</code> as the qualified name of the generator</li>
5629 <li><code>gi_exc_state</code> as a tuple of exception data if the generator call raises an exception</li>
5630 </ul>
5631 <p>The coroutine and <a href="https://realpython.com/async-io-python/#other-features-async-for-and-async-generators-comprehensions">async generators</a> have the same fields but prepended with <code>cr</code> and <code>ag</code> respectively.</p>
5632 <p>If you call <code>__next__()</code> on the generator object, the next value is yielded until eventually a <code>StopIteration</code> is raised:</p>
5633 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">gen</span><span class="o">.</span><span class="fm">__next__</span><span class="p">()</span>
5634 <span class="go">1</span>
5635 <span class="gp">>>> </span><span class="n">gen</span><span class="o">.</span><span class="fm">__next__</span><span class="p">()</span>
5636 <span class="go">2</span>
5637 <span class="gp">>>> </span><span class="n">gen</span><span class="o">.</span><span class="fm">__next__</span><span class="p">()</span>
5638 <span class="go">3</span>
5639 <span class="gp">>>> </span><span class="n">gen</span><span class="o">.</span><span class="fm">__next__</span><span class="p">()</span>
5640 <span class="go">4</span>
5641 <span class="gp">>>> </span><span class="n">gen</span><span class="o">.</span><span class="fm">__next__</span><span class="p">()</span>
5642 <span class="gt">Traceback (most recent call last):</span>
5643 File <span class="nb">"<stdin>"</span>, line <span class="m">1</span>, in <span class="n"><module></span>
5644 <span class="gr">StopIteration</span>
5645 </pre></div>
5646
5647 <p>Each time <code>__next__()</code> is called, the code object inside the generators <code>gi_code</code> field is executed as a new frame and the return value is pushed to the value stack.</p>
5648 <p>You can also see that <code>gi_code</code> is the compiled code object for the generator function by importing the <code>dis</code> module and disassembling the bytecode inside:</p>
5649 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">gen</span> <span class="o">=</span> <span class="n">example</span><span class="p">()</span>
5650 <span class="gp">>>> </span><span class="kn">import</span> <span class="nn">dis</span>
5651 <span class="gp">>>> </span><span class="n">dis</span><span class="o">.</span><span class="n">disco</span><span class="p">(</span><span class="n">gen</span><span class="o">.</span><span class="n">gi_code</span><span class="p">)</span>
5652 <span class="go"> 2 0 LOAD_CONST 1 (1)</span>
5653 <span class="go"> 2 LOAD_CONST 2 (2)</span>
5654 <span class="go"> 4 LOAD_CONST 3 (3)</span>
5655 <span class="go"> 6 LOAD_CONST 4 (4)</span>
5656 <span class="go"> 8 BUILD_LIST 4</span>
5657 <span class="go"> 10 STORE_FAST 0 (l)</span>
5658
5659 <span class="go"> 3 12 SETUP_LOOP 18 (to 32)</span>
5660 <span class="go"> 14 LOAD_FAST 0 (l)</span>
5661 <span class="go"> 16 GET_ITER</span>
5662 <span class="go"> >> 18 FOR_ITER 10 (to 30)</span>
5663 <span class="go"> 20 STORE_FAST 1 (i)</span>
5664
5665 <span class="go"> 4 22 LOAD_FAST 1 (i)</span>
5666 <span class="go"> 24 YIELD_VALUE</span>
5667 <span class="go"> 26 POP_TOP</span>
5668 <span class="go"> 28 JUMP_ABSOLUTE 18</span>
5669 <span class="go"> >> 30 POP_BLOCK</span>
5670 <span class="go"> >> 32 LOAD_CONST 0 (None)</span>
5671 <span class="go"> 34 RETURN_VALUE</span>
5672 </pre></div>
5673
5674 <p>Whenever <code>__next__()</code> is called on a generator object, <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Objects/genobject.c#L541"><code>gen_iternext()</code></a> is called with the generator instance, which immediately calls <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Objects/genobject.c#L153"><code>gen_send_ex()</code></a> inside <code>Objects/genobject.c</code>.</p>
5675 <p><code>gen_send_ex()</code> is the function that converts a generator object into the next yielded result. You’ll see many similarities with the way frames are constructed in <code>Python/ceval.c</code> from a code object as these functions have similar tasks.</p>
5676 <p>The <code>gen_send_ex()</code> function is shared with generators, coroutines, and async generators and has the following steps:</p>
5677 <ol>
5678 <li>
5679 <p>The current thread state is fetched</p>
5680 </li>
5681 <li>
5682 <p>The frame object from the generator object is fetched</p>
5683 </li>
5684 <li>
5685 <p>If the generator is running when <code>__next__()</code> was called, raise a <code>ValueError</code></p>
5686 </li>
5687 <li>
5688 <p>If the frame inside the generator is at the top of the stack:</p>
5689 <ul>
5690 <li>In the case of a coroutine, if the coroutine is not already marked as closing, a <code>RuntimeError</code> is raised</li>
5691 <li>If this is an async generator, raise a <code>StopAsyncIteration</code></li>
5692 <li>For a standard generator, a <code>StopIteration</code> is raised.</li>
5693 </ul>
5694 </li>
5695 <li>
5696 <p>If the last instruction in the frame (<code>f->f_lasti</code>) is still -1 because it has just been started, and this is a coroutine or async generator, then a non-None value can’t be passed as an argument, so an exception is raised</p>
5697 </li>
5698 <li>
5699 <p>Else, this is the first time it’s being called, and arguments are allowed. The value of the argument is pushed to the frame’s value stack</p>
5700 </li>
5701 <li>
5702 <p>The <code>f_back</code> field of the frame is the caller to which return values are sent, so this is set to the current frame in the thread. This means that the return value is sent to the caller, not the creator of the generator</p>
5703 </li>
5704 <li>
5705 <p>The generator is marked as running</p>
5706 </li>
5707 <li>
5708 <p>The last exception in the generator’s exception info is copied from the last exception in the thread state</p>
5709 </li>
5710 <li>
5711 <p>The thread state exception info is set to the address of the generator’s exception info. This means that if the caller enters a breakpoint around the execution of a generator, the stack trace goes through the generator and the offending code is clear</p>
5712 </li>
5713 <li>
5714 <p>The frame inside the generator is executed within the <code>Python/ceval.c</code> main execution loop, and the value returned</p>
5715 </li>
5716 <li>
5717 <p>The thread state last exception is reset to the value before the frame was called</p>
5718 </li>
5719 <li>
5720 <p>The generator is marked as not running</p>
5721 </li>
5722 <li>
5723 <p>The following cases then match the return value and any exceptions thrown by the call to the generator. Remember that generators should raise a <code>StopIteration</code> when they are exhausted, either manually, or by not yielding a value. Coroutines and async generators should not:</p>
5724 <ul>
5725 <li>If no result was returned from the frame, raise a <code>StopIteration</code> for generators and <code>StopAsyncIteration</code> for async generators</li>
5726 <li>If a <code>StopIteration</code> was explicitly raised, but this is a coroutine or an async generator, raise a <code>RuntimeError</code> as this is not allowed</li>
5727 <li>If a <code>StopAsyncIteration</code> was explicitly raised and this is an async generator, raise a <code>RuntimeError</code>, as this is not allowed</li>
5728 </ul>
5729 </li>
5730 <li>
5731 <p>Lastly, the result is returned back to the caller of <code>__next__()</code></p>
5732 </li>
5733 </ol>
5734 <div class="highlight c"><pre><span></span><span class="k">static</span> <span class="n">PyObject</span> <span class="o">*</span>
5735 <span class="nf">gen_send_ex</span><span class="p">(</span><span class="n">PyGenObject</span> <span class="o">*</span><span class="n">gen</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">arg</span><span class="p">,</span> <span class="kt">int</span> <span class="n">exc</span><span class="p">,</span> <span class="kt">int</span> <span class="n">closing</span><span class="p">)</span>
5736 <span class="p">{</span>
5737 <span class="hll"> <span class="n">PyThreadState</span> <span class="o">*</span><span class="n">tstate</span> <span class="o">=</span> <span class="n">_PyThreadState_GET</span><span class="p">();</span> <span class="c1">// 1.</span>
5738 </span><span class="hll"> <span class="n">PyFrameObject</span> <span class="o">*</span><span class="n">f</span> <span class="o">=</span> <span class="n">gen</span><span class="o">-></span><span class="n">gi_frame</span><span class="p">;</span> <span class="c1">// 2.</span>
5739 </span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">result</span><span class="p">;</span>
5740
5741 <span class="hll"> <span class="k">if</span> <span class="p">(</span><span class="n">gen</span><span class="o">-></span><span class="n">gi_running</span><span class="p">)</span> <span class="p">{</span> <span class="c1">// 3.</span>
5742 </span> <span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">msg</span> <span class="o">=</span> <span class="s">"generator already executing"</span><span class="p">;</span>
5743 <span class="k">if</span> <span class="p">(</span><span class="n">PyCoro_CheckExact</span><span class="p">(</span><span class="n">gen</span><span class="p">))</span> <span class="p">{</span>
5744 <span class="n">msg</span> <span class="o">=</span> <span class="s">"coroutine already executing"</span><span class="p">;</span>
5745 <span class="p">}</span>
5746 <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="n">PyAsyncGen_CheckExact</span><span class="p">(</span><span class="n">gen</span><span class="p">))</span> <span class="p">{</span>
5747 <span class="n">msg</span> <span class="o">=</span> <span class="s">"async generator already executing"</span><span class="p">;</span>
5748 <span class="p">}</span>
5749 <span class="n">PyErr_SetString</span><span class="p">(</span><span class="n">PyExc_ValueError</span><span class="p">,</span> <span class="n">msg</span><span class="p">);</span>
5750 <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
5751 <span class="p">}</span>
5752 <span class="hll"> <span class="k">if</span> <span class="p">(</span><span class="n">f</span> <span class="o">==</span> <span class="nb">NULL</span> <span class="o">||</span> <span class="n">f</span><span class="o">-></span><span class="n">f_stacktop</span> <span class="o">==</span> <span class="nb">NULL</span><span class="p">)</span> <span class="p">{</span> <span class="c1">// 4.</span>
5753 </span> <span class="k">if</span> <span class="p">(</span><span class="n">PyCoro_CheckExact</span><span class="p">(</span><span class="n">gen</span><span class="p">)</span> <span class="o">&&</span> <span class="o">!</span><span class="n">closing</span><span class="p">)</span> <span class="p">{</span>
5754 <span class="cm">/* `gen` is an exhausted coroutine: raise an error,</span>
5755 <span class="cm"> except when called from gen_close(), which should</span>
5756 <span class="cm"> always be a silent method. */</span>
5757 <span class="hll"> <span class="n">PyErr_SetString</span><span class="p">(</span>
5758 </span><span class="hll"> <span class="n">PyExc_RuntimeError</span><span class="p">,</span>
5759 </span><span class="hll"> <span class="s">"cannot reuse already awaited coroutine"</span><span class="p">);</span> <span class="c1">// 4a.</span>
5760 </span> <span class="p">}</span>
5761 <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="n">arg</span> <span class="o">&&</span> <span class="o">!</span><span class="n">exc</span><span class="p">)</span> <span class="p">{</span>
5762 <span class="cm">/* `gen` is an exhausted generator:</span>
5763 <span class="cm"> only set exception if called from send(). */</span>
5764 <span class="k">if</span> <span class="p">(</span><span class="n">PyAsyncGen_CheckExact</span><span class="p">(</span><span class="n">gen</span><span class="p">))</span> <span class="p">{</span>
5765 <span class="hll"> <span class="n">PyErr_SetNone</span><span class="p">(</span><span class="n">PyExc_StopAsyncIteration</span><span class="p">);</span> <span class="c1">// 4b.</span>
5766 </span> <span class="p">}</span>
5767 <span class="k">else</span> <span class="p">{</span>
5768 <span class="hll"> <span class="n">PyErr_SetNone</span><span class="p">(</span><span class="n">PyExc_StopIteration</span><span class="p">);</span> <span class="c1">// 4c.</span>
5769 </span> <span class="p">}</span>
5770 <span class="p">}</span>
5771 <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
5772 <span class="p">}</span>
5773
5774 <span class="k">if</span> <span class="p">(</span><span class="n">f</span><span class="o">-></span><span class="n">f_lasti</span> <span class="o">==</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
5775 <span class="hll"> <span class="k">if</span> <span class="p">(</span><span class="n">arg</span> <span class="o">&&</span> <span class="n">arg</span> <span class="o">!=</span> <span class="n">Py_None</span><span class="p">)</span> <span class="p">{</span> <span class="c1">// 5.</span>
5776 </span> <span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">msg</span> <span class="o">=</span> <span class="s">"can't send non-None value to a "</span>
5777 <span class="s">"just-started generator"</span><span class="p">;</span>
5778 <span class="k">if</span> <span class="p">(</span><span class="n">PyCoro_CheckExact</span><span class="p">(</span><span class="n">gen</span><span class="p">))</span> <span class="p">{</span>
5779 <span class="n">msg</span> <span class="o">=</span> <span class="n">NON_INIT_CORO_MSG</span><span class="p">;</span>
5780 <span class="p">}</span>
5781 <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="n">PyAsyncGen_CheckExact</span><span class="p">(</span><span class="n">gen</span><span class="p">))</span> <span class="p">{</span>
5782 <span class="n">msg</span> <span class="o">=</span> <span class="s">"can't send non-None value to a "</span>
5783 <span class="s">"just-started async generator"</span><span class="p">;</span>
5784 <span class="p">}</span>
5785 <span class="n">PyErr_SetString</span><span class="p">(</span><span class="n">PyExc_TypeError</span><span class="p">,</span> <span class="n">msg</span><span class="p">);</span>
5786 <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
5787 <span class="p">}</span>
5788 <span class="hll"> <span class="p">}</span> <span class="k">else</span> <span class="p">{</span> <span class="c1">// 6.</span>
5789 </span> <span class="cm">/* Push arg onto the frame's value stack */</span>
5790 <span class="n">result</span> <span class="o">=</span> <span class="n">arg</span> <span class="o">?</span> <span class="nl">arg</span> <span class="p">:</span> <span class="n">Py_None</span><span class="p">;</span>
5791 <span class="n">Py_INCREF</span><span class="p">(</span><span class="n">result</span><span class="p">);</span>
5792 <span class="o">*</span><span class="p">(</span><span class="n">f</span><span class="o">-></span><span class="n">f_stacktop</span><span class="o">++</span><span class="p">)</span> <span class="o">=</span> <span class="n">result</span><span class="p">;</span>
5793 <span class="p">}</span>
5794
5795 <span class="cm">/* Generators always return to their most recent caller, not</span>
5796 <span class="cm"> * necessarily their creator. */</span>
5797 <span class="n">Py_XINCREF</span><span class="p">(</span><span class="n">tstate</span><span class="o">-></span><span class="n">frame</span><span class="p">);</span>
5798 <span class="n">assert</span><span class="p">(</span><span class="n">f</span><span class="o">-></span><span class="n">f_back</span> <span class="o">==</span> <span class="nb">NULL</span><span class="p">);</span>
5799 <span class="hll"> <span class="n">f</span><span class="o">-></span><span class="n">f_back</span> <span class="o">=</span> <span class="n">tstate</span><span class="o">-></span><span class="n">frame</span><span class="p">;</span> <span class="c1">// 7.</span>
5800 </span>
5801 <span class="hll"> <span class="n">gen</span><span class="o">-></span><span class="n">gi_running</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span> <span class="c1">// 8.</span>
5802 </span><span class="hll"> <span class="n">gen</span><span class="o">-></span><span class="n">gi_exc_state</span><span class="p">.</span><span class="n">previous_item</span> <span class="o">=</span> <span class="n">tstate</span><span class="o">-></span><span class="n">exc_info</span><span class="p">;</span> <span class="c1">// 9.</span>
5803 </span><span class="hll"> <span class="n">tstate</span><span class="o">-></span><span class="n">exc_info</span> <span class="o">=</span> <span class="o">&</span><span class="n">gen</span><span class="o">-></span><span class="n">gi_exc_state</span><span class="p">;</span> <span class="c1">// 10.</span>
5804 </span><span class="hll"> <span class="n">result</span> <span class="o">=</span> <span class="n">PyEval_EvalFrameEx</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="n">exc</span><span class="p">);</span> <span class="c1">// 11.</span>
5805 </span><span class="hll"> <span class="n">tstate</span><span class="o">-></span><span class="n">exc_info</span> <span class="o">=</span> <span class="n">gen</span><span class="o">-></span><span class="n">gi_exc_state</span><span class="p">.</span><span class="n">previous_item</span><span class="p">;</span> <span class="c1">// 12.</span>
5806 </span> <span class="n">gen</span><span class="o">-></span><span class="n">gi_exc_state</span><span class="p">.</span><span class="n">previous_item</span> <span class="o">=</span> <span class="nb">NULL</span><span class="p">;</span>
5807 <span class="hll"> <span class="n">gen</span><span class="o">-></span><span class="n">gi_running</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="c1">// 13.</span>
5808 </span>
5809 <span class="cm">/* Don't keep the reference to f_back any longer than necessary. It</span>
5810 <span class="cm"> * may keep a chain of frames alive or it could create a reference</span>
5811 <span class="cm"> * cycle. */</span>
5812 <span class="n">assert</span><span class="p">(</span><span class="n">f</span><span class="o">-></span><span class="n">f_back</span> <span class="o">==</span> <span class="n">tstate</span><span class="o">-></span><span class="n">frame</span><span class="p">);</span>
5813 <span class="n">Py_CLEAR</span><span class="p">(</span><span class="n">f</span><span class="o">-></span><span class="n">f_back</span><span class="p">);</span>
5814
5815 <span class="cm">/* If the generator just returned (as opposed to yielding), signal</span>
5816 <span class="cm"> * that the generator is exhausted. */</span>
5817 <span class="hll"> <span class="k">if</span> <span class="p">(</span><span class="n">result</span> <span class="o">&&</span> <span class="n">f</span><span class="o">-></span><span class="n">f_stacktop</span> <span class="o">==</span> <span class="nb">NULL</span><span class="p">)</span> <span class="p">{</span> <span class="c1">// 14a.</span>
5818 </span> <span class="k">if</span> <span class="p">(</span><span class="n">result</span> <span class="o">==</span> <span class="n">Py_None</span><span class="p">)</span> <span class="p">{</span>
5819 <span class="cm">/* Delay exception instantiation if we can */</span>
5820 <span class="k">if</span> <span class="p">(</span><span class="n">PyAsyncGen_CheckExact</span><span class="p">(</span><span class="n">gen</span><span class="p">))</span> <span class="p">{</span>
5821 <span class="n">PyErr_SetNone</span><span class="p">(</span><span class="n">PyExc_StopAsyncIteration</span><span class="p">);</span>
5822 <span class="p">}</span>
5823 <span class="k">else</span> <span class="p">{</span>
5824 <span class="n">PyErr_SetNone</span><span class="p">(</span><span class="n">PyExc_StopIteration</span><span class="p">);</span>
5825 <span class="p">}</span>
5826 <span class="p">}</span>
5827 <span class="k">else</span> <span class="p">{</span>
5828 <span class="cm">/* Async generators cannot return anything but None */</span>
5829 <span class="n">assert</span><span class="p">(</span><span class="o">!</span><span class="n">PyAsyncGen_CheckExact</span><span class="p">(</span><span class="n">gen</span><span class="p">));</span>
5830 <span class="n">_PyGen_SetStopIterationValue</span><span class="p">(</span><span class="n">result</span><span class="p">);</span>
5831 <span class="p">}</span>
5832 <span class="n">Py_CLEAR</span><span class="p">(</span><span class="n">result</span><span class="p">);</span>
5833 <span class="p">}</span>
5834 <span class="hll"> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">result</span> <span class="o">&&</span> <span class="n">PyErr_ExceptionMatches</span><span class="p">(</span><span class="n">PyExc_StopIteration</span><span class="p">))</span> <span class="p">{</span> <span class="c1">// 14b.</span>
5835 </span> <span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">msg</span> <span class="o">=</span> <span class="s">"generator raised StopIteration"</span><span class="p">;</span>
5836 <span class="k">if</span> <span class="p">(</span><span class="n">PyCoro_CheckExact</span><span class="p">(</span><span class="n">gen</span><span class="p">))</span> <span class="p">{</span>
5837 <span class="n">msg</span> <span class="o">=</span> <span class="s">"coroutine raised StopIteration"</span><span class="p">;</span>
5838 <span class="p">}</span>
5839 <span class="k">else</span> <span class="k">if</span> <span class="n">PyAsyncGen_CheckExact</span><span class="p">(</span><span class="n">gen</span><span class="p">)</span> <span class="p">{</span>
5840 <span class="n">msg</span> <span class="o">=</span> <span class="s">"async generator raised StopIteration"</span><span class="p">;</span>
5841 <span class="p">}</span>
5842 <span class="n">_PyErr_FormatFromCause</span><span class="p">(</span><span class="n">PyExc_RuntimeError</span><span class="p">,</span> <span class="s">"%s"</span><span class="p">,</span> <span class="n">msg</span><span class="p">);</span>
5843
5844 <span class="p">}</span>
5845 <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">result</span> <span class="o">&&</span> <span class="n">PyAsyncGen_CheckExact</span><span class="p">(</span><span class="n">gen</span><span class="p">)</span> <span class="o">&&</span>
5846 <span class="hll"> <span class="n">PyErr_ExceptionMatches</span><span class="p">(</span><span class="n">PyExc_StopAsyncIteration</span><span class="p">))</span> <span class="c1">// 14c.</span>
5847 </span> <span class="p">{</span>
5848 <span class="cm">/* code in `gen` raised a StopAsyncIteration error:</span>
5849 <span class="cm"> raise a RuntimeError.</span>
5850 <span class="cm"> */</span>
5851 <span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">msg</span> <span class="o">=</span> <span class="s">"async generator raised StopAsyncIteration"</span><span class="p">;</span>
5852 <span class="n">_PyErr_FormatFromCause</span><span class="p">(</span><span class="n">PyExc_RuntimeError</span><span class="p">,</span> <span class="s">"%s"</span><span class="p">,</span> <span class="n">msg</span><span class="p">);</span>
5853 <span class="p">}</span>
5854 <span class="p">...</span>
5855
5856 <span class="hll"> <span class="k">return</span> <span class="n">result</span><span class="p">;</span> <span class="c1">// 15.</span>
5857 </span><span class="p">}</span>
5858 </pre></div>
5859
5860 <p>Going back to the evaluation of code objects whenever a function or module is called, there was a special case for generators, coroutines, and async generators in <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/ceval.c#L4045"><code>_PyEval_EvalCodeWithName()</code></a>. This function checks for the <code>CO_GENERATOR</code>, <code>CO_COROUTINE</code>, and <code>CO_ASYNC_GENERATOR</code> flags on the code object.</p>
5861 <p>When a new coroutine is created using <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Objects/genobject.c#L1152"><code>PyCoro_New()</code></a>, a new async generator is created with <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Objects/genobject.c#L1428"><code>PyAsyncGen_New()</code></a> or a generator with <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Objects/genobject.c#L811"><code>PyGen_NewWithQualName()</code></a>. These objects are returned early instead of returning an evaluated frame, which is why you get a generator object after calling a function with a yield statement:</p>
5862 <div class="highlight c"><pre><span></span><span class="n">PyObject</span> <span class="o">*</span>
5863 <span class="nf">_PyEval_EvalCodeWithName</span><span class="p">(</span><span class="n">PyObject</span> <span class="o">*</span><span class="n">_co</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">globals</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">locals</span><span class="p">,</span> <span class="p">...</span>
5864 <span class="p">...</span>
5865 <span class="cm">/* Handle generator/coroutine/asynchronous generator */</span>
5866 <span class="k">if</span> <span class="p">(</span><span class="n">co</span><span class="o">-></span><span class="n">co_flags</span> <span class="o">&</span> <span class="p">(</span><span class="n">CO_GENERATOR</span> <span class="o">|</span> <span class="n">CO_COROUTINE</span> <span class="o">|</span> <span class="n">CO_ASYNC_GENERATOR</span><span class="p">))</span> <span class="p">{</span>
5867 <span class="n">PyObject</span> <span class="o">*</span><span class="n">gen</span><span class="p">;</span>
5868 <span class="n">PyObject</span> <span class="o">*</span><span class="n">coro_wrapper</span> <span class="o">=</span> <span class="n">tstate</span><span class="o">-></span><span class="n">coroutine_wrapper</span><span class="p">;</span>
5869 <span class="kt">int</span> <span class="n">is_coro</span> <span class="o">=</span> <span class="n">co</span><span class="o">-></span><span class="n">co_flags</span> <span class="o">&</span> <span class="n">CO_COROUTINE</span><span class="p">;</span>
5870 <span class="p">...</span>
5871 <span class="cm">/* Create a new generator that owns the ready to run frame</span>
5872 <span class="cm"> * and return that as the value. */</span>
5873 <span class="k">if</span> <span class="p">(</span><span class="n">is_coro</span><span class="p">)</span> <span class="p">{</span>
5874 <span class="n">gen</span> <span class="o">=</span> <span class="n">PyCoro_New</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">qualname</span><span class="p">);</span>
5875 <span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="n">co</span><span class="o">-></span><span class="n">co_flags</span> <span class="o">&</span> <span class="n">CO_ASYNC_GENERATOR</span><span class="p">)</span> <span class="p">{</span>
5876 <span class="n">gen</span> <span class="o">=</span> <span class="n">PyAsyncGen_New</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">qualname</span><span class="p">);</span>
5877 <span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
5878 <span class="n">gen</span> <span class="o">=</span> <span class="n">PyGen_NewWithQualName</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">qualname</span><span class="p">);</span>
5879 <span class="p">}</span>
5880 <span class="p">...</span>
5881 <span class="k">return</span> <span class="n">gen</span><span class="p">;</span>
5882 <span class="p">}</span>
5883 <span class="p">...</span>
5884 </pre></div>
5885
5886 <p>The flags in the code object were injected by the compiler after traversing the AST and seeing the <code>yield</code> or <code>yield from</code> statements or seeing the <code>coroutine</code> decorator.</p>
5887 <p><code>PyGen_NewWithQualName()</code> will call <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Objects/genobject.c#L778"><code>gen_new_with_qualname()</code></a> with the generated frame and then create the <code>PyGenObject</code> with <code>NULL</code> values and the compiled code object:</p>
5888 <div class="highlight c"><pre><span></span><span class="k">static</span> <span class="n">PyObject</span> <span class="o">*</span>
5889 <span class="nf">gen_new_with_qualname</span><span class="p">(</span><span class="n">PyTypeObject</span> <span class="o">*</span><span class="n">type</span><span class="p">,</span> <span class="n">PyFrameObject</span> <span class="o">*</span><span class="n">f</span><span class="p">,</span>
5890 <span class="n">PyObject</span> <span class="o">*</span><span class="n">name</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">qualname</span><span class="p">)</span>
5891 <span class="p">{</span>
5892 <span class="n">PyGenObject</span> <span class="o">*</span><span class="n">gen</span> <span class="o">=</span> <span class="n">PyObject_GC_New</span><span class="p">(</span><span class="n">PyGenObject</span><span class="p">,</span> <span class="n">type</span><span class="p">);</span>
5893 <span class="k">if</span> <span class="p">(</span><span class="n">gen</span> <span class="o">==</span> <span class="nb">NULL</span><span class="p">)</span> <span class="p">{</span>
5894 <span class="n">Py_DECREF</span><span class="p">(</span><span class="n">f</span><span class="p">);</span>
5895 <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
5896 <span class="p">}</span>
5897 <span class="n">gen</span><span class="o">-></span><span class="n">gi_frame</span> <span class="o">=</span> <span class="n">f</span><span class="p">;</span>
5898 <span class="n">f</span><span class="o">-></span><span class="n">f_gen</span> <span class="o">=</span> <span class="p">(</span><span class="n">PyObject</span> <span class="o">*</span><span class="p">)</span> <span class="n">gen</span><span class="p">;</span>
5899 <span class="n">Py_INCREF</span><span class="p">(</span><span class="n">f</span><span class="o">-></span><span class="n">f_code</span><span class="p">);</span>
5900 <span class="n">gen</span><span class="o">-></span><span class="n">gi_code</span> <span class="o">=</span> <span class="p">(</span><span class="n">PyObject</span> <span class="o">*</span><span class="p">)(</span><span class="n">f</span><span class="o">-></span><span class="n">f_code</span><span class="p">);</span>
5901 <span class="n">gen</span><span class="o">-></span><span class="n">gi_running</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
5902 <span class="n">gen</span><span class="o">-></span><span class="n">gi_weakreflist</span> <span class="o">=</span> <span class="nb">NULL</span><span class="p">;</span>
5903 <span class="n">gen</span><span class="o">-></span><span class="n">gi_exc_state</span><span class="p">.</span><span class="n">exc_type</span> <span class="o">=</span> <span class="nb">NULL</span><span class="p">;</span>
5904 <span class="n">gen</span><span class="o">-></span><span class="n">gi_exc_state</span><span class="p">.</span><span class="n">exc_value</span> <span class="o">=</span> <span class="nb">NULL</span><span class="p">;</span>
5905 <span class="n">gen</span><span class="o">-></span><span class="n">gi_exc_state</span><span class="p">.</span><span class="n">exc_traceback</span> <span class="o">=</span> <span class="nb">NULL</span><span class="p">;</span>
5906 <span class="n">gen</span><span class="o">-></span><span class="n">gi_exc_state</span><span class="p">.</span><span class="n">previous_item</span> <span class="o">=</span> <span class="nb">NULL</span><span class="p">;</span>
5907 <span class="k">if</span> <span class="p">(</span><span class="n">name</span> <span class="o">!=</span> <span class="nb">NULL</span><span class="p">)</span>
5908 <span class="n">gen</span><span class="o">-></span><span class="n">gi_name</span> <span class="o">=</span> <span class="n">name</span><span class="p">;</span>
5909 <span class="k">else</span>
5910 <span class="n">gen</span><span class="o">-></span><span class="n">gi_name</span> <span class="o">=</span> <span class="p">((</span><span class="n">PyCodeObject</span> <span class="o">*</span><span class="p">)</span><span class="n">gen</span><span class="o">-></span><span class="n">gi_code</span><span class="p">)</span><span class="o">-></span><span class="n">co_name</span><span class="p">;</span>
5911 <span class="n">Py_INCREF</span><span class="p">(</span><span class="n">gen</span><span class="o">-></span><span class="n">gi_name</span><span class="p">);</span>
5912 <span class="k">if</span> <span class="p">(</span><span class="n">qualname</span> <span class="o">!=</span> <span class="nb">NULL</span><span class="p">)</span>
5913 <span class="n">gen</span><span class="o">-></span><span class="n">gi_qualname</span> <span class="o">=</span> <span class="n">qualname</span><span class="p">;</span>
5914 <span class="k">else</span>
5915 <span class="n">gen</span><span class="o">-></span><span class="n">gi_qualname</span> <span class="o">=</span> <span class="n">gen</span><span class="o">-></span><span class="n">gi_name</span><span class="p">;</span>
5916 <span class="n">Py_INCREF</span><span class="p">(</span><span class="n">gen</span><span class="o">-></span><span class="n">gi_qualname</span><span class="p">);</span>
5917 <span class="n">_PyObject_GC_TRACK</span><span class="p">(</span><span class="n">gen</span><span class="p">);</span>
5918 <span class="k">return</span> <span class="p">(</span><span class="n">PyObject</span> <span class="o">*</span><span class="p">)</span><span class="n">gen</span><span class="p">;</span>
5919 <span class="p">}</span>
5920 </pre></div>
5921
5922 <p>Bringing this all together you can see how the generator expression is a powerful syntax where a single keyword, <code>yield</code> triggers a whole flow to create a unique object, copy a compiled code object as a property, set a frame, and store a list of variables in the local scope.</p>
5923 <p>To the user of the generator expression, this all seems like magic, but under the covers it’s not <em>that</em> complex.</p>
5924 <h3 id="conclusion_4">Conclusion</h3>
5925 <p>Now that you understand how some built-in types, you can explore other types. </p>
5926 <p>When exploring Python classes, it is important to remember there are built-in types, written in C and classes inheriting from those types, written in Python or C.</p>
5927 <p>Some libraries have types written in C instead of inheriting from the built-in types. One example is <code>numpy</code>, a library for numeric arrays. The <a href="https://docs.scipy.org/doc/numpy/reference/generated/numpy.array.html"><code>nparray</code></a> type is written in C, is highly efficient and performant.</p>
5928 <p>In the next Part, we will explore the classes and functions defined in the standard library.</p>
5929 <h2 h1="h1" id="part-5-the-cpython-standard-library">Part 5: The CPython Standard Library</h2>
5930 <p>Python has always come “batteries included.” This statement means that with a standard CPython distribution, there are libraries for working with files, threads, networks, web sites, music, keyboards, screens, text, and a whole manner of utilities.</p>
5931 <p>Some of the batteries that come with CPython are more like AA batteries. They’re useful for everything, like the <code>collections</code> module and the <code>sys</code> module. Some of them are a bit more obscure, like a small watch battery that you never know when it might come in useful.</p>
5932 <p>There are 2 types of modules in the CPython standard library:</p>
5933 <ol>
5934 <li>Those written in pure Python that provides a utility</li>
5935 <li>Those written in C with Python wrappers</li>
5936 </ol>
5937 <p>We will explore both types.</p>
5938 <h3 id="python-modules">Python Modules</h3>
5939 <p>The modules written in pure Python are all located in the <code>Lib/</code> directory in the source code. Some of the larger modules have submodules in subfolders, like the <code>email</code> module.</p>
5940 <p>An easy module to look at would be the <code>colorsys</code> module. It’s only a few hundred lines of Python code. You may not have come across it before. The <code>colorsys</code> module has some utility functions for converting color scales.</p>
5941 <p>When you install a Python distribution from source, standard library modules are copied from the <code>Lib</code> folder into the distribution folder. This folder is always part of your path when you start Python, so you can <code>import</code> the modules without having to worry about where they’re located. </p>
5942 <p>For example:</p>
5943 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">colorsys</span>
5944 <span class="gp">>>> </span><span class="n">colorsys</span>
5945 <span class="go"><module 'colorsys' from '/usr/shared/lib/python3.7/colorsys.py'></span>
5946
5947 <span class="gp">>>> </span><span class="n">colorsys</span><span class="o">.</span><span class="n">rgb_to_hls</span><span class="p">(</span><span class="mi">255</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">)</span>
5948 <span class="go">(0.0, 127.5, -1.007905138339921) </span>
5949 </pre></div>
5950
5951 <p>We can see the source code of <code>rgb_to_hls()</code> inside <code>Lib/colorsys.py</code>:</p>
5952 <div class="highlight python"><pre><span></span><span class="c1"># HLS: Hue, Luminance, Saturation</span>
5953 <span class="c1"># H: position in the spectrum</span>
5954 <span class="c1"># L: color lightness</span>
5955 <span class="c1"># S: color saturation</span>
5956
5957 <span class="k">def</span> <span class="nf">rgb_to_hls</span><span class="p">(</span><span class="n">r</span><span class="p">,</span> <span class="n">g</span><span class="p">,</span> <span class="n">b</span><span class="p">):</span>
5958 <span class="n">maxc</span> <span class="o">=</span> <span class="nb">max</span><span class="p">(</span><span class="n">r</span><span class="p">,</span> <span class="n">g</span><span class="p">,</span> <span class="n">b</span><span class="p">)</span>
5959 <span class="n">minc</span> <span class="o">=</span> <span class="nb">min</span><span class="p">(</span><span class="n">r</span><span class="p">,</span> <span class="n">g</span><span class="p">,</span> <span class="n">b</span><span class="p">)</span>
5960 <span class="c1"># XXX Can optimize (maxc+minc) and (maxc-minc)</span>
5961 <span class="n">l</span> <span class="o">=</span> <span class="p">(</span><span class="n">minc</span><span class="o">+</span><span class="n">maxc</span><span class="p">)</span><span class="o">/</span><span class="mf">2.0</span>
5962 <span class="k">if</span> <span class="n">minc</span> <span class="o">==</span> <span class="n">maxc</span><span class="p">:</span>
5963 <span class="k">return</span> <span class="mf">0.0</span><span class="p">,</span> <span class="n">l</span><span class="p">,</span> <span class="mf">0.0</span>
5964 <span class="k">if</span> <span class="n">l</span> <span class="o"><=</span> <span class="mf">0.5</span><span class="p">:</span>
5965 <span class="n">s</span> <span class="o">=</span> <span class="p">(</span><span class="n">maxc</span><span class="o">-</span><span class="n">minc</span><span class="p">)</span> <span class="o">/</span> <span class="p">(</span><span class="n">maxc</span><span class="o">+</span><span class="n">minc</span><span class="p">)</span>
5966 <span class="k">else</span><span class="p">:</span>
5967 <span class="n">s</span> <span class="o">=</span> <span class="p">(</span><span class="n">maxc</span><span class="o">-</span><span class="n">minc</span><span class="p">)</span> <span class="o">/</span> <span class="p">(</span><span class="mf">2.0</span><span class="o">-</span><span class="n">maxc</span><span class="o">-</span><span class="n">minc</span><span class="p">)</span>
5968 <span class="n">rc</span> <span class="o">=</span> <span class="p">(</span><span class="n">maxc</span><span class="o">-</span><span class="n">r</span><span class="p">)</span> <span class="o">/</span> <span class="p">(</span><span class="n">maxc</span><span class="o">-</span><span class="n">minc</span><span class="p">)</span>
5969 <span class="n">gc</span> <span class="o">=</span> <span class="p">(</span><span class="n">maxc</span><span class="o">-</span><span class="n">g</span><span class="p">)</span> <span class="o">/</span> <span class="p">(</span><span class="n">maxc</span><span class="o">-</span><span class="n">minc</span><span class="p">)</span>
5970 <span class="n">bc</span> <span class="o">=</span> <span class="p">(</span><span class="n">maxc</span><span class="o">-</span><span class="n">b</span><span class="p">)</span> <span class="o">/</span> <span class="p">(</span><span class="n">maxc</span><span class="o">-</span><span class="n">minc</span><span class="p">)</span>
5971 <span class="k">if</span> <span class="n">r</span> <span class="o">==</span> <span class="n">maxc</span><span class="p">:</span>
5972 <span class="n">h</span> <span class="o">=</span> <span class="n">bc</span><span class="o">-</span><span class="n">gc</span>
5973 <span class="k">elif</span> <span class="n">g</span> <span class="o">==</span> <span class="n">maxc</span><span class="p">:</span>
5974 <span class="n">h</span> <span class="o">=</span> <span class="mf">2.0</span><span class="o">+</span><span class="n">rc</span><span class="o">-</span><span class="n">bc</span>
5975 <span class="k">else</span><span class="p">:</span>
5976 <span class="n">h</span> <span class="o">=</span> <span class="mf">4.0</span><span class="o">+</span><span class="n">gc</span><span class="o">-</span><span class="n">rc</span>
5977 <span class="n">h</span> <span class="o">=</span> <span class="p">(</span><span class="n">h</span><span class="o">/</span><span class="mf">6.0</span><span class="p">)</span> <span class="o">%</span> <span class="mf">1.0</span>
5978 <span class="k">return</span> <span class="n">h</span><span class="p">,</span> <span class="n">l</span><span class="p">,</span> <span class="n">s</span>
5979 </pre></div>
5980
5981 <p>There’s nothing special about this function, it’s just standard Python. You’ll find similar things with all of the pure Python standard library modules. They’re just written in plain Python, well laid out and easy to understand. You may even spot improvements or bugs, so you can make changes to them and contribute it to the Python distribution. We’ll cover that toward the end of this article.</p>
5982 <h3 id="python-and-c-modules">Python and C Modules</h3>
5983 <p>The remainder of modules are written in C, or a combination or Python and C. The source code for these is in <code>Lib/</code> for the Python component, and <code>Modules/</code> for the C component. There are two exceptions to this rule, the <code>sys</code> module, found in <code>Python/sysmodule.c</code> and the <code>__builtins__</code> module, found in <code>Python/bltinmodule.c</code>.</p>
5984 <p>Python will <code>import * from __builtins__</code> when an interpreter is instantiated, so all of the functions like <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/bltinmodule.c#L1821"><code>print()</code></a>, <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/bltinmodule.c#L688"><code>chr()</code></a>, <a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Python/bltinmodule.c#L672"><code>format()</code></a>, etc. are found within <code>Python/bltinmodule.c</code>.</p>
5985 <p>Because the <code>sys</code> module is so specific to the interpreter and the internals of CPython, that is found inside the <code>Python</code> directly. It is also marked as an “implementation detail” of CPython and not found in other distributions.</p>
5986 <p>The built-in <code>print()</code> function was probably the first thing you learned to do in Python. So what happens when you type <code>print("hello world!")</code>?</p>
5987 <ol>
5988 <li>The argument <code>"hello world"</code> was converted from a string constant to a <code>PyUnicodeObject</code> by the compiler</li>
5989 <li><code>builtin_print()</code> was executed with 1 argument, and NULL <code>kwnames</code></li>
5990 <li>The <code>file</code> variable is set to <code>PyId_stdout</code>, the system’s <code>stdout</code> handle</li>
5991 <li>Each argument is sent to <code>file</code></li>
5992 <li>A line break, <code>\n</code> is sent to <code>file</code></li>
5993 </ol>
5994 <div class="highlight c"><pre><span></span><span class="k">static</span> <span class="n">PyObject</span> <span class="o">*</span>
5995 <span class="nf">builtin_print</span><span class="p">(</span><span class="n">PyObject</span> <span class="o">*</span><span class="n">self</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="k">const</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="n">Py_ssize_t</span> <span class="n">nargs</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">kwnames</span><span class="p">)</span>
5996 <span class="p">{</span>
5997 <span class="p">...</span>
5998 <span class="k">if</span> <span class="p">(</span><span class="n">file</span> <span class="o">==</span> <span class="nb">NULL</span> <span class="o">||</span> <span class="n">file</span> <span class="o">==</span> <span class="n">Py_None</span><span class="p">)</span> <span class="p">{</span>
5999 <span class="hll"> <span class="n">file</span> <span class="o">=</span> <span class="n">_PySys_GetObjectId</span><span class="p">(</span><span class="o">&</span><span class="n">PyId_stdout</span><span class="p">);</span>
6000 </span> <span class="p">...</span>
6001 <span class="p">}</span>
6002 <span class="p">...</span>
6003 <span class="k">for</span> <span class="p">(</span><span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">nargs</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
6004 <span class="k">if</span> <span class="p">(</span><span class="n">i</span> <span class="o">></span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
6005 <span class="k">if</span> <span class="p">(</span><span class="n">sep</span> <span class="o">==</span> <span class="nb">NULL</span><span class="p">)</span>
6006 <span class="hll"> <span class="n">err</span> <span class="o">=</span> <span class="n">PyFile_WriteString</span><span class="p">(</span><span class="s">" "</span><span class="p">,</span> <span class="n">file</span><span class="p">);</span>
6007 </span> <span class="k">else</span>
6008 <span class="n">err</span> <span class="o">=</span> <span class="n">PyFile_WriteObject</span><span class="p">(</span><span class="n">sep</span><span class="p">,</span> <span class="n">file</span><span class="p">,</span>
6009 <span class="n">Py_PRINT_RAW</span><span class="p">);</span>
6010 <span class="k">if</span> <span class="p">(</span><span class="n">err</span><span class="p">)</span>
6011 <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
6012 <span class="p">}</span>
6013 <span class="n">err</span> <span class="o">=</span> <span class="n">PyFile_WriteObject</span><span class="p">(</span><span class="n">args</span><span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="n">file</span><span class="p">,</span> <span class="n">Py_PRINT_RAW</span><span class="p">);</span>
6014 <span class="k">if</span> <span class="p">(</span><span class="n">err</span><span class="p">)</span>
6015 <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
6016 <span class="p">}</span>
6017
6018 <span class="k">if</span> <span class="p">(</span><span class="n">end</span> <span class="o">==</span> <span class="nb">NULL</span><span class="p">)</span>
6019 <span class="hll"> <span class="n">err</span> <span class="o">=</span> <span class="n">PyFile_WriteString</span><span class="p">(</span><span class="s">"</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">file</span><span class="p">);</span>
6020 </span> <span class="k">else</span>
6021 <span class="n">err</span> <span class="o">=</span> <span class="n">PyFile_WriteObject</span><span class="p">(</span><span class="n">end</span><span class="p">,</span> <span class="n">file</span><span class="p">,</span> <span class="n">Py_PRINT_RAW</span><span class="p">);</span>
6022 <span class="p">...</span>
6023 <span class="n">Py_RETURN_NONE</span><span class="p">;</span>
6024 <span class="p">}</span>
6025 </pre></div>
6026
6027 <p>The contents of some modules written in C expose operating system functions. Because the CPython source code needs to compile to macOS, Windows, Linux, and other *nix-based operating systems, there are some special cases.</p>
6028 <p>The <code>time</code> module is a good example. The way that Windows keeps and stores time in the Operating System is fundamentally different than Linux and macOS. This is one of the reasons why the accuracy of the clock functions differs <a href="https://docs.python.org/3/library/time.html#time.clock_gettime_ns">between operating systems</a>.</p>
6029 <p>In <code>Modules/timemodule.c</code>, the operating system time functions for Unix-based systems are imported from <code><sys/times.h></code>:</p>
6030 <div class="highlight c"><pre><span></span><span class="cp">#ifdef HAVE_SYS_TIMES_H</span>
6031 <span class="cp">#include</span> <span class="cpf"><sys/times.h></span><span class="cp"></span>
6032 <span class="cp">#endif</span>
6033 <span class="p">...</span>
6034 <span class="cp">#ifdef MS_WINDOWS</span>
6035 <span class="cp">#define WIN32_LEAN_AND_MEAN</span>
6036 <span class="cp">#include</span> <span class="cpf"><windows.h></span><span class="cp"></span>
6037 <span class="cp">#include</span> <span class="cpf">"pythread.h"</span><span class="cp"></span>
6038 <span class="cp">#endif </span><span class="cm">/* MS_WINDOWS */</span><span class="cp"></span>
6039 <span class="p">...</span>
6040 </pre></div>
6041
6042 <p>Later in the file, <code>time_process_time_ns()</code> is defined as a wrapper for <code>_PyTime_GetProcessTimeWithInfo()</code>:</p>
6043 <div class="highlight c"><pre><span></span><span class="k">static</span> <span class="n">PyObject</span> <span class="o">*</span>
6044 <span class="nf">time_process_time_ns</span><span class="p">(</span><span class="n">PyObject</span> <span class="o">*</span><span class="n">self</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">unused</span><span class="p">)</span>
6045 <span class="p">{</span>
6046 <span class="n">_PyTime_t</span> <span class="n">t</span><span class="p">;</span>
6047 <span class="k">if</span> <span class="p">(</span><span class="n">_PyTime_GetProcessTimeWithInfo</span><span class="p">(</span><span class="o">&</span><span class="n">t</span><span class="p">,</span> <span class="nb">NULL</span><span class="p">)</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
6048 <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
6049 <span class="p">}</span>
6050 <span class="k">return</span> <span class="n">_PyTime_AsNanosecondsObject</span><span class="p">(</span><span class="n">t</span><span class="p">);</span>
6051 <span class="p">}</span>
6052 </pre></div>
6053
6054 <p><a href="https://github.com/python/cpython/blob/d93605de7232da5e6a182fd1d5c220639e900159/Modules/timemodule.c#L1120"><code>_PyTime_GetProcessTimeWithInfo()</code></a> is implemented multiple different ways in the source code, but only certain parts are compiled into the binary for the module, depending on the operating system. Windows systems will call <code>GetProcessTimes()</code> and Unix systems will call <code>clock_gettime()</code>.</p>
6055 <p>Other modules that have multiple implementations for the same API are <a href="https://realpython.com/intro-to-python-threading/">the threading module</a>, the file system module, and the networking modules. Because the Operating Systems behave differently, the CPython source code implements the same behavior as best as it can and exposes it using a consistent, abstracted API.</p>
6056 <h3 id="the-cpython-regression-test-suite">The CPython Regression Test Suite</h3>
6057 <p>CPython has a robust and extensive test suite covering the core interpreter, the standard library, the tooling and distribution for both Windows and Linux/macOS.</p>
6058 <p>The test suite is located in <code>Lib/test</code> and written almost entirely in Python.</p>
6059 <p>The full test suite is a Python package, so can be run using the Python interpreter that you’ve compiled. Change directory to the <code>Lib</code> directory and run <code>python -m test -j2</code>, where <code>j2</code> means to use 2 CPUs.</p>
6060 <p>On Windows use the <code>rt.bat</code> script inside the PCBuild folder, ensuring that you have built the <strong>Release</strong> configuration from Visual Studio in advance:</p>
6061 <div class="highlight sh"><pre><span></span><span class="gp">$</span> <span class="nb">cd</span> PCbuild
6062 <span class="gp">$</span> rt.bat -q
6063
6064 <span class="go">C:\repos\cpython\PCbuild>"C:\repos\cpython\PCbuild\win32\python.exe" -u -Wd -E -bb -m test</span>
6065 <span class="go">== CPython 3.8.0b4</span>
6066 <span class="go">== Windows-10-10.0.17134-SP0 little-endian</span>
6067 <span class="go">== cwd: C:\repos\cpython\build\test_python_2784</span>
6068 <span class="go">== CPU count: 2</span>
6069 <span class="go">== encodings: locale=cp1252, FS=utf-8</span>
6070 <span class="go">Run tests sequentially</span>
6071 <span class="go">0:00:00 [ 1/420] test_grammar</span>
6072 <span class="go">0:00:00 [ 2/420] test_opcodes</span>
6073 <span class="go">0:00:00 [ 3/420] test_dict</span>
6074 <span class="go">0:00:00 [ 4/420] test_builtin</span>
6075 <span class="go">...</span>
6076 </pre></div>
6077
6078 <p>On Linux:</p>
6079 <div class="highlight sh"><pre><span></span><span class="gp">$</span> <span class="nb">cd</span> Lib
6080 <span class="gp">$</span> ../python -m <span class="nb">test</span> -j2
6081 <span class="go">== CPython 3.8.0b4</span>
6082 <span class="go">== macOS-10.14.3-x86_64-i386-64bit little-endian</span>
6083 <span class="go">== cwd: /Users/anthonyshaw/cpython/build/test_python_23399</span>
6084 <span class="go">== CPU count: 4</span>
6085 <span class="go">== encodings: locale=UTF-8, FS=utf-8</span>
6086 <span class="go">Run tests in parallel using 2 child processes</span>
6087 <span class="go">0:00:00 load avg: 2.14 [ 1/420] test_opcodes passed</span>
6088 <span class="go">0:00:00 load avg: 2.14 [ 2/420] test_grammar passed</span>
6089 <span class="go">...</span>
6090 </pre></div>
6091
6092 <p>On macOS:</p>
6093 <div class="highlight sh"><pre><span></span><span class="gp">$</span> <span class="nb">cd</span> Lib
6094 <span class="gp">$</span> ../python.exe -m <span class="nb">test</span> -j2
6095 <span class="go">== CPython 3.8.0b4</span>
6096 <span class="go">== macOS-10.14.3-x86_64-i386-64bit little-endian</span>
6097 <span class="go">== cwd: /Users/anthonyshaw/cpython/build/test_python_23399</span>
6098 <span class="go">== CPU count: 4</span>
6099 <span class="go">== encodings: locale=UTF-8, FS=utf-8</span>
6100 <span class="go">Run tests in parallel using 2 child processes</span>
6101 <span class="go">0:00:00 load avg: 2.14 [ 1/420] test_opcodes passed</span>
6102 <span class="go">0:00:00 load avg: 2.14 [ 2/420] test_grammar passed</span>
6103 <span class="go">...</span>
6104 </pre></div>
6105
6106 <p>Some tests require certain flags; otherwise they are skipped. For example, many of the IDLE tests require a GUI.</p>
6107 <p>To see a list of test suites in the configuration, use the <code>--list-tests</code> flag:</p>
6108 <div class="highlight sh"><pre><span></span><span class="gp">$</span> ../python.exe -m <span class="nb">test</span> --list-tests
6109
6110 <span class="go">test_grammar</span>
6111 <span class="go">test_opcodes</span>
6112 <span class="go">test_dict</span>
6113 <span class="go">test_builtin</span>
6114 <span class="go">test_exceptions</span>
6115 <span class="go">...</span>
6116 </pre></div>
6117
6118 <p>You can run specific tests by providing the test suite as the first argument:</p>
6119 <div class="highlight sh"><pre><span></span><span class="gp">$</span> ../python.exe -m <span class="nb">test</span> test_webbrowser
6120
6121 <span class="go">Run tests sequentially</span>
6122 <span class="go">0:00:00 load avg: 2.74 [1/1] test_webbrowser</span>
6123
6124 <span class="go">== Tests result: SUCCESS ==</span>
6125
6126 <span class="go">1 test OK.</span>
6127
6128 <span class="go">Total duration: 117 ms</span>
6129 <span class="go">Tests result: SUCCESS</span>
6130 </pre></div>
6131
6132 <p>You can also see a detailed list of tests that were executed with the result using the <code>-v</code> argument:</p>
6133 <div class="highlight sh"><pre><span></span><span class="gp">$</span> ../python.exe -m <span class="nb">test</span> test_webbrowser -v
6134
6135 <span class="go">== CPython 3.8.0b4 </span>
6136 <span class="go">== macOS-10.14.3-x86_64-i386-64bit little-endian</span>
6137 <span class="go">== cwd: /Users/anthonyshaw/cpython/build/test_python_24562</span>
6138 <span class="go">== CPU count: 4</span>
6139 <span class="go">== encodings: locale=UTF-8, FS=utf-8</span>
6140 <span class="go">Run tests sequentially</span>
6141 <span class="go">0:00:00 load avg: 2.36 [1/1] test_webbrowser</span>
6142 <span class="go">test_open (test.test_webbrowser.BackgroundBrowserCommandTest) ... ok</span>
6143 <span class="go">test_register (test.test_webbrowser.BrowserRegistrationTest) ... ok</span>
6144 <span class="go">test_register_default (test.test_webbrowser.BrowserRegistrationTest) ... ok</span>
6145 <span class="go">test_register_preferred (test.test_webbrowser.BrowserRegistrationTest) ... ok</span>
6146 <span class="go">test_open (test.test_webbrowser.ChromeCommandTest) ... ok</span>
6147 <span class="go">test_open_new (test.test_webbrowser.ChromeCommandTest) ... ok</span>
6148 <span class="go">...</span>
6149 <span class="go">test_open_with_autoraise_false (test.test_webbrowser.OperaCommandTest) ... ok</span>
6150
6151 <span class="go">----------------------------------------------------------------------</span>
6152
6153 <span class="go">Ran 34 tests in 0.056s</span>
6154
6155 <span class="go">OK (skipped=2)</span>
6156
6157 <span class="go">== Tests result: SUCCESS ==</span>
6158
6159 <span class="go">1 test OK.</span>
6160
6161 <span class="go">Total duration: 134 ms</span>
6162 <span class="go">Tests result: SUCCESS</span>
6163 </pre></div>
6164
6165 <p>Understanding how to use the test suite and checking the state of the version you have compiled is very important if you wish to make changes to CPython. Before you start making changes, you should run the whole test suite and make sure everything is passing.</p>
6166 <h3 id="installing-a-custom-version">Installing a Custom Version</h3>
6167 <p>From your source repository, if you’re happy with your changes and want to use them inside your system, you can install it as a custom version.</p>
6168 <p>For macOS and Linux, you can use the <code>altinstall</code> command, which won’t create symlinks for <code>python3</code> and install a standalone version:</p>
6169 <div class="highlight sh"><pre><span></span><span class="gp">$</span> make altinstall
6170 </pre></div>
6171
6172 <p>For Windows, you have to change the build configuration from <code>Debug</code> to <code>Release</code>, then copy the packaged binaries to a directory on your computer which is part of the system path.</p>
6173 <h2 id="the-cpython-source-code-conclusion">The CPython Source Code: Conclusion</h2>
6174 <p>Congratulations, you made it! Did your tea get cold? Make yourself another cup. You’ve earned it.</p>
6175 <p>Now that you’ve seen the CPython source code, the modules, the compiler, and the tooling, you may wish to make some changes and contribute them back to the Python ecosystem.</p>
6176 <p>The <a href="https://devguide.python.org/">official dev guide</a> contains plenty of resources for beginners. You’ve already taken the first step, to understand the source, knowing how to change, compile, and test the CPython applications.</p>
6177 <p>Think back to all the things you’ve learned about CPython over this article. All the pieces of magic to which you’ve learned the secrets. The journey doesn’t stop here. </p>
6178 <p>This might be a good time to learn more about Python and C. Who knows: you could be contributing more and more to the CPython project!</p>
6179 <hr />
6180 <p><em>[ Improve Your Python With π Python Tricks π β Get a short & sweet Python Trick delivered to your inbox every couple of days. <a href="https://realpython.com/python-tricks/?utm_source=realpython&utm_medium=rss&utm_campaign=footer">>> Click here to learn more and see examples</a> ]</em></p>#
6181 datePublished: #Wed Aug 21 16:10:00 2019#
6182 dateUpdated: #Wed Aug 21 16:10:00 2019#
6183 # Person begin ####################
6184 name: #Real Python#
6185 # Person end ######################
6186 # Item end ########################
6187 # Item begin ######################
6188 id: #https://realpython.com/courses/python-histograms/#
6189 title: #Python Histogram Plotting: NumPy, Matplotlib, Pandas & Seaborn#
6190 link: #https://realpython.com/courses/python-histograms/#
6191 description: #In this course, you'll be equipped to make production-quality, presentation-ready Python histogram plots with a range of choices and features. It's your one-stop shop for constructing and manipulating histograms with Python's scientific stack.#
6192 content: #<p>In this course, you’ll be equipped to make production-quality, presentation-ready Python histogram plots with a range of choices and features.</p>
6193 <p>If you have introductory to intermediate knowledge in Python and statistics, then you can use this article as a one-stop shop for building and plotting histograms in Python using libraries from its scientific stack, including NumPy, Matplotlib, Pandas, and Seaborn.</p>
6194 <p>A histogram is a great tool for quickly assessing a <a href="https://en.wikipedia.org/wiki/Probability_distribution">probability distribution</a> that is intuitively understood by almost any audience. Python offers a handful of different options for building and plotting histograms. Most people know a histogram by its graphical representation, which is similar to a bar graph:</p>
6195 <p><a href="https://files.realpython.com/media/commute_times.621e5b1ce062.png" target="_blank"><img class="img-fluid mx-auto d-block w-75" src="https://files.realpython.com/media/commute_times.621e5b1ce062.png" width="1152" height="888" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/commute_times.621e5b1ce062.png&w=288&sig=408f56b07d4fb71d47171405f51e3cb58a7e6cc2 288w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/commute_times.621e5b1ce062.png&w=576&sig=a2b0324c81a7cd89fe7ceaacdcd4ae797ad1a587 576w, https://files.realpython.com/media/commute_times.621e5b1ce062.png 1152w" sizes="75vw" alt="Histogram of commute times for 1000 commuters"/></a></p>
6196 <p>This course will guide you through creating plots like the one above as well as more complex ones. Here’s what you’ll cover:</p>
6197 <ul>
6198 <li>Building histograms in pure Python, without use of third party libraries</li>
6199 <li>Constructing histograms with NumPy to summarize the underlying data</li>
6200 <li>Plotting the resulting histogram with Matplotlib, Pandas, and Seaborn</li>
6201 </ul>
6202 <div class="alert alert-warning" role="alert"><p><strong>Free Bonus:</strong> Short on time? <a href="https://realpython.com/optins/view/histograms-cheatsheet/" class="alert-link" data-toggle="modal" data-target="#modal-histograms-cheatsheet" data-focus="false">Click here to get access to a free two-page Python histograms cheat sheet</a> that summarizes the techniques explained in this tutorial.</p></div>
6203 <hr />
6204 <p><em>[ Improve Your Python With π Python Tricks π β Get a short & sweet Python Trick delivered to your inbox every couple of days. <a href="https://realpython.com/python-tricks/?utm_source=realpython&utm_medium=rss&utm_campaign=footer">>> Click here to learn more and see examples</a> ]</em></p>#
6205 datePublished: #Tue Aug 20 14:00:00 2019#
6206 dateUpdated: #Tue Aug 20 14:00:00 2019#
6207 # Person begin ####################
6208 name: #Real Python#
6209 # Person end ######################
6210 # Item end ########################
6211 # Item begin ######################
6212 id: #https://realpython.com/how-to-make-a-discord-bot-python/#
6213 title: #How to Make a Discord Bot in Python#
6214 link: #https://realpython.com/how-to-make-a-discord-bot-python/#
6215 description: #In this step-by-step tutorial, you'll learn how to make a Discord bot in Python and interact with several APIs. You'll learn how to handle events, accept commands, validate and verify input, and all the basics that can help you create useful and exciting automations!#
6216 content: #<p>In a world where video games are so important to so many people, communication and community around games are vital. Discord offers both of those and more in one well-designed package. In this tutorial, you’ll learn how to make a Discord bot in Python so that you can make the most of this fantastic platform.</p>
6217 <p><strong>By the end of this article you’ll learn:</strong></p>
6218 <ul>
6219 <li>What Discord is and why it’s so valuable</li>
6220 <li>How to make a Discord bot through the Developer Portal</li>
6221 <li>How to create Discord connections</li>
6222 <li>How to handle events</li>
6223 <li>How to accept commands and validate assumptions</li>
6224 <li>How to interact with various Discord APIs</li>
6225 </ul>
6226 <p>You’ll begin by learning what Discord is and why it’s valuable.</p>
6227 <div class="alert alert-warning" role="alert"><p><strong>Free Bonus:</strong> <a href="" class="alert-link" data-toggle="modal" data-target="#modal-python-tricks-sample" data-focus="false">Click here to get access to a chapter from Python Tricks: The Book</a> that shows you Python's best practices with simple examples you can apply instantly to write more beautiful + Pythonic code.</p></div>
6228
6229 <h2 id="what-is-discord">What Is Discord?</h2>
6230 <p><a href="https://discordapp.com/">Discord</a> is a voice and text communication platform for gamers.</p>
6231 <p>Players, streamers, and developers use Discord to discuss games, answer questions, chat while they play, and much more. It even has a game store, complete with critical reviews and a subscription service. It is nearly a one-stop shop for gaming communities.</p>
6232 <p>While there are many things you can build using Discord’s <a href="https://discordapp.com/developers/docs/intro">APIs</a>, this tutorial will focus on a particular learning outcome: how to make a Discord bot in Python.</p>
6233 <h2 id="what-is-a-bot">What Is a Bot?</h2>
6234 <p>Discord is growing in popularity. As such, automated processes, such as banning inappropriate users and reacting to user requests are vital for a community to thrive and grow.</p>
6235 <p>Automated programs that look and act like users and automatically respond to events and commands on Discord are called <strong>bot users</strong>. Discord bot users (or just <strong>bots</strong>) have nearly <a href="https://discordbots.org">unlimited applications</a>.</p>
6236 <p>For example, let’s say you’re managing a new Discord guild and a user joins for the very first time. Excited, you may personally reach out to that user and welcome them to your community. You might also tell them about your channels or ask them to introduce themselves.</p>
6237 <p>The user feels welcomed and enjoys the discussions that happen in your guild and they, in turn, invite friends.</p>
6238 <p>Over time, your community grows so big that it’s no longer feasible to personally reach out to each new member, but you still want to send them something to recognize them as a new member of the guild.</p>
6239 <p>With a bot, it’s possible to automatically react to the new member joining your guild. You can even customize its behavior based on context and control how it interacts with each new user.</p>
6240 <p>This is great, but it’s only one small example of how a bot can be useful. There are so many opportunities for you to be creative with bots, once you know how to make them.</p>
6241 <div class="alert alert-primary" role="alert">
6242 <p><strong>Note:</strong> Although Discord allows you to create bots that deal with voice communication, this article will stick to the text side of the service.</p>
6243 </div>
6244 <p>There are two key steps when you’re creating a bot:</p>
6245 <ol>
6246 <li>Create the bot user on Discord and register it with a guild.</li>
6247 <li>Write code that uses Discord’s APIs and implements your bot’s behaviors.</li>
6248 </ol>
6249 <p>In the next section, you’ll learn how to make a Discord bot in Discord’s <a href="https://discordapp.com/developers/applications">Developer Portal</a>.</p>
6250 <h2 id="how-to-make-a-discord-bot-in-the-developer-portal">How to Make a Discord Bot in the Developer Portal</h2>
6251 <p>Before you can dive into any Python code to handle events and create exciting automations, you need to first create a few Discord components:</p>
6252 <ol>
6253 <li>An account</li>
6254 <li>An application</li>
6255 <li>A bot</li>
6256 <li>A guild</li>
6257 </ol>
6258 <p>You’ll learn more about each piece in the following sections.</p>
6259 <p>Once you’ve created all of these components, you’ll tie them together by registering your bot with your guild.</p>
6260 <p>You can get started by heading to Discord’s <a href="http://discordapp.com/developers/applications">Developer Portal</a>.</p>
6261 <h3 id="creating-a-discord-account">Creating a Discord Account</h3>
6262 <p>The first thing you’ll see is a landing page where you’ll need to either login, if you have an existing account, or create a new account:</p>
6263 <p><a href="https://files.realpython.com/media/discord-bot-register-user.41a9c2bc4db9.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-register-user.41a9c2bc4db9.png" width="3024" height="1762" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-register-user.41a9c2bc4db9.png&w=756&sig=6d39dcc4f6fc11eb5de6e2b457fc387d04dcb138 756w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-register-user.41a9c2bc4db9.png&w=1512&sig=454c3dfb7830bfefbd4fc267e92af61669936082 1512w, https://files.realpython.com/media/discord-bot-register-user.41a9c2bc4db9.png 3024w" sizes="75vw" alt="Discord: Account Login Screen"/></a></p>
6264 <p>If you need to create a new account, then click on the <em>Register</em> button below <em>Login</em> and enter your account information.</p>
6265 <div class="alert alert-primary" role="alert">
6266 <p><strong>Important:</strong> You’ll need to verify your email before you’re able to move on.</p>
6267 </div>
6268 <p>Once you’re finished, you’ll be redirected to the Developer Portal home page, where you’ll create your application.</p>
6269 <h3 id="creating-an-application">Creating an Application</h3>
6270 <p>An <strong>application</strong> allows you to interact with Discord’s APIs by providing authentication tokens, designating permissions, and so on.</p>
6271 <p>To create a new application, select <em>New Application</em>:</p>
6272 <p><a href="https://files.realpython.com/media/discord-bot-new-app.40b4a51bb57d.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-new-app.40b4a51bb57d.png" width="3024" height="1765" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-new-app.40b4a51bb57d.png&w=756&sig=094b6a170204052aa3e03749c858002b90ccb1bd 756w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-new-app.40b4a51bb57d.png&w=1512&sig=fb20c2f10526eafc64138f33c8e8f08e5c895d57 1512w, https://files.realpython.com/media/discord-bot-new-app.40b4a51bb57d.png 3024w" sizes="75vw" alt="Discord: My Applications Screen"/></a></p>
6273 <p>Next, you’ll be prompted to name your application. Select a name and click <em>Create</em>:</p>
6274 <p><a href="https://files.realpython.com/media/discord-bot-name-application.8ccfc8a69cb5.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-name-application.8ccfc8a69cb5.png" width="3024" height="1771" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-name-application.8ccfc8a69cb5.png&w=756&sig=663b4e94bcf90cb6c34d416805e4a488dfea3a36 756w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-name-application.8ccfc8a69cb5.png&w=1512&sig=46630dceba46441199dfd02f62565bc7be1ca896 1512w, https://files.realpython.com/media/discord-bot-name-application.8ccfc8a69cb5.png 3024w" sizes="75vw" alt="Discord: Naming an Application"/></a></p>
6275 <p>Congratulations! You made a Discord application. On the resulting screen, you can see information about your application:</p>
6276 <p><a href="https://files.realpython.com/media/discord-bot-app-info.146a24d590a6.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-app-info.146a24d590a6.png" width="3015" height="1767" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-app-info.146a24d590a6.png&w=753&sig=08d7b87d0cdda73c1ce9f040123d5fbe6b3ecc76 753w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-app-info.146a24d590a6.png&w=1507&sig=8d9fdeb253e6d562be88406244262b8797e50525 1507w, https://files.realpython.com/media/discord-bot-app-info.146a24d590a6.png 3015w" sizes="75vw" alt="Discord: Application General Information"/></a></p>
6277 <p>Keep in mind that any program that interacts with Discord APIs requires a Discord application, not just bots. Bot-related APIs are only a subset of Discord’s total interface.</p>
6278 <p>However, since this tutorial is about how to make a Discord bot, navigate to the <em>Bot</em> tab on the left-hand navigation list.</p>
6279 <h3 id="creating-a-bot">Creating a Bot</h3>
6280 <p>As you learned in the previous sections, a bot user is one that listens to and automatically reacts to certain events and commands on Discord.</p>
6281 <p>For your code to actually be manifested on Discord, you’ll need to create a bot user. To do so, select <em>Add Bot</em>:</p>
6282 <p><a href="https://files.realpython.com/media/discord-bot-add-bot.4735c88ff16b.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-add-bot.4735c88ff16b.png" width="3021" height="1761" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-add-bot.4735c88ff16b.png&w=755&sig=1bd131ee29f9c52aed1698d09c26f4267abb5f02 755w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-add-bot.4735c88ff16b.png&w=1510&sig=3b9afe030a68a934e1f3730ebab7c9bd7e8981cf 1510w, https://files.realpython.com/media/discord-bot-add-bot.4735c88ff16b.png 3021w" sizes="75vw" alt="Discord: Add Bot"/></a></p>
6283 <p>Once you confirm that you want to add the bot to your application, you’ll see the new bot user in the portal:</p>
6284 <p><a href="https://files.realpython.com/media/discord-bot-created.fbdf4a021810.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-created.fbdf4a021810.png" width="3009" height="1760" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-created.fbdf4a021810.png&w=752&sig=8d9dc28ff753dff981137aef1b7e5add00a429a1 752w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-created.fbdf4a021810.png&w=1504&sig=2ec8e17f1a229eb212a96d7f85c9b680391776f1 1504w, https://files.realpython.com/media/discord-bot-created.fbdf4a021810.png 3009w" sizes="75vw" alt="Discord: Bot Created Successfully"/></a></p>
6285 <p>Notice that, by default, your bot user will inherit the name of your application. Instead, update the username to something more bot-like, such as <code>RealPythonTutorialBot</code>, and <em>Save Changes</em>:</p>
6286 <p><a href="https://files.realpython.com/media/discord-bot-rename-bot.008fd6ed6354.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-rename-bot.008fd6ed6354.png" width="3023" height="1770" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-rename-bot.008fd6ed6354.png&w=755&sig=39e7af71df9ef2095bce4d7e964973af010903c0 755w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-rename-bot.008fd6ed6354.png&w=1511&sig=252ca9f5542ba888771dd8aa32cf36b65cbc6317 1511w, https://files.realpython.com/media/discord-bot-rename-bot.008fd6ed6354.png 3023w" sizes="75vw" alt="Discord: Rename Bot"/></a></p>
6287 <p>Now, the bot’s all set and ready to go, but to where?</p>
6288 <p>A bot user is not useful if it’s not interacting with other users. Next, you’ll create a guild so that your bot can interact with other users.</p>
6289 <h3 id="creating-a-guild">Creating a Guild</h3>
6290 <p>A <strong>guild</strong> (or a <strong>server</strong>, as it is often called in Discord’s user interface) is a specific group of channels where users congregate to chat.</p>
6291 <div class="alert alert-primary" role="alert">
6292 <p><strong>Note:</strong> While <strong>guild</strong> and <strong>server</strong> are interchangeable, this article will use the term <strong>guild</strong> primarily because the APIs stick to the same term. The term <strong>server</strong> will only be used when referring to a guild in the graphical UI.</p>
6293 </div>
6294 <p>For example, say you want to create a space where users can come together and talk about your latest game. You’d start by creating a guild. Then, in your guild, you could have multiple channels, such as:</p>
6295 <ul>
6296 <li><strong>General Discussion:</strong> A channel for users to talk about whatever they want</li>
6297 <li><strong>Spoilers, Beware:</strong> A channel for users who have finished your game to talk about all the end game reveals</li>
6298 <li><strong>Announcements:</strong> A channel for you to announce game updates and for users to discuss them</li>
6299 </ul>
6300 <p>Once you’ve created your guild, you’d invite other users to populate it.</p>
6301 <p>So, to create a guild, head to your Discord <a href="https://discordapp.com/channels/@me">home</a> page:</p>
6302 <p><a href="https://files.realpython.com/media/discord-bot-homepage.f533b989cedd.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-homepage.f533b989cedd.png" width="3028" height="1717" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-homepage.f533b989cedd.png&w=757&sig=34788068aad51ef7668a3e5ff0583199519ec3c9 757w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-homepage.f533b989cedd.png&w=1514&sig=e36d4b45395b91c00d546eee352c80cee2f8165a 1514w, https://files.realpython.com/media/discord-bot-homepage.f533b989cedd.png 3028w" sizes="75vw" alt="Discord: User Account Home Page"/></a></p>
6303 <p>From this home page, you can view and add friends, direct messages, and guilds. From here, select the <em>+</em> icon on the left-hand side of the web page to <em>Add a Server</em>:</p>
6304 <p><a href="https://files.realpython.com/media/discord-bot-add-server.bd5a5a58c50c.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-add-server.bd5a5a58c50c.png" width="3027" height="1721" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-add-server.bd5a5a58c50c.png&w=756&sig=1113434567a2bedb5b26726cce843adc7e5791d6 756w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-add-server.bd5a5a58c50c.png&w=1513&sig=5cb289f097eac68358ee2400d26e45ee9e377fd6 1513w, https://files.realpython.com/media/discord-bot-add-server.bd5a5a58c50c.png 3027w" sizes="75vw" alt="Discord: Add Server"/></a></p>
6305 <p>This will present two options, <em>Create a server</em> and <em>Join a Server</em>. In this case, select <em>Create a server</em> and enter a name for your guild:</p>
6306 <p><a href="https://files.realpython.com/media/discord-bot-create-server.922dba753792.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-create-server.922dba753792.png" width="3023" height="1716" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-create-server.922dba753792.png&w=755&sig=e5083d753497fc156dfec6c3b351abd790b8c0d8 755w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-create-server.922dba753792.png&w=1511&sig=4709fdcaa3feede390c444431adbb70f998f3f04 1511w, https://files.realpython.com/media/discord-bot-create-server.922dba753792.png 3023w" sizes="75vw" alt="Discord: Naming a Server"/></a></p>
6307 <p>Once you’ve finished creating your guild, you’ll be able to see the users on the right-hand side and the channels on the left:</p>
6308 <p><a href="https://files.realpython.com/media/discord-bot-server.cba61f3781cf.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-server.cba61f3781cf.png" width="3026" height="1721" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-server.cba61f3781cf.png&w=756&sig=b8d05142625b9e2ce814747d1cf64921eba8e048 756w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-server.cba61f3781cf.png&w=1513&sig=6b6139eeecc4f1f64523d0a4d96f69e1a623c558 1513w, https://files.realpython.com/media/discord-bot-server.cba61f3781cf.png 3026w" sizes="75vw" alt="Discord: Newly Created Server"/></a></p>
6309 <p>The final step on Discord is to register your bot with your new guild.</p>
6310 <h3 id="adding-a-bot-to-a-guild">Adding a Bot to a Guild</h3>
6311 <p>A bot can’t accept invites like a normal user can. Instead, you’ll add your bot using the OAuth2 protocol.</p>
6312 <div class="alert alert-primary" role="alert">
6313 <p><strong>Technical Detail:</strong> <a href="https://oauth.net/2/">OAuth2</a> is a protocol for dealing with authorization, where a service can grant a client application limited access based on the application’s credentials and allowed scopes.</p>
6314 </div>
6315 <p>To do so, head back to the <a href="http://discordapp.com/developers/applications">Developer Portal</a> and select the OAuth2 page from the left-hand navigation:</p>
6316 <p><a href="https://files.realpython.com/media/discord-bot-oauth2.7c000bfe571b.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-oauth2.7c000bfe571b.png" width="3008" height="1770" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-oauth2.7c000bfe571b.png&w=752&sig=a7d51e04cbb851359c7fe643d80acb8e1674d8a2 752w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-oauth2.7c000bfe571b.png&w=1504&sig=5ba25b1a754b3d08bb200328f56ea71e8b71db81 1504w, https://files.realpython.com/media/discord-bot-oauth2.7c000bfe571b.png 3008w" sizes="75vw" alt="Discord: Application OAuth2"/></a></p>
6317 <p>From this window, you’ll see the OAuth2 URL Generator.</p>
6318 <p>This tool generates an authorization URL that hits Discord’s OAuth2 API and authorizes API access using your application’s credentials.</p>
6319 <p>In this case, you’ll want to grant your application’s bot user access to Discord APIs using your application’s OAuth2 credentials.</p>
6320 <p>To do this, scroll down and select <em>bot</em> from the <em>SCOPES</em> options and <em>Administrator</em> from <em>BOT PERMISSIONS</em>:</p>
6321 <p><a href="https://files.realpython.com/media/discord-bot-scopes.ee333b7a5987.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-scopes.ee333b7a5987.png" width="3012" height="1766" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-scopes.ee333b7a5987.png&w=753&sig=3df33d0353cab7ddbf213779e83c711e5d629e21 753w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-scopes.ee333b7a5987.png&w=1506&sig=259c0a008e687ca6bcb131a63d825e4a3f5133c5 1506w, https://files.realpython.com/media/discord-bot-scopes.ee333b7a5987.png 3012w" sizes="75vw" alt="Discord: Application Scopes and Bot Permissions"/></a></p>
6322 <p>Now, Discord has generated your application’s authorization URL with the selected scope and permissions.</p>
6323 <div class="alert alert-primary" role="alert">
6324 <p><strong>Disclaimer:</strong> While we’re using <em>Administrator</em> for the purposes of this tutorial, you should be as granular as possible when granting permissions in a real-world application.</p>
6325 </div>
6326 <p>Select <em>Copy</em> beside the URL that was generated for you, paste it into your browser, and select your guild from the dropdown options:</p>
6327 <p><a href="https://files.realpython.com/media/discord-bot-select-server.3cd1af626256.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-select-server.3cd1af626256.png" width="3023" height="1715" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-select-server.3cd1af626256.png&w=755&sig=bfa0dba079ebee97013871c01220f66505eed020 755w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-select-server.3cd1af626256.png&w=1511&sig=f2e91e59e77705a053b9930c27d1bf198ff6c7dc 1511w, https://files.realpython.com/media/discord-bot-select-server.3cd1af626256.png 3023w" sizes="75vw" alt="Discord: Add Bot to a Server"/></a></p>
6328 <p>Click <em>Authorize</em>, and you’re done!</p>
6329 <div class="alert alert-primary" role="alert">
6330 <p><strong>Note:</strong> You might get a <a href="https://en.wikipedia.org/wiki/ReCAPTCHA">reCAPTCHA</a> before moving on. If so, you’ll need to prove you’re a human.</p>
6331 </div>
6332 <p>If you go back to your guild, then you’ll see that the bot has been added:</p>
6333 <p><a href="https://files.realpython.com/media/discord-bot-added-to-guild.4a6b4477bc1e.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-added-to-guild.4a6b4477bc1e.png" width="3024" height="1719" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-added-to-guild.4a6b4477bc1e.png&w=756&sig=39634e4118660e59511dee28f86e7bb290192f3d 756w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-added-to-guild.4a6b4477bc1e.png&w=1512&sig=add9851baae9338210082db661a6277cad86ef5f 1512w, https://files.realpython.com/media/discord-bot-added-to-guild.4a6b4477bc1e.png 3024w" sizes="75vw" alt="Discord: Bot Added to Guild"/></a></p>
6334 <p>In summary, you’ve created:</p>
6335 <ul>
6336 <li>An <strong>application</strong> that your bot will use to authenticate with Discord’s APIs</li>
6337 <li>A <strong>bot</strong> user that you’ll use to interact with other users and events in your guild</li>
6338 <li>A <strong>guild</strong> in which your user account and your bot user will be active</li>
6339 <li>A <strong>Discord</strong> account with which you created everything else and that you’ll use to interact with your bot</li>
6340 </ul>
6341 <p>Now, you know how to make a Discord bot using the Developer Portal. Next comes the fun stuff: implementing your bot in Python!</p>
6342 <h2 id="how-to-make-a-discord-bot-in-python">How to Make a Discord Bot in Python</h2>
6343 <p>Since you’re learning how to make a Discord bot with Python, you’ll be using <code>discord.py</code>.</p>
6344 <p><a href="https://discordpy.readthedocs.io/en/latest/index.html"><code>discord.py</code></a> is a Python library that exhaustively implements Discord’s APIs in an efficient and Pythonic way. This includes utilizing Python’s implementation of <a href="https://realpython.com/async-io-python/">Async IO</a>.</p>
6345 <p>Begin by installing <code>discord.py</code> with <a href="https://realpython.com/what-is-pip/"><code>pip</code></a>:</p>
6346 <div class="highlight sh"><pre><span></span><span class="gp">$</span> pip install -U discord.py
6347 </pre></div>
6348
6349 <p>Now that you’ve installed <code>discord.py</code>, you’ll use it to create your first connection to Discord!</p>
6350 <h2 id="creating-a-discord-connection">Creating a Discord Connection</h2>
6351 <p>The first step in implementing your bot user is to create a connection to Discord. With <code>discord.py</code>, you do this by creating an instance of <code>Client</code>:</p>
6352 <div class="highlight python"><pre><span></span><span class="c1"># bot.py</span>
6353 <span class="kn">import</span> <span class="nn">os</span>
6354
6355 <span class="kn">import</span> <span class="nn">discord</span>
6356 <span class="kn">from</span> <span class="nn">dotenv</span> <span class="k">import</span> <span class="n">load_dotenv</span>
6357
6358 <span class="n">load_dotenv</span><span class="p">()</span>
6359 <span class="n">token</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s1">'DISCORD_TOKEN'</span><span class="p">)</span>
6360
6361 <span class="n">client</span> <span class="o">=</span> <span class="n">discord</span><span class="o">.</span><span class="n">Client</span><span class="p">()</span>
6362
6363 <span class="nd">@client</span><span class="o">.</span><span class="n">event</span>
6364 <span class="k">async</span> <span class="k">def</span> <span class="nf">on_ready</span><span class="p">():</span>
6365 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'</span><span class="si">{client.user}</span><span class="s1"> has connected to Discord!'</span><span class="p">)</span>
6366
6367 <span class="n">client</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">token</span><span class="p">)</span>
6368 </pre></div>
6369
6370 <p>A <code>Client</code> is an object that represents a connection to Discord. A <code>Client</code> handles events, tracks state, and generally interacts with Discord APIs.</p>
6371 <p>Here, you’ve created a <code>Client</code> and implemented its <code>on_ready()</code> event handler, which handles the event when the <code>Client</code> has established a connection to Discord and it has finished preparing the data that Discord has sent, such as login state, guild and channel data, and more.</p>
6372 <p>In other words, <code>on_ready()</code> will be called (and your message will be printed) once <code>client</code> is ready for further action. You’ll learn more about event handlers later in this article.</p>
6373 <p>When you’re working with secrets such as your Discord token, it’s good practice to read it into your program from an environment variable. Using environment variables helps you:</p>
6374 <ul>
6375 <li>Avoid putting the secrets into source control</li>
6376 <li>Use different variables for development and production environments without changing your code</li>
6377 </ul>
6378 <p>While you could <code>export DISCORD_TOKEN={your-bot-token}</code>, an easier solution is to save a <code>.env</code> file on all machines that will be running this code. This is not only easier, since you won’t have to <code>export</code> your token every time you clear your shell, but it also protects you from storing your secrets in your shell’s history.</p>
6379 <p>Create a file named <code>.env</code> in the same directory as <code>bot.py</code>:</p>
6380 <div class="highlight text"><pre><span></span># .env
6381 DISCORD_TOKEN={your-bot-token}
6382 </pre></div>
6383
6384 <p>You’ll need to replace <code>{your-bot-token}</code> with your bot’s token, which you can get by going back to the <em>Bot</em> page on the <a href="http://discordapp.com/developers/applications">Developer Portal</a> and clicking <em>Copy</em> under the <em>TOKEN</em> section:</p>
6385 <p><a href="https://files.realpython.com/media/discord-bot-copy-token.1228e6cb6cba.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-copy-token.1228e6cb6cba.png" width="3024" height="1767" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-copy-token.1228e6cb6cba.png&w=756&sig=57425662c2056deac50ebeb9a8d2891d4a8a6b53 756w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-copy-token.1228e6cb6cba.png&w=1512&sig=c8c437a86a1fb23a94e35050f5c969f5f20c4814 1512w, https://files.realpython.com/media/discord-bot-copy-token.1228e6cb6cba.png 3024w" sizes="75vw" alt="Discord: Copy Bot Token"/></a></p>
6386 <p>Looking back at the <code>bot.py</code> code, you’ll notice a library called <a href="https://github.com/theskumar/python-dotenv"><code>dotenv</code></a>. This library is handy for working with <code>.env</code> files. <code>load_dotenv()</code> loads environment variables from a <code>.env</code> file into your shell’s environment variables so that you can use them in your code.</p>
6387 <p>Install <code>dotenv</code> with <code>pip</code>:</p>
6388 <div class="highlight sh"><pre><span></span><span class="gp">$</span> pip install -U python-dotenv
6389 </pre></div>
6390
6391 <p>Finally, <code>client.run()</code> runs your <code>Client</code> using your bot’s token.</p>
6392 <p>Now that you’ve set up both <code>bot.py</code> and <code>.env</code>, you can run your code:</p>
6393 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python bot.py
6394 <span class="go">RealPythonTutorialBot#9643 has connected to Discord!</span>
6395 </pre></div>
6396
6397 <p>Great! Your <code>Client</code> has connected to Discord using your bot’s token. In the next section, you’ll build on this <code>Client</code> by interacting with more Discord APIs.</p>
6398 <h2 id="interacting-with-discord-apis">Interacting With Discord APIs</h2>
6399 <p>Using a <code>Client</code>, you have access to a wide range of Discord APIs.</p>
6400 <p>For example, let’s say you wanted to write the name and identifier of the guild that you registered your bot user with to the console.</p>
6401 <p>First, you’ll need to add a new environment variable:</p>
6402 <div class="highlight text"><pre><span></span># .env
6403 DISCORD_TOKEN={your-bot-token}
6404 <span class="hll">DISCORD_GUILD={your-guild-name}
6405 </span></pre></div>
6406
6407 <p>Don’t forget that you’ll need to replace the two placeholders with actual values:</p>
6408 <ol>
6409 <li><code>{your-bot-token}</code></li>
6410 <li><code>{your-guild-name}</code></li>
6411 </ol>
6412 <p>Remember that Discord calls <code>on_ready()</code>, which you used before, once the <code>Client</code> has made the connection and prepared the data. So, you can rely on the guild data being available inside <code>on_ready()</code>:</p>
6413 <div class="highlight python"><pre><span></span><span class="c1"># bot.py</span>
6414 <span class="kn">import</span> <span class="nn">os</span>
6415
6416 <span class="kn">import</span> <span class="nn">discord</span>
6417 <span class="kn">from</span> <span class="nn">dotenv</span> <span class="k">import</span> <span class="n">load_dotenv</span>
6418
6419 <span class="n">load_dotenv</span><span class="p">()</span>
6420 <span class="n">TOKEN</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s1">'DISCORD_TOKEN'</span><span class="p">)</span>
6421 <span class="n">GUILD</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s1">'DISCORD_GUILD'</span><span class="p">)</span>
6422
6423 <span class="n">client</span> <span class="o">=</span> <span class="n">discord</span><span class="o">.</span><span class="n">Client</span><span class="p">()</span>
6424
6425 <span class="nd">@client</span><span class="o">.</span><span class="n">event</span>
6426 <span class="k">async</span> <span class="k">def</span> <span class="nf">on_ready</span><span class="p">():</span>
6427 <span class="k">for</span> <span class="n">guild</span> <span class="ow">in</span> <span class="n">client</span><span class="o">.</span><span class="n">guilds</span><span class="p">:</span>
6428 <span class="k">if</span> <span class="n">guild</span><span class="o">.</span><span class="n">name</span> <span class="o">==</span> <span class="n">GUILD</span><span class="p">:</span>
6429 <span class="k">break</span>
6430
6431 <span class="nb">print</span><span class="p">(</span>
6432 <span class="n">f</span><span class="s1">'</span><span class="si">{client.user}</span><span class="s1"> is connected to the following guild:</span><span class="se">\n</span><span class="s1">'</span>
6433 <span class="n">f</span><span class="s1">'</span><span class="si">{guild.name}</span><span class="s1">(id: </span><span class="si">{guild.id}</span><span class="s1">)'</span>
6434 <span class="p">)</span>
6435
6436 <span class="n">client</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">TOKEN</span><span class="p">)</span>
6437 </pre></div>
6438
6439 <p>Here, you looped through the guild data that Discord has sent <code>client</code>, namely <code>client.guilds</code>. Then, you found the guild with the matching name and printed a <a href="https://realpython.com/python-f-strings/">formatted string</a> to <code>stdout</code>.</p>
6440 <div class="alert alert-primary" role="alert">
6441 <p><strong>Note:</strong> Even though you can be pretty confident at this point in the tutorial that your bot is only connected to a single guild (so <code>client.guilds[0]</code> would be simpler), it’s important to realize that a bot user can be connected to many guilds.</p>
6442 <p>Therefore, a more robust solution is to loop through <code>client.guilds</code> to find the one you’re looking for.</p>
6443 </div>
6444 <p>Run the program to see the results:</p>
6445 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python bot.py
6446 <span class="go">RealPythonTutorialBot#9643 is connected to the following guild:</span>
6447 <span class="go">RealPythonTutorialServer(id: 571759877328732195)</span>
6448 </pre></div>
6449
6450 <p>Great! You can see the name of your bot, the name of your server, and the server’s identification number.</p>
6451 <p>Another interesting bit of data you can pull from a guild is the list of users who are members of the guild:</p>
6452 <div class="highlight python"><pre><span></span><span class="c1"># bot.py</span>
6453 <span class="kn">import</span> <span class="nn">os</span>
6454
6455 <span class="kn">import</span> <span class="nn">discord</span>
6456 <span class="kn">from</span> <span class="nn">dotenv</span> <span class="k">import</span> <span class="n">load_dotenv</span>
6457
6458 <span class="n">load_dotenv</span><span class="p">()</span>
6459 <span class="n">TOKEN</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s1">'DISCORD_TOKEN'</span><span class="p">)</span>
6460 <span class="n">GUILD</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s1">'DISCORD_GUILD'</span><span class="p">)</span>
6461
6462 <span class="n">client</span> <span class="o">=</span> <span class="n">discord</span><span class="o">.</span><span class="n">Client</span><span class="p">()</span>
6463
6464 <span class="nd">@client</span><span class="o">.</span><span class="n">event</span>
6465 <span class="k">async</span> <span class="k">def</span> <span class="nf">on_ready</span><span class="p">():</span>
6466 <span class="k">for</span> <span class="n">guild</span> <span class="ow">in</span> <span class="n">client</span><span class="o">.</span><span class="n">guilds</span><span class="p">:</span>
6467 <span class="k">if</span> <span class="n">guild</span><span class="o">.</span><span class="n">name</span> <span class="o">==</span> <span class="n">GUILD</span><span class="p">:</span>
6468 <span class="k">break</span>
6469
6470 <span class="nb">print</span><span class="p">(</span>
6471 <span class="n">f</span><span class="s1">'</span><span class="si">{client.user}</span><span class="s1"> is connected to the following guild:</span><span class="se">\n</span><span class="s1">'</span>
6472 <span class="n">f</span><span class="s1">'</span><span class="si">{guild.name}</span><span class="s1">(id: </span><span class="si">{guild.id}</span><span class="s1">)</span><span class="se">\n</span><span class="s1">'</span>
6473 <span class="p">)</span>
6474
6475 <span class="n">members</span> <span class="o">=</span> <span class="s1">'</span><span class="se">\n</span><span class="s1"> - '</span><span class="o">.</span><span class="n">join</span><span class="p">([</span><span class="n">member</span><span class="o">.</span><span class="n">name</span> <span class="k">for</span> <span class="n">member</span> <span class="ow">in</span> <span class="n">guild</span><span class="o">.</span><span class="n">members</span><span class="p">])</span>
6476 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Guild Members:</span><span class="se">\n</span><span class="s1"> - </span><span class="si">{members}</span><span class="s1">'</span><span class="p">)</span>
6477
6478 <span class="n">client</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">TOKEN</span><span class="p">)</span>
6479 </pre></div>
6480
6481 <p>By looping through <code>guild.members</code>, you pulled the names of all of the members of the guild and printed them with a formatted string.</p>
6482 <p>When you run the program, you should see at least the name of the account you created the guild with and the name of the bot user itself:</p>
6483 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python bot.py
6484 <span class="go">RealPythonTutorialBot#9643 is connected to the following guild:</span>
6485 <span class="go">RealPythonTutorialServer(id: 571759877328732195)</span>
6486
6487 <span class="go">Guild Members:</span>
6488 <span class="go"> - aronq2</span>
6489 <span class="go"> - RealPythonTutorialBot</span>
6490 </pre></div>
6491
6492 <p>These examples barely scratch the surface of the APIs available on Discord, be sure to check out their <a href="https://discordpy.readthedocs.io/en/latest/api.html#">documentation</a> to see all that they have to offer.</p>
6493 <p>Next, you’ll learn about some utility functions and how they can simplify these examples.</p>
6494 <h2 id="using-utility-functions">Using Utility Functions</h2>
6495 <p>Let’s take another look at the example from the last section where you printed the name and identifier of the bot’s guild:</p>
6496 <div class="highlight python"><pre><span></span><span class="c1"># bot.py</span>
6497 <span class="kn">import</span> <span class="nn">os</span>
6498
6499 <span class="kn">import</span> <span class="nn">discord</span>
6500 <span class="kn">from</span> <span class="nn">dotenv</span> <span class="k">import</span> <span class="n">load_dotenv</span>
6501
6502 <span class="n">load_dotenv</span><span class="p">()</span>
6503 <span class="n">TOKEN</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s1">'DISCORD_TOKEN'</span><span class="p">)</span>
6504 <span class="n">GUILD</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s1">'DISCORD_GUILD'</span><span class="p">)</span>
6505
6506 <span class="n">client</span> <span class="o">=</span> <span class="n">discord</span><span class="o">.</span><span class="n">Client</span><span class="p">()</span>
6507
6508 <span class="nd">@client</span><span class="o">.</span><span class="n">event</span>
6509 <span class="k">async</span> <span class="k">def</span> <span class="nf">on_ready</span><span class="p">():</span>
6510 <span class="k">for</span> <span class="n">guild</span> <span class="ow">in</span> <span class="n">client</span><span class="o">.</span><span class="n">guilds</span><span class="p">:</span>
6511 <span class="k">if</span> <span class="n">guild</span><span class="o">.</span><span class="n">name</span> <span class="o">==</span> <span class="n">GUILD</span><span class="p">:</span>
6512 <span class="k">break</span>
6513
6514 <span class="nb">print</span><span class="p">(</span>
6515 <span class="n">f</span><span class="s1">'</span><span class="si">{client.user}</span><span class="s1"> is connected to the following guild:</span><span class="se">\n</span><span class="s1">'</span>
6516 <span class="n">f</span><span class="s1">'</span><span class="si">{guild.name}</span><span class="s1">(id: </span><span class="si">{guild.id}</span><span class="s1">)'</span>
6517 <span class="p">)</span>
6518
6519 <span class="n">client</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">TOKEN</span><span class="p">)</span>
6520 </pre></div>
6521
6522 <p>You could clean up this code by using some of the utility functions available in <code>discord.py</code>.</p>
6523 <p><a href="https://discordpy.readthedocs.io/en/latest/api.html#discord.utils.find"><code>discord.utils.find()</code></a> is one utility that can improve the simplicity and readability of this code by replacing the <code>for</code> loop with an intuitive, abstracted function:</p>
6524 <div class="highlight python"><pre><span></span><span class="c1"># bot.py</span>
6525 <span class="kn">import</span> <span class="nn">os</span>
6526
6527 <span class="kn">import</span> <span class="nn">discord</span>
6528 <span class="kn">from</span> <span class="nn">dotenv</span> <span class="k">import</span> <span class="n">load_dotenv</span>
6529
6530 <span class="n">load_dotenv</span><span class="p">()</span>
6531 <span class="n">TOKEN</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s1">'DISCORD_TOKEN'</span><span class="p">)</span>
6532 <span class="n">GUILD</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s1">'DISCORD_GUILD'</span><span class="p">)</span>
6533
6534 <span class="n">client</span> <span class="o">=</span> <span class="n">discord</span><span class="o">.</span><span class="n">Client</span><span class="p">()</span>
6535
6536 <span class="nd">@client</span><span class="o">.</span><span class="n">event</span>
6537 <span class="k">async</span> <span class="k">def</span> <span class="nf">on_ready</span><span class="p">():</span>
6538 <span class="n">guild</span> <span class="o">=</span> <span class="n">discord</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="k">lambda</span> <span class="n">g</span><span class="p">:</span> <span class="n">g</span><span class="o">.</span><span class="n">name</span> <span class="o">==</span> <span class="n">GUILD</span><span class="p">,</span> <span class="n">client</span><span class="o">.</span><span class="n">guilds</span><span class="p">)</span>
6539 <span class="nb">print</span><span class="p">(</span>
6540 <span class="n">f</span><span class="s1">'</span><span class="si">{client.user}</span><span class="s1"> is connected to the following guild:</span><span class="se">\n</span><span class="s1">'</span>
6541 <span class="n">f</span><span class="s1">'</span><span class="si">{guild.name}</span><span class="s1">(id: </span><span class="si">{guild.id}</span><span class="s1">)'</span>
6542 <span class="p">)</span>
6543
6544 <span class="n">client</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">TOKEN</span><span class="p">)</span>
6545 </pre></div>
6546
6547 <p><code>find()</code> takes a function, called a <strong>predicate</strong>, which identifies some characteristic of the element in the iterable that you’re looking for. Here, you used a particular type of anonymous function, called a <a href="https://realpython.com/python-lambda/">lambda</a>, as the predicate.</p>
6548 <p>In this case, you’re trying to find the guild with the same name as the one you stored in the <code>DISCORD_GUILD</code> environment variable. Once <code>find()</code> locates an element in the iterable that satisfies the predicate, it will return the element. This is essentially equivalent to the <code>break</code> statement in the previous example, but cleaner.</p>
6549 <p><code>discord.py</code> has even abstracted this concept one step further with the <a href="https://discordpy.readthedocs.io/en/latest/api.html#discord.utils.get"><code>get()</code> utility</a>:</p>
6550 <div class="highlight python"><pre><span></span><span class="c1"># bot.py</span>
6551 <span class="kn">import</span> <span class="nn">os</span>
6552
6553 <span class="kn">import</span> <span class="nn">discord</span>
6554 <span class="kn">from</span> <span class="nn">dotenv</span> <span class="k">import</span> <span class="n">load_dotenv</span>
6555
6556 <span class="n">load_dotenv</span><span class="p">()</span>
6557 <span class="n">TOKEN</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s1">'DISCORD_TOKEN'</span><span class="p">)</span>
6558 <span class="n">GUILD</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s1">'DISCORD_GUILD'</span><span class="p">)</span>
6559
6560 <span class="n">client</span> <span class="o">=</span> <span class="n">discord</span><span class="o">.</span><span class="n">Client</span><span class="p">()</span>
6561
6562 <span class="nd">@client</span><span class="o">.</span><span class="n">event</span>
6563 <span class="k">async</span> <span class="k">def</span> <span class="nf">on_ready</span><span class="p">():</span>
6564 <span class="n">guild</span> <span class="o">=</span> <span class="n">discord</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">client</span><span class="o">.</span><span class="n">guilds</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="n">GUILD</span><span class="p">)</span>
6565 <span class="nb">print</span><span class="p">(</span>
6566 <span class="n">f</span><span class="s1">'</span><span class="si">{client.user}</span><span class="s1"> is connected to the following guild:</span><span class="se">\n</span><span class="s1">'</span>
6567 <span class="n">f</span><span class="s1">'</span><span class="si">{guild.name}</span><span class="s1">(id: </span><span class="si">{guild.id}</span><span class="s1">)'</span>
6568 <span class="p">)</span>
6569
6570 <span class="n">client</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">TOKEN</span><span class="p">)</span>
6571 </pre></div>
6572
6573 <p><code>get()</code> takes the iterable and some keyword arguments. The keyword arguments represent attributes of the elements in the iterable that must all be satisfied for <code>get()</code> to return the element.</p>
6574 <p>In this example, you’ve identified <code>name=GUILD</code> as the attribute that must be satisfied.</p>
6575 <div class="alert alert-primary" role="alert">
6576 <p><strong>Technical Detail:</strong> Under the hood, <code>get()</code> actually uses the <code>attrs</code> keyword arguments to build a predicate, which it then uses to call <code>find()</code>.</p>
6577 </div>
6578 <p>Now that you’ve learned the basics of interacting with APIs, you’ll dive a little deeper into the function that you’ve been using to access them: <code>on_ready()</code>.</p>
6579 <h2 id="responding-to-events">Responding to Events</h2>
6580 <p>You already learned that <code>on_ready()</code> is an event. In fact, you might have noticed that it is identified as such in the code by the <code>client.event</code> <a href="https://realpython.com/primer-on-python-decorators/">decorator</a>.</p>
6581 <p>But what is an event?</p>
6582 <p>An <strong>event</strong> is something that happens on Discord that you can use to trigger a reaction in your code. Your code will listen for and then respond to events.</p>
6583 <p>Using the example you’ve seen already, the <code>on_ready()</code> event handler handles the event that the <code>Client</code> has made a connection to Discord and prepared its response data.</p>
6584 <p>So, when Discord fires an event, <code>discord.py</code> will route the event data to the corresponding event handler on your connected <code>Client</code>.</p>
6585 <p>There are two ways in <code>discord.py</code> to implement an event handler:</p>
6586 <ol>
6587 <li>Using the <code>client.event</code> decorator</li>
6588 <li>Creating a subclass of <code>Client</code> and overriding its handler methods</li>
6589 </ol>
6590 <p>You already saw the implementation using the decorator. Next, take a look at how to subclass <code>Client</code>:</p>
6591 <div class="highlight python"><pre><span></span><span class="c1"># bot.py</span>
6592 <span class="kn">import</span> <span class="nn">os</span>
6593
6594 <span class="kn">import</span> <span class="nn">discord</span>
6595 <span class="kn">from</span> <span class="nn">dotenv</span> <span class="k">import</span> <span class="n">load_dotenv</span>
6596
6597 <span class="n">load_dotenv</span><span class="p">()</span>
6598 <span class="n">token</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s1">'DISCORD_TOKEN'</span><span class="p">)</span>
6599
6600 <span class="k">class</span> <span class="nc">CustomClient</span><span class="p">(</span><span class="n">discord</span><span class="o">.</span><span class="n">Client</span><span class="p">):</span>
6601 <span class="k">async</span> <span class="k">def</span> <span class="nf">on_ready</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
6602 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'</span><span class="si">{self.user}</span><span class="s1"> has connected to Discord!'</span><span class="p">)</span>
6603
6604 <span class="n">client</span> <span class="o">=</span> <span class="n">CustomClient</span><span class="p">()</span>
6605 <span class="n">client</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">token</span><span class="p">)</span>
6606 </pre></div>
6607
6608 <p>Here, just like before, you’ve created a <code>client</code> variable and called <code>.run()</code> with your Discord token. The actual <code>Client</code> is different, however. Instead of using the normal base class, <code>client</code> is an instance of <code>CustomClient</code>, which has an overridden <code>on_ready()</code> function.</p>
6609 <p>There is no difference between the two implementation styles of events, but this tutorial will primarily use the decorator version because it looks similar to how you implement <code>Bot</code> commands, which is a topic you’ll cover in a bit.</p>
6610 <div class="alert alert-primary" role="alert">
6611 <p><strong>Technical Detail:</strong> Regardless of how you implement your event handler, one thing must be consistent: all event handlers in <code>discord.py</code> must be <a href="https://realpython.com/async-io-python/#the-asyncawait-syntax-and-native-coroutines">coroutines</a>.</p>
6612 </div>
6613 <p>Now that you’ve learned how to create an event handler, let’s walk through some different examples of handlers you can create.</p>
6614 <h3 id="welcoming-new-members">Welcoming New Members</h3>
6615 <p>Previously, you saw the example of responding to the event where a member joins a guild. In that example, your bot user could send them a message, welcoming them to your Discord community.</p>
6616 <p>Now, you’ll implement that behavior in your <code>Client</code>, using event handlers, and verify its behavior in Discord:</p>
6617 <div class="highlight python"><pre><span></span><span class="c1"># bot.py</span>
6618 <span class="kn">import</span> <span class="nn">os</span>
6619
6620 <span class="kn">import</span> <span class="nn">discord</span>
6621 <span class="kn">from</span> <span class="nn">dotenv</span> <span class="k">import</span> <span class="n">load_dotenv</span>
6622
6623 <span class="n">load_dotenv</span><span class="p">()</span>
6624 <span class="n">token</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s1">'DISCORD_TOKEN'</span><span class="p">)</span>
6625
6626 <span class="n">client</span> <span class="o">=</span> <span class="n">discord</span><span class="o">.</span><span class="n">Client</span><span class="p">()</span>
6627
6628 <span class="nd">@client</span><span class="o">.</span><span class="n">event</span>
6629 <span class="k">async</span> <span class="k">def</span> <span class="nf">on_ready</span><span class="p">():</span>
6630 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'</span><span class="si">{client.user.name}</span><span class="s1"> has connected to Discord!'</span><span class="p">)</span>
6631
6632 <span class="nd">@client</span><span class="o">.</span><span class="n">event</span>
6633 <span class="k">async</span> <span class="k">def</span> <span class="nf">on_member_join</span><span class="p">(</span><span class="n">member</span><span class="p">):</span>
6634 <span class="k">await</span> <span class="n">member</span><span class="o">.</span><span class="n">create_dm</span><span class="p">()</span>
6635 <span class="k">await</span> <span class="n">member</span><span class="o">.</span><span class="n">dm_channel</span><span class="o">.</span><span class="n">send</span><span class="p">(</span>
6636 <span class="n">f</span><span class="s1">'Hi </span><span class="si">{member.name}</span><span class="s1">, welcome to my Discord server!'</span>
6637 <span class="p">)</span>
6638
6639 <span class="n">client</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">token</span><span class="p">)</span>
6640 </pre></div>
6641
6642 <p>Like before, you handled the <code>on_ready()</code> event by printing the bot user’s name in a formatted string. New, however, is the implementation of the <code>on_member_join()</code> event handler.</p>
6643 <p><code>on_member_join()</code>, as its name suggests, handles the event of a new member joining a guild.</p>
6644 <p>In this example, you used <code>member.create_dm()</code> to create a direct message channel. Then, you used that channel to <code>.send()</code> a direct message to that new member.</p>
6645 <div class="alert alert-primary" role="alert">
6646 <p><strong>Technical Detail:</strong> Notice the <code>await</code> keyword before <code>member.create_dm()</code> and <code>member.dm_channel.send()</code>.</p>
6647 <p><code>await</code> suspends the execution of the surrounding coroutine until the execution of each coroutine has finished.</p>
6648 </div>
6649 <p>Now, let’s test out your bot’s new behavior.</p>
6650 <p>First, run your new version of <code>bot.py</code> and wait for the <code>on_ready()</code> event to fire, logging your message to <code>stdout</code>:</p>
6651 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python bot.py
6652 <span class="go">RealPythonTutorialBot has connected to Discord!</span>
6653 </pre></div>
6654
6655 <p>Now, head over to <a href="https://discordapp.com/">Discord</a>, log in, and navigate to your guild by selecting it from the left-hand side of the screen:</p>
6656 <p><a href="https://files.realpython.com/media/discord-bot-navigate-to-server.dfef0364630f.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-navigate-to-server.dfef0364630f.png" width="3024" height="1767" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-navigate-to-server.dfef0364630f.png&w=756&sig=e385f92a8203695f50bbd2d55922fc8ae3b4599e 756w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-navigate-to-server.dfef0364630f.png&w=1512&sig=7b1fd07e2bfbee2e23b36aa922a614d938b29698 1512w, https://files.realpython.com/media/discord-bot-navigate-to-server.dfef0364630f.png 3024w" sizes="75vw" alt="Discord: Navigate to Server"/></a></p>
6657 <p>Select <em>Invite People</em> just beside the guild list where you selected your guild. Check the box that says <em>Set this link to never expire</em> and copy the link:</p>
6658 <p><a href="https://files.realpython.com/media/discord-bot-copy-invite.0dd6b229c819.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-copy-invite.0dd6b229c819.png" width="3024" height="1766" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-copy-invite.0dd6b229c819.png&w=756&sig=81a88bffad8cbfa324a0eef38a1b6ff7ccfc67a5 756w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-copy-invite.0dd6b229c819.png&w=1512&sig=3c9f1911a3ebeb646c31e570e6a9ca8cff23f8cd 1512w, https://files.realpython.com/media/discord-bot-copy-invite.0dd6b229c819.png 3024w" sizes="75vw" alt="Discord: Copy Invite Link"/></a></p>
6659 <p>Now, with the invite link copied, create a new account and join the guild using your invite link:</p>
6660 <p><a href="https://files.realpython.com/media/discord-bot-accept-invite.4b33a1ba7062.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-accept-invite.4b33a1ba7062.png" width="3021" height="1767" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-accept-invite.4b33a1ba7062.png&w=755&sig=3b1a1db4101d859112e0a1d224d2aa8cc7606715 755w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-accept-invite.4b33a1ba7062.png&w=1510&sig=c5efa82e1d5940ff89422813371ddc83054cdbce 1510w, https://files.realpython.com/media/discord-bot-accept-invite.4b33a1ba7062.png 3021w" sizes="75vw" alt="Discord: Accept Invite"/></a></p>
6661 <p>First, you’ll see that Discord introduced you to the guild by default with an automated message. More importantly though, notice the badge on the left-hand side of the screen that notifies you of a new message:</p>
6662 <p><a href="https://files.realpython.com/media/discord-bot-direct-message-notification.95e423f72678.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-direct-message-notification.95e423f72678.png" width="3022" height="1768" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-direct-message-notification.95e423f72678.png&w=755&sig=84f08a7484e333f40081aa4071dafc4d331ca7f2 755w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-direct-message-notification.95e423f72678.png&w=1511&sig=51ed613c1cbe18a474ca12e6b648cb18d9a0505a 1511w, https://files.realpython.com/media/discord-bot-direct-message-notification.95e423f72678.png 3022w" sizes="75vw" alt="Discord: Direct Message Notification"/></a></p>
6663 <p>When you select it, you’ll see a private message from your bot user:</p>
6664 <p><a href="https://files.realpython.com/media/discord-bot-direct-message.7f49832b7bb7.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-direct-message.7f49832b7bb7.png" width="3024" height="1769" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-direct-message.7f49832b7bb7.png&w=756&sig=70663b008b61bdb83ce3f2f3b6cc4d4abda8a914 756w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-direct-message.7f49832b7bb7.png&w=1512&sig=5874eab3fb67f6bd0ba0b3a4ed5c6d401e23f8c8 1512w, https://files.realpython.com/media/discord-bot-direct-message.7f49832b7bb7.png 3024w" sizes="75vw" alt="Discord: Direct Message"/></a></p>
6665 <p>Perfect! Your bot user is now interacting with other users with minimal code.</p>
6666 <p>Next, you’ll learn how to respond to specific user messages in the chat.</p>
6667 <h3 id="responding-to-messages">Responding to Messages</h3>
6668 <p>Let’s add on to the previous functionality of your bot by handling the <code>on_message()</code> event.</p>
6669 <p><code>on_message()</code> occurs when a message is posted in a channel that your bot has access to. In this example, you’ll respond to the message <code>'99!'</code> with a one-liner from the television show <a href="https://www.nbc.com/brooklyn-nine-nine">Brooklyn Nine-Nine</a>:</p>
6670 <div class="highlight python"><pre><span></span><span class="nd">@client</span><span class="o">.</span><span class="n">event</span>
6671 <span class="k">async</span> <span class="k">def</span> <span class="nf">on_message</span><span class="p">(</span><span class="n">message</span><span class="p">):</span>
6672 <span class="k">if</span> <span class="n">message</span><span class="o">.</span><span class="n">author</span> <span class="o">==</span> <span class="n">client</span><span class="o">.</span><span class="n">user</span><span class="p">:</span>
6673 <span class="k">return</span>
6674
6675 <span class="n">brooklyn_99_quotes</span> <span class="o">=</span> <span class="p">[</span>
6676 <span class="s1">'I</span><span class="se">\'</span><span class="s1">m the human form of the π― emoji.'</span><span class="p">,</span>
6677 <span class="s1">'Bingpot!'</span><span class="p">,</span>
6678 <span class="p">(</span>
6679 <span class="s1">'Cool. Cool cool cool cool cool cool cool, '</span>
6680 <span class="s1">'no doubt no doubt no doubt no doubt.'</span>
6681 <span class="p">),</span>
6682 <span class="p">]</span>
6683
6684 <span class="k">if</span> <span class="n">message</span><span class="o">.</span><span class="n">content</span> <span class="o">==</span> <span class="s1">'99!'</span><span class="p">:</span>
6685 <span class="n">response</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">choice</span><span class="p">(</span><span class="n">brooklyn_99_quotes</span><span class="p">)</span>
6686 <span class="k">await</span> <span class="n">message</span><span class="o">.</span><span class="n">channel</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">response</span><span class="p">)</span>
6687 </pre></div>
6688
6689 <p>The bulk of this event handler looks at the <code>message.content</code>, checks to see if it’s equal to <code>'99!'</code>, and responds by sending a random quote to the message’s channel if it is.</p>
6690 <p>The other piece is an important one:</p>
6691 <div class="highlight python"><pre><span></span><span class="k">if</span> <span class="n">message</span><span class="o">.</span><span class="n">author</span> <span class="o">==</span> <span class="n">client</span><span class="o">.</span><span class="n">user</span><span class="p">:</span>
6692 <span class="k">return</span>
6693 </pre></div>
6694
6695 <p>Because a <code>Client</code> can’t tell the difference between a bot user and a normal user account, your <code>on_message()</code> handler should protect against a potentially recursive case where the bot sends a message that it might, itself, handle.</p>
6696 <p>To illustrate, let’s say you want your bot to listen for users telling each other <code>'Happy Birthday'</code>. You could implement your <code>on_message()</code> handler like this:</p>
6697 <div class="highlight python"><pre><span></span><span class="nd">@client</span><span class="o">.</span><span class="n">event</span>
6698 <span class="k">async</span> <span class="k">def</span> <span class="nf">on_message</span><span class="p">(</span><span class="n">message</span><span class="p">):</span>
6699 <span class="k">if</span> <span class="s1">'happy birthday'</span> <span class="ow">in</span> <span class="n">message</span><span class="o">.</span><span class="n">content</span><span class="o">.</span><span class="n">lower</span><span class="p">():</span>
6700 <span class="k">await</span> <span class="n">message</span><span class="o">.</span><span class="n">channel</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="s1">'Happy Birthday! ππ'</span><span class="p">)</span>
6701 </pre></div>
6702
6703 <p>Aside from the potentially spammy nature of this event handler, it also has a devastating side effect. The message that the bot responds with contains the same message it’s going to handle!</p>
6704 <p>So, if one person in the channel tells another “Happy Birthday,” then the bot will also chime in… again… and again… and again:</p>
6705 <p><a href="https://files.realpython.com/media/discord-bot-happy-birthday-repetition.864acfe23979.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-happy-birthday-repetition.864acfe23979.png" width="3028" height="1769" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-happy-birthday-repetition.864acfe23979.png&w=757&sig=1e56e02abfd0366024a7df56f2111fbb023f42be 757w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-happy-birthday-repetition.864acfe23979.png&w=1514&sig=2f67cb61941a012ff7d260910406a5c427c8f4c4 1514w, https://files.realpython.com/media/discord-bot-happy-birthday-repetition.864acfe23979.png 3028w" sizes="75vw" alt="Discord: Happy Birthday Message Repetition"/></a></p>
6706 <p>That’s why it’s important to compare the <code>message.author</code> to the <code>client.user</code> (your bot user), and ignore any of its own messages.</p>
6707 <p>So, let’s fix <code>bot.py</code>:</p>
6708 <div class="highlight python"><pre><span></span><span class="c1"># bot.py</span>
6709 <span class="kn">import</span> <span class="nn">os</span>
6710 <span class="kn">import</span> <span class="nn">random</span>
6711
6712 <span class="kn">import</span> <span class="nn">discord</span>
6713 <span class="kn">from</span> <span class="nn">dotenv</span> <span class="k">import</span> <span class="n">load_dotenv</span>
6714
6715 <span class="n">load_dotenv</span><span class="p">()</span>
6716 <span class="n">token</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s1">'DISCORD_TOKEN'</span><span class="p">)</span>
6717
6718 <span class="n">client</span> <span class="o">=</span> <span class="n">discord</span><span class="o">.</span><span class="n">Client</span><span class="p">()</span>
6719
6720 <span class="nd">@client</span><span class="o">.</span><span class="n">event</span>
6721 <span class="k">async</span> <span class="k">def</span> <span class="nf">on_ready</span><span class="p">():</span>
6722 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'</span><span class="si">{client.user.name}</span><span class="s1"> has connected to Discord!'</span><span class="p">)</span>
6723
6724 <span class="nd">@client</span><span class="o">.</span><span class="n">event</span>
6725 <span class="k">async</span> <span class="k">def</span> <span class="nf">on_member_join</span><span class="p">(</span><span class="n">member</span><span class="p">):</span>
6726 <span class="k">await</span> <span class="n">member</span><span class="o">.</span><span class="n">create_dm</span><span class="p">()</span>
6727 <span class="k">await</span> <span class="n">member</span><span class="o">.</span><span class="n">dm_channel</span><span class="o">.</span><span class="n">send</span><span class="p">(</span>
6728 <span class="n">f</span><span class="s1">'Hi </span><span class="si">{member.name}</span><span class="s1">, welcome to my Discord server!'</span>
6729 <span class="p">)</span>
6730
6731 <span class="nd">@client</span><span class="o">.</span><span class="n">event</span>
6732 <span class="k">async</span> <span class="k">def</span> <span class="nf">on_message</span><span class="p">(</span><span class="n">message</span><span class="p">):</span>
6733 <span class="k">if</span> <span class="n">message</span><span class="o">.</span><span class="n">author</span> <span class="o">==</span> <span class="n">client</span><span class="o">.</span><span class="n">user</span><span class="p">:</span>
6734 <span class="k">return</span>
6735
6736 <span class="n">brooklyn_99_quotes</span> <span class="o">=</span> <span class="p">[</span>
6737 <span class="s1">'I</span><span class="se">\'</span><span class="s1">m the human form of the π― emoji.'</span><span class="p">,</span>
6738 <span class="s1">'Bingpot!'</span><span class="p">,</span>
6739 <span class="p">(</span>
6740 <span class="s1">'Cool. Cool cool cool cool cool cool cool, '</span>
6741 <span class="s1">'no doubt no doubt no doubt no doubt.'</span>
6742 <span class="p">),</span>
6743 <span class="p">]</span>
6744
6745 <span class="k">if</span> <span class="n">message</span><span class="o">.</span><span class="n">content</span> <span class="o">==</span> <span class="s1">'99!'</span><span class="p">:</span>
6746 <span class="n">response</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">choice</span><span class="p">(</span><span class="n">brooklyn_99_quotes</span><span class="p">)</span>
6747 <span class="k">await</span> <span class="n">message</span><span class="o">.</span><span class="n">channel</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">response</span><span class="p">)</span>
6748
6749 <span class="n">client</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">token</span><span class="p">)</span>
6750 </pre></div>
6751
6752 <p>Don’t forget to <code>import random</code> at the top of the module, since the <code>on_message()</code> handler utilizes <code>random.choice()</code>.</p>
6753 <p>Run the program:</p>
6754 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python bot.py
6755 <span class="go">RealPythonTutorialBot has connected to Discord!</span>
6756 </pre></div>
6757
6758 <p>Finally, head over to Discord to test it out:</p>
6759 <p><a href="https://files.realpython.com/media/discord-bot-brooklyn-99-quotes.e934592e025e.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-brooklyn-99-quotes.e934592e025e.png" width="3027" height="1767" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-brooklyn-99-quotes.e934592e025e.png&w=756&sig=4f9a47703207d0920393a69f3411bd7d5914521b 756w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-brooklyn-99-quotes.e934592e025e.png&w=1513&sig=d78013102c8901874e9632b66e4a8d1b26f7b7fc 1513w, https://files.realpython.com/media/discord-bot-brooklyn-99-quotes.e934592e025e.png 3027w" sizes="75vw" alt="Discord: Quotes From Brooklyn Nine-Nine"/></a></p>
6760 <p>Great! Now that you’ve seen a few different ways to handle some common Discord events, you’ll learn how to deal with errors that event handlers may raise.</p>
6761 <h3 id="handling-exceptions">Handling Exceptions</h3>
6762 <p>As you’ve seen already, <code>discord.py</code> is an event-driven system. This focus on events extends even to exceptions. When one event handler <a href="https://realpython.com/python-exceptions/">raises an <code>Exception</code></a>, Discord calls <code>on_error()</code>.</p>
6763 <p>The default behavior of <code>on_error()</code> is to write the error message and stack trace to <code>stderr</code>. To test this, add a special message handler to <code>on_message()</code>:</p>
6764 <div class="highlight python"><pre><span></span><span class="c1"># bot.py</span>
6765 <span class="kn">import</span> <span class="nn">os</span>
6766 <span class="kn">import</span> <span class="nn">random</span>
6767
6768 <span class="kn">import</span> <span class="nn">discord</span>
6769 <span class="kn">from</span> <span class="nn">dotenv</span> <span class="k">import</span> <span class="n">load_dotenv</span>
6770
6771 <span class="n">load_dotenv</span><span class="p">()</span>
6772 <span class="n">token</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s1">'DISCORD_TOKEN'</span><span class="p">)</span>
6773
6774 <span class="n">client</span> <span class="o">=</span> <span class="n">discord</span><span class="o">.</span><span class="n">Client</span><span class="p">()</span>
6775
6776 <span class="nd">@client</span><span class="o">.</span><span class="n">event</span>
6777 <span class="k">async</span> <span class="k">def</span> <span class="nf">on_ready</span><span class="p">():</span>
6778 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'</span><span class="si">{client.user.name}</span><span class="s1"> has connected to Discord!'</span><span class="p">)</span>
6779
6780 <span class="nd">@client</span><span class="o">.</span><span class="n">event</span>
6781 <span class="k">async</span> <span class="k">def</span> <span class="nf">on_member_join</span><span class="p">(</span><span class="n">member</span><span class="p">):</span>
6782 <span class="k">await</span> <span class="n">member</span><span class="o">.</span><span class="n">create_dm</span><span class="p">()</span>
6783 <span class="k">await</span> <span class="n">member</span><span class="o">.</span><span class="n">dm_channel</span><span class="o">.</span><span class="n">send</span><span class="p">(</span>
6784 <span class="n">f</span><span class="s1">'Hi </span><span class="si">{member.name}</span><span class="s1">, welcome to my Discord server!'</span>
6785 <span class="p">)</span>
6786
6787 <span class="nd">@client</span><span class="o">.</span><span class="n">event</span>
6788 <span class="k">async</span> <span class="k">def</span> <span class="nf">on_message</span><span class="p">(</span><span class="n">message</span><span class="p">):</span>
6789 <span class="k">if</span> <span class="n">message</span><span class="o">.</span><span class="n">author</span> <span class="o">==</span> <span class="n">client</span><span class="o">.</span><span class="n">user</span><span class="p">:</span>
6790 <span class="k">return</span>
6791
6792 <span class="n">brooklyn_99_quotes</span> <span class="o">=</span> <span class="p">[</span>
6793 <span class="s1">'I</span><span class="se">\'</span><span class="s1">m the human form of the π― emoji.'</span><span class="p">,</span>
6794 <span class="s1">'Bingpot!'</span><span class="p">,</span>
6795 <span class="p">(</span>
6796 <span class="s1">'Cool. Cool cool cool cool cool cool cool, '</span>
6797 <span class="s1">'no doubt no doubt no doubt no doubt.'</span>
6798 <span class="p">),</span>
6799 <span class="p">]</span>
6800
6801 <span class="k">if</span> <span class="n">message</span><span class="o">.</span><span class="n">content</span> <span class="o">==</span> <span class="s1">'99!'</span><span class="p">:</span>
6802 <span class="n">response</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">choice</span><span class="p">(</span><span class="n">brooklyn_99_quotes</span><span class="p">)</span>
6803 <span class="k">await</span> <span class="n">message</span><span class="o">.</span><span class="n">channel</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">response</span><span class="p">)</span>
6804 <span class="hll"> <span class="k">elif</span> <span class="n">message</span><span class="o">.</span><span class="n">content</span> <span class="o">==</span> <span class="s1">'raise-exception'</span><span class="p">:</span>
6805 </span><span class="hll"> <span class="k">raise</span> <span class="n">discord</span><span class="o">.</span><span class="n">DiscordException</span>
6806 </span>
6807 <span class="n">client</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">token</span><span class="p">)</span>
6808 </pre></div>
6809
6810 <p>The new <code>raise-exception</code> message handler allows you to raise a <code>DiscordException</code> on command.</p>
6811 <p>Run the program and type <code>raise-exception</code> into the Discord channel:</p>
6812 <p><a href="https://files.realpython.com/media/discord-bot-raise-exception.7fcae85fb06e.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-raise-exception.7fcae85fb06e.png" width="3026" height="1765" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-raise-exception.7fcae85fb06e.png&w=756&sig=5d0026bbd91a3b8bb1bd561f0779dde3150d588a 756w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-raise-exception.7fcae85fb06e.png&w=1513&sig=7b226812216c03ef139c4c33823dd6318dc34ec5 1513w, https://files.realpython.com/media/discord-bot-raise-exception.7fcae85fb06e.png 3026w" sizes="75vw" alt="Discord: Raise Exception Message"/></a></p>
6813 <p>You should now see the <code>Exception</code> that was raised by your <code>on_message()</code> handler in the console:</p>
6814 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python bot.py
6815 <span class="go">RealPythonTutorialBot has connected to Discord!</span>
6816 <span class="go">Ignoring exception in on_message</span>
6817 <span class="go">Traceback (most recent call last):</span>
6818 <span class="go"> File "/Users/alex.ronquillo/.pyenv/versions/discord-venv/lib/python3.7/site-packages/discord/client.py", line 255, in _run_event</span>
6819 <span class="go"> await coro(*args, **kwargs)</span>
6820 <span class="go"> File "bot.py", line 42, in on_message</span>
6821 <span class="go"> raise discord.DiscordException</span>
6822 <span class="go">discord.errors.DiscordException</span>
6823 </pre></div>
6824
6825 <p>The exception was caught by the default error handler, so the output contains the message <code>Ignoring exception in on_message</code>. Let’s fix that by handling that particular error. To do so, you’ll catch the <code>DiscordException</code> and <a href="https://realpython.com/working-with-files-in-python/">write it to a file</a> instead.</p>
6826 <p>The <code>on_error()</code> event handler takes the <code>event</code> as the first argument. In this case, we expect the <code>event</code> to be <code>'on_message'</code>. It also accepts <code>*args</code> and <code>**kwargs</code> as flexible, positional and keyword arguments passed to the original event handler.</p>
6827 <p>So, since <code>on_message()</code> takes a single argument, <code>message</code>, we expect <code>args[0]</code> to be the <code>message</code> that the user sent in the Discord channel:</p>
6828 <div class="highlight python"><pre><span></span><span class="nd">@client</span><span class="o">.</span><span class="n">event</span>
6829 <span class="k">async</span> <span class="k">def</span> <span class="nf">on_error</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
6830 <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'err.log'</span><span class="p">,</span> <span class="s1">'a'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
6831 <span class="k">if</span> <span class="n">event</span> <span class="o">==</span> <span class="s1">'on_message'</span><span class="p">:</span>
6832 <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">f</span><span class="s1">'Unhandled message: </span><span class="si">{args[0]}</span><span class="se">\n</span><span class="s1">'</span><span class="p">)</span>
6833 <span class="k">else</span><span class="p">:</span>
6834 <span class="k">raise</span>
6835 </pre></div>
6836
6837 <p>If the <code>Exception</code> originated in the <code>on_message()</code> event handler, you <code>.write()</code> a formatted string to the file <code>err.log</code>. If another event raises an <code>Exception</code>, then we simply want our handler to re-raise the exception to invoke the default behavior.</p>
6838 <p>Run <code>bot.py</code> and send the <code>raise-exception</code> message again to view the output in <code>err.log</code>:</p>
6839 <div class="highlight sh"><pre><span></span><span class="gp">$</span> cat err.log
6840 <span class="go">Unhandled message: <Message id=573845548923224084 pinned=False author=<Member id=543612676807327754 name='alexronquillo' discriminator='0933' bot=False nick=None guild=<Guild id=571759877328732195 name='RealPythonTutorialServer' chunked=True>>></span>
6841 </pre></div>
6842
6843 <p>Instead of only a stack trace, you have a more informative error, showing the <code>message</code> that caused <code>on_message()</code> to raise the <code>DiscordException</code>, saved to a file for longer persistence.</p>
6844 <div class="alert alert-primary" role="alert">
6845 <p><strong>Technical Detail:</strong> If you want to take the actual <code>Exception</code> into account when you’re writing your error messages to <code>err.log</code>, then you can use functions from <code>sys</code>, such as <a href="https://docs.python.org/library/sys.html#sys.exc_info"><code>exc_info()</code></a>.</p>
6846 </div>
6847 <p>Now that you have some experience handling different events and interacting with Discord APIs, you’ll learn about a subclass of <code>Client</code> called <code>Bot</code>, which implements some handy, bot-specific functionality.</p>
6848 <h2 id="connecting-a-bot">Connecting a Bot</h2>
6849 <p>A <code>Bot</code> is a subclass of <code>Client</code> that adds a little bit of extra functionality that is useful when you’re creating bot users. For example, a <code>Bot</code> can handle events and commands, invoke validation checks, and more.</p>
6850 <p>Before you get into the features specific to <code>Bot</code>, convert <code>bot.py</code> to use a <code>Bot</code> instead of a <code>Client</code>:</p>
6851 <div class="highlight python"><pre><span></span><span class="c1"># bot.py</span>
6852 <span class="kn">import</span> <span class="nn">os</span>
6853 <span class="kn">import</span> <span class="nn">random</span>
6854 <span class="kn">from</span> <span class="nn">dotenv</span> <span class="k">import</span> <span class="n">load_dotenv</span>
6855
6856 <span class="c1"># 1</span>
6857 <span class="kn">from</span> <span class="nn">discord.ext</span> <span class="k">import</span> <span class="n">commands</span>
6858
6859 <span class="n">load_dotenv</span><span class="p">()</span>
6860 <span class="n">token</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s1">'DISCORD_TOKEN'</span><span class="p">)</span>
6861
6862 <span class="c1"># 2</span>
6863 <span class="n">bot</span> <span class="o">=</span> <span class="n">commands</span><span class="o">.</span><span class="n">Bot</span><span class="p">(</span><span class="n">command_prefix</span><span class="o">=</span><span class="s1">'!'</span><span class="p">)</span>
6864
6865 <span class="nd">@bot</span><span class="o">.</span><span class="n">event</span>
6866 <span class="k">async</span> <span class="k">def</span> <span class="nf">on_ready</span><span class="p">():</span>
6867 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'</span><span class="si">{bot.user.name}</span><span class="s1"> has connected to Discord!'</span><span class="p">)</span>
6868
6869 <span class="n">bot</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">token</span><span class="p">)</span>
6870 </pre></div>
6871
6872 <p>As you can see, <code>Bot</code> can handle events the same way that <code>Client</code> does. However, notice the differences between <code>Client</code> and <code>Bot</code>:</p>
6873 <ol>
6874 <li><code>Bot</code> is imported from the <code>discord.ext.commands</code> module.</li>
6875 <li>The <code>Bot</code> initializer requires a <code>command_prefix</code>, which you’ll learn more about in the next section.</li>
6876 </ol>
6877 <p>The extensions library, <code>ext</code>, offers several interesting components to help you create a Discord <code>Bot</code>. One such component is the <a href="https://discordpy.readthedocs.io/en/latest/ext/commands/commands.html"><code>Command</code></a>.</p>
6878 <h3 id="using-bot-commands">Using <code>Bot</code> Commands</h3>
6879 <p>In general terms, a <strong>command</strong> is an order that a user gives to a bot so that it will do something. Commands are different from events because they are:</p>
6880 <ul>
6881 <li>Arbitrarily defined</li>
6882 <li>Directly called by the user</li>
6883 <li>Flexible, in terms of their interface</li>
6884 </ul>
6885 <p>In technical terms, a <strong><code>Command</code></strong> is an object that wraps a function that is invoked by a text command in Discord. The text command must start with the <code>command_prefix</code>, defined by the <code>Bot</code> object.</p>
6886 <p>Let’s take a look at an old event to better understand what this looks like:</p>
6887 <div class="highlight python"><pre><span></span><span class="c1"># bot.py</span>
6888 <span class="kn">import</span> <span class="nn">os</span>
6889 <span class="kn">import</span> <span class="nn">random</span>
6890
6891 <span class="kn">import</span> <span class="nn">discord</span>
6892 <span class="kn">from</span> <span class="nn">dotenv</span> <span class="k">import</span> <span class="n">load_dotenv</span>
6893
6894 <span class="n">load_dotenv</span><span class="p">()</span>
6895 <span class="n">TOKEN</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s1">'DISCORD_TOKEN'</span><span class="p">)</span>
6896
6897 <span class="n">client</span> <span class="o">=</span> <span class="n">discord</span><span class="o">.</span><span class="n">Client</span><span class="p">()</span>
6898
6899 <span class="nd">@client</span><span class="o">.</span><span class="n">event</span>
6900 <span class="k">async</span> <span class="k">def</span> <span class="nf">on_message</span><span class="p">(</span><span class="n">message</span><span class="p">):</span>
6901 <span class="k">if</span> <span class="n">message</span><span class="o">.</span><span class="n">author</span> <span class="o">==</span> <span class="n">client</span><span class="o">.</span><span class="n">user</span><span class="p">:</span>
6902 <span class="k">return</span>
6903
6904 <span class="n">brooklyn_99_quotes</span> <span class="o">=</span> <span class="p">[</span>
6905 <span class="s1">'I</span><span class="se">\'</span><span class="s1">m the human form of the π― emoji.'</span><span class="p">,</span>
6906 <span class="s1">'Bingpot!'</span><span class="p">,</span>
6907 <span class="p">(</span>
6908 <span class="s1">'Cool. Cool cool cool cool cool cool cool, '</span>
6909 <span class="s1">'no doubt no doubt no doubt no doubt.'</span>
6910 <span class="p">),</span>
6911 <span class="p">]</span>
6912
6913 <span class="k">if</span> <span class="n">message</span><span class="o">.</span><span class="n">content</span> <span class="o">==</span> <span class="s1">'99!'</span><span class="p">:</span>
6914 <span class="n">response</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">choice</span><span class="p">(</span><span class="n">brooklyn_99_quotes</span><span class="p">)</span>
6915 <span class="k">await</span> <span class="n">message</span><span class="o">.</span><span class="n">channel</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">response</span><span class="p">)</span>
6916
6917 <span class="n">client</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">TOKEN</span><span class="p">)</span>
6918 </pre></div>
6919
6920 <p>Here, you created an <code>on_message()</code> event handler, which receives the <code>message</code> string and compares it to a pre-defined option: <code>'99!'</code>.</p>
6921 <p>Using a <code>Command</code>, you can convert this example to be more specific:</p>
6922 <div class="highlight python"><pre><span></span><span class="c1"># bot.py</span>
6923 <span class="kn">import</span> <span class="nn">os</span>
6924 <span class="kn">import</span> <span class="nn">random</span>
6925
6926 <span class="kn">from</span> <span class="nn">discord.ext</span> <span class="k">import</span> <span class="n">commands</span>
6927 <span class="kn">from</span> <span class="nn">dotenv</span> <span class="k">import</span> <span class="n">load_dotenv</span>
6928
6929 <span class="n">load_dotenv</span><span class="p">()</span>
6930 <span class="n">token</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s1">'DISCORD_TOKEN'</span><span class="p">)</span>
6931
6932 <span class="n">bot</span> <span class="o">=</span> <span class="n">commands</span><span class="o">.</span><span class="n">Bot</span><span class="p">(</span><span class="n">command_prefix</span><span class="o">=</span><span class="s1">'!'</span><span class="p">)</span>
6933
6934 <span class="nd">@bot</span><span class="o">.</span><span class="n">command</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s1">'99'</span><span class="p">)</span>
6935 <span class="k">async</span> <span class="k">def</span> <span class="nf">nine_nine</span><span class="p">(</span><span class="n">ctx</span><span class="p">):</span>
6936 <span class="n">brooklyn_99_quotes</span> <span class="o">=</span> <span class="p">[</span>
6937 <span class="s1">'I</span><span class="se">\'</span><span class="s1">m the human form of the π― emoji.'</span><span class="p">,</span>
6938 <span class="s1">'Bingpot!'</span><span class="p">,</span>
6939 <span class="p">(</span>
6940 <span class="s1">'Cool. Cool cool cool cool cool cool cool, '</span>
6941 <span class="s1">'no doubt no doubt no doubt no doubt.'</span>
6942 <span class="p">),</span>
6943 <span class="p">]</span>
6944
6945 <span class="n">response</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">choice</span><span class="p">(</span><span class="n">brooklyn_99_quotes</span><span class="p">)</span>
6946 <span class="k">await</span> <span class="n">ctx</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">response</span><span class="p">)</span>
6947
6948 <span class="n">bot</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">token</span><span class="p">)</span>
6949 </pre></div>
6950
6951 <p>There are several important characteristics to understand about using <code>Command</code>:</p>
6952 <ol>
6953 <li>
6954 <p>Instead of using <code>bot.event</code> like before, you use <code>bot.command()</code>, passing the invocation command (<code>name</code>) as its argument.</p>
6955 </li>
6956 <li>
6957 <p>The function will now only be called when <code>!99</code> is mentioned in chat. This is different than the <code>on_message()</code> event, which was executed any time a user sent a message, regardless of the content.</p>
6958 </li>
6959 <li>
6960 <p>The command must be prefixed with the exclamation point (<code>!</code> ) because that’s the <code>command_prefix</code> that you defined in the initializer for your <code>Bot</code>.</p>
6961 </li>
6962 <li>
6963 <p>Any <code>Command</code> function (technically called a <code>callback</code>) must accept at least one parameter, called <code>ctx</code>, which is the <a href="https://discordpy.readthedocs.io/en/latest/ext/commands/commands.html#invocation-context"><code>Context</code></a> surrounding the invoked <code>Command</code>.</p>
6964 </li>
6965 </ol>
6966 <p>A <code>Context</code> holds data such as the channel and guild that the user called the <code>Command</code> from.</p>
6967 <p>Run the program:</p>
6968 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python bot.py
6969 </pre></div>
6970
6971 <p>With your bot running, you can now head to Discord to try out your new command:</p>
6972 <p><a href="https://files.realpython.com/media/discord-bot-brooklyn-99-command.f01b21540756.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-brooklyn-99-command.f01b21540756.png" width="3031" height="1769" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-brooklyn-99-command.f01b21540756.png&w=757&sig=dff1081b5b379d39c1cdd41115bdef90fa4a833c 757w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-brooklyn-99-command.f01b21540756.png&w=1515&sig=a05851348e0e7b2217a0900c531a16a42bf4f32e 1515w, https://files.realpython.com/media/discord-bot-brooklyn-99-command.f01b21540756.png 3031w" sizes="75vw" alt="Discord: Brooklyn Nine-Nine Command"/></a></p>
6973 <p>From the user’s point of view, the practical difference is that the prefix helps formalize the command, rather than simply reacting to a particular <code>on_message()</code> event.</p>
6974 <p>This comes with other great benefits as well. For example, you can invoke the <code>help</code> command to see all the commands that your <code>Bot</code> handles:</p>
6975 <p><a href="https://files.realpython.com/media/discord-bot-help-command.a2ec772cc910.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-help-command.a2ec772cc910.png" width="3027" height="1767" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-help-command.a2ec772cc910.png&w=756&sig=122cba0d422a63e39e790208f553a16ebbceb674 756w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-help-command.a2ec772cc910.png&w=1513&sig=e8ef5a87dcca2de43f9ff198bb191746c5757465 1513w, https://files.realpython.com/media/discord-bot-help-command.a2ec772cc910.png 3027w" sizes="75vw" alt="Discord: Help Command"/></a></p>
6976 <p>If you want to add a description to your command so that the <code>help</code> message is more informative, simply pass a <code>help</code> description to the <code>.command()</code> decorator:</p>
6977 <div class="highlight python"><pre><span></span><span class="c1"># bot.py</span>
6978 <span class="kn">import</span> <span class="nn">os</span>
6979 <span class="kn">import</span> <span class="nn">random</span>
6980
6981 <span class="kn">from</span> <span class="nn">discord.ext</span> <span class="k">import</span> <span class="n">commands</span>
6982 <span class="kn">from</span> <span class="nn">dotenv</span> <span class="k">import</span> <span class="n">load_dotenv</span>
6983
6984 <span class="n">load_dotenv</span><span class="p">()</span>
6985 <span class="n">token</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s1">'DISCORD_TOKEN'</span><span class="p">)</span>
6986
6987 <span class="n">bot</span> <span class="o">=</span> <span class="n">commands</span><span class="o">.</span><span class="n">Bot</span><span class="p">(</span><span class="n">command_prefix</span><span class="o">=</span><span class="s1">'!'</span><span class="p">)</span>
6988
6989 <span class="nd">@bot</span><span class="o">.</span><span class="n">command</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s1">'99'</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">'Responds with a random quote from Brooklyn 99'</span><span class="p">)</span>
6990 <span class="k">async</span> <span class="k">def</span> <span class="nf">nine_nine</span><span class="p">(</span><span class="n">ctx</span><span class="p">):</span>
6991 <span class="n">brooklyn_99_quotes</span> <span class="o">=</span> <span class="p">[</span>
6992 <span class="s1">'I</span><span class="se">\'</span><span class="s1">m the human form of the π― emoji.'</span><span class="p">,</span>
6993 <span class="s1">'Bingpot!'</span><span class="p">,</span>
6994 <span class="p">(</span>
6995 <span class="s1">'Cool. Cool cool cool cool cool cool cool, '</span>
6996 <span class="s1">'no doubt no doubt no doubt no doubt.'</span>
6997 <span class="p">),</span>
6998 <span class="p">]</span>
6999
7000 <span class="n">response</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">choice</span><span class="p">(</span><span class="n">brooklyn_99_quotes</span><span class="p">)</span>
7001 <span class="k">await</span> <span class="n">ctx</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">response</span><span class="p">)</span>
7002
7003 <span class="n">bot</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">token</span><span class="p">)</span>
7004 </pre></div>
7005
7006 <p>Now, when the user invokes the <code>help</code> command, your bot will present a description of your command:</p>
7007 <p><a href="https://files.realpython.com/media/discord-bot-help-description.7f710c984c66.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-help-description.7f710c984c66.png" width="3030" height="1771" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-help-description.7f710c984c66.png&w=757&sig=183786aba9f5ba8b48f494d4403adfc38436116d 757w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-help-description.7f710c984c66.png&w=1515&sig=9fde033f6aae07bf16a859ca882fd6b5ffbcf574 1515w, https://files.realpython.com/media/discord-bot-help-description.7f710c984c66.png 3030w" sizes="75vw" alt="Discord: Informative Help Description"/></a></p>
7008 <p>Keep in mind that all of this functionality exists only for the <code>Bot</code> subclass, not the <code>Client</code> superclass.</p>
7009 <p><code>Command</code> has another useful functionality: the ability to use a <code>Converter</code> to change the types of its arguments.</p>
7010 <h3 id="converting-parameters-automatically">Converting Parameters Automatically</h3>
7011 <p>Another benefit of using commands is the ability to <strong>convert</strong> parameters.</p>
7012 <p>Sometimes, you require a parameter to be a certain type, but arguments to a <code>Command</code> function are, by default, strings. A <a href="https://discordpy.readthedocs.io/en/latest/ext/commands/commands.html#converters"><code>Converter</code></a> lets you convert those parameters to the type that you expect.</p>
7013 <p>For example, if you want to build a <code>Command</code> for your bot user to simulate rolling some dice (knowing what you’ve learned so far), you might define it like this:</p>
7014 <div class="highlight python"><pre><span></span><span class="nd">@bot</span><span class="o">.</span><span class="n">command</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s1">'roll_dice'</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">'Simulates rolling dice.'</span><span class="p">)</span>
7015 <span class="k">async</span> <span class="k">def</span> <span class="nf">roll</span><span class="p">(</span><span class="n">ctx</span><span class="p">,</span> <span class="n">number_of_dice</span><span class="p">,</span> <span class="n">number_of_sides</span><span class="p">):</span>
7016 <span class="n">dice</span> <span class="o">=</span> <span class="p">[</span>
7017 <span class="nb">str</span><span class="p">(</span><span class="n">random</span><span class="o">.</span><span class="n">choice</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">number_of_sides</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)))</span>
7018 <span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">number_of_dice</span><span class="p">)</span>
7019 <span class="p">]</span>
7020 <span class="k">await</span> <span class="n">ctx</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="s1">', '</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">dice</span><span class="p">))</span>
7021 </pre></div>
7022
7023 <p>You defined <code>roll</code> to take two parameters:</p>
7024 <ol>
7025 <li>The number of dice to roll</li>
7026 <li>The number of sides per die</li>
7027 </ol>
7028 <p>Then, you decorated it with <code>.command()</code> so that you can invoke it with the <code>!roll_dice</code> command. Finally, you <code>.send()</code> the results in a message back to the <code>channel</code>.</p>
7029 <p>While this looks correct, it isn’t. Unfortunately, if you run <code>bot.py</code>, and invoke the <code>!roll_dice</code> command in your Discord channel, you’ll see the following error:</p>
7030 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python bot.py
7031 <span class="go">Ignoring exception in command roll_dice:</span>
7032 <span class="go">Traceback (most recent call last):</span>
7033 <span class="go"> File "/Users/alex.ronquillo/.pyenv/versions/discord-venv/lib/python3.7/site-packages/discord/ext/commands/core.py", line 63, in wrapped</span>
7034 <span class="go"> ret = await coro(*args, **kwargs)</span>
7035 <span class="go"> File "bot.py", line 40, in roll</span>
7036 <span class="go"> for _ in range(number_of_dice)</span>
7037 <span class="go">TypeError: 'str' object cannot be interpreted as an integer</span>
7038
7039 <span class="go">The above exception was the direct cause of the following exception:</span>
7040
7041 <span class="go">Traceback (most recent call last):</span>
7042 <span class="go"> File "/Users/alex.ronquillo/.pyenv/versions/discord-venv/lib/python3.7/site-packages/discord/ext/commands/bot.py", line 860, in invoke</span>
7043 <span class="go"> await ctx.command.invoke(ctx)</span>
7044 <span class="go"> File "/Users/alex.ronquillo/.pyenv/versions/discord-venv/lib/python3.7/site-packages/discord/ext/commands/core.py", line 698, in invoke</span>
7045 <span class="go"> await injected(*ctx.args, **ctx.kwargs)</span>
7046 <span class="go"> File "/Users/alex.ronquillo/.pyenv/versions/discord-venv/lib/python3.7/site-packages/discord/ext/commands/core.py", line 72, in wrapped</span>
7047 <span class="go"> raise CommandInvokeError(exc) from exc</span>
7048 <span class="go">discord.ext.commands.errors.CommandInvokeError: Command raised an exception: TypeError: 'str' object cannot be interpreted as an integer</span>
7049 </pre></div>
7050
7051 <p>In other words, <code>range()</code> can’t accept a <code>str</code> as an argument. Instead, it must be an <code>int</code>. While you could cast each value to an <code>int</code>, there is a better way: you can use a <code>Converter</code> .</p>
7052 <p>In <code>discord.py</code>, a <code>Converter</code> is defined using Python 3’s <a href="https://realpython.com/python-type-checking/#annotations">function annotations</a>:</p>
7053 <div class="highlight python"><pre><span></span><span class="nd">@bot</span><span class="o">.</span><span class="n">command</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s1">'roll_dice'</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">'Simulates rolling dice.'</span><span class="p">)</span>
7054 <span class="k">async</span> <span class="k">def</span> <span class="nf">roll</span><span class="p">(</span><span class="n">ctx</span><span class="p">,</span> <span class="n">number_of_dice</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span> <span class="n">number_of_sides</span><span class="p">:</span> <span class="nb">int</span><span class="p">):</span>
7055 <span class="n">dice</span> <span class="o">=</span> <span class="p">[</span>
7056 <span class="nb">str</span><span class="p">(</span><span class="n">random</span><span class="o">.</span><span class="n">choice</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">number_of_sides</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)))</span>
7057 <span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">number_of_dice</span><span class="p">)</span>
7058 <span class="p">]</span>
7059 <span class="k">await</span> <span class="n">ctx</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="s1">', '</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">dice</span><span class="p">))</span>
7060 </pre></div>
7061
7062 <p>You added <code>: int</code> annotations to the two parameters that you expect to be of type <code>int</code>. Try the command again:</p>
7063 <p><a href="https://files.realpython.com/media/discord-bot-roll-dice.0255e76f078e.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-roll-dice.0255e76f078e.png" width="3025" height="1769" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-roll-dice.0255e76f078e.png&w=756&sig=ab57569c628b33d801c556a37291da4c2bf38898 756w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-roll-dice.0255e76f078e.png&w=1512&sig=561a833205a5df4d50976436b4640f10942e6be3 1512w, https://files.realpython.com/media/discord-bot-roll-dice.0255e76f078e.png 3025w" sizes="75vw" alt="Discord: Bot Dice-Rolling Command"/></a></p>
7064 <p>With that little change, your command works! The difference is that you’re now converting the command arguments to <code>int</code>, which makes them compatible with your function’s logic.</p>
7065 <div class="alert alert-primary" role="alert">
7066 <p><strong>Note:</strong> A <code>Converter</code> can be any callable, not merely data types. The argument will be passed to the callable, and the return value will be passed into the <code>Command</code>.</p>
7067 </div>
7068 <p>Next, you’ll learn about the <code>Check</code> object and how it can improve your commands.</p>
7069 <h3 id="checking-command-predicates">Checking Command Predicates</h3>
7070 <p>A <code>Check</code> is a predicate that is evaluated before a <code>Command</code> is executed to ensure that the <code>Context</code> surrounding the <code>Command</code> invocation is valid.</p>
7071 <p>In an earlier example, you did something similar to verify that the user who sent a message that the bot handles was not the bot user, itself:</p>
7072 <div class="highlight python"><pre><span></span><span class="k">if</span> <span class="n">message</span><span class="o">.</span><span class="n">author</span> <span class="o">==</span> <span class="n">client</span><span class="o">.</span><span class="n">user</span><span class="p">:</span>
7073 <span class="k">return</span>
7074 </pre></div>
7075
7076 <p>The <code>commands</code> extension provides a cleaner and more usable mechanism for performing this kind of check, namely using <code>Check</code> objects.</p>
7077 <p>To demonstrate how this works, assume you want to support a command <code>!create_channel <channel_name></code> that creates a new channel. However, you only want to allow administrators the ability to create new channels with this command.</p>
7078 <p>First, you’ll need to create a new member role in the admin. Go into the Discord guild and select the <em>{Server Name} β Server Settings</em> menu:</p>
7079 <p><a href="https://files.realpython.com/media/discord-bot-server-settings.1eb7e71e881b.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-server-settings.1eb7e71e881b.png" width="3023" height="1768" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-server-settings.1eb7e71e881b.png&w=755&sig=65edbf90c63743f93f195bb32661ebd7dc8c453f 755w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-server-settings.1eb7e71e881b.png&w=1511&sig=9576ffae79458bead937645d0d0fc693e91c9972 1511w, https://files.realpython.com/media/discord-bot-server-settings.1eb7e71e881b.png 3023w" sizes="75vw" alt="Discord: Server Settings Screen"/></a></p>
7080 <p>Then, select <em>Roles</em> from the left-hand navigation list:</p>
7081 <p><a href="https://files.realpython.com/media/discord-bot-roles.bdc21374afa9.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-roles.bdc21374afa9.png" width="3002" height="1769" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-roles.bdc21374afa9.png&w=750&sig=c9353734130e1c77c2e9022990b89f79db8806b7 750w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-roles.bdc21374afa9.png&w=1501&sig=c990b12b692cfdee8eccd91f8dec3e70e1977a67 1501w, https://files.realpython.com/media/discord-bot-roles.bdc21374afa9.png 3002w" sizes="75vw" alt="Discord: Navigate to Roles"/></a></p>
7082 <p>Finally select the <em>+</em> sign next to <em>ROLES</em> and enter the name <code>admin</code> and select <em>Save Changes</em>:</p>
7083 <p><a href="https://files.realpython.com/media/discord-bot-new-role.7e8d95291d0d.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-new-role.7e8d95291d0d.png" width="3027" height="1767" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-new-role.7e8d95291d0d.png&w=756&sig=f69a116bd35511e1c5b1b70d26b96c483d0f723e 756w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-new-role.7e8d95291d0d.png&w=1513&sig=3d883a2bb6f60a880175c06d523790ca07cc208c 1513w, https://files.realpython.com/media/discord-bot-new-role.7e8d95291d0d.png 3027w" sizes="75vw" alt="Discord: Create New Admin Role"/></a></p>
7084 <p>Now, you’ve created an <code>admin</code> role that you can assign to particular users. Next, you’ll update <code>bot.py</code> to <code>Check</code> the user’s role before allowing them to initiate the command:</p>
7085 <div class="highlight python"><pre><span></span><span class="c1"># bot.py</span>
7086 <span class="kn">import</span> <span class="nn">os</span>
7087
7088 <span class="kn">import</span> <span class="nn">discord</span>
7089 <span class="kn">from</span> <span class="nn">discord.ext</span> <span class="k">import</span> <span class="n">commands</span>
7090 <span class="kn">from</span> <span class="nn">dotenv</span> <span class="k">import</span> <span class="n">load_dotenv</span>
7091
7092 <span class="n">load_dotenv</span><span class="p">()</span>
7093 <span class="n">TOKEN</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s1">'DISCORD_TOKEN'</span><span class="p">)</span>
7094
7095 <span class="n">bot</span> <span class="o">=</span> <span class="n">commands</span><span class="o">.</span><span class="n">Bot</span><span class="p">(</span><span class="n">command_prefix</span><span class="o">=</span><span class="s1">'!'</span><span class="p">)</span>
7096
7097 <span class="nd">@bot</span><span class="o">.</span><span class="n">command</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s1">'create-channel'</span><span class="p">)</span>
7098 <span class="nd">@commands</span><span class="o">.</span><span class="n">has_role</span><span class="p">(</span><span class="s1">'admin'</span><span class="p">)</span>
7099 <span class="k">async</span> <span class="k">def</span> <span class="nf">create_channel</span><span class="p">(</span><span class="n">ctx</span><span class="p">,</span> <span class="n">channel_name</span><span class="o">=</span><span class="s1">'real-python'</span><span class="p">):</span>
7100 <span class="n">guild</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">guild</span>
7101 <span class="n">existing_channel</span> <span class="o">=</span> <span class="n">discord</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">guild</span><span class="o">.</span><span class="n">channels</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="n">channel_name</span><span class="p">)</span>
7102 <span class="k">if</span> <span class="ow">not</span> <span class="n">existing_channel</span><span class="p">:</span>
7103 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Creating a new channel: </span><span class="si">{channel_name}</span><span class="s1">'</span><span class="p">)</span>
7104 <span class="k">await</span> <span class="n">guild</span><span class="o">.</span><span class="n">create_text_channel</span><span class="p">(</span><span class="n">channel_name</span><span class="p">)</span>
7105
7106 <span class="n">bot</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">TOKEN</span><span class="p">)</span>
7107 </pre></div>
7108
7109 <p>In <code>bot.py</code>, you have a new <code>Command</code> function, called <code>create_channel()</code> which takes an optional <code>channel_name</code> and creates that channel. <code>create_channel()</code> is also decorated with a <code>Check</code> called <code>has_role()</code>.</p>
7110 <p>You also use <code>discord.utils.get()</code> to ensure that you don’t create a channel with the same name as an existing channel.</p>
7111 <p>If you run this program as it is and type <code>!create-channel</code> into your Discord channel, then you’ll see the following error message:</p>
7112 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python bot.py
7113 <span class="go">Ignoring exception in command create-channel:</span>
7114 <span class="go">Traceback (most recent call last):</span>
7115 <span class="go"> File "/Users/alex.ronquillo/.pyenv/versions/discord-venv/lib/python3.7/site-packages/discord/ext/commands/bot.py", line 860, in invoke</span>
7116 <span class="go"> await ctx.command.invoke(ctx)</span>
7117 <span class="go"> File "/Users/alex.ronquillo/.pyenv/versions/discord-venv/lib/python3.7/site-packages/discord/ext/commands/core.py", line 691, in invoke</span>
7118 <span class="go"> await self.prepare(ctx)</span>
7119 <span class="go"> File "/Users/alex.ronquillo/.pyenv/versions/discord-venv/lib/python3.7/site-packages/discord/ext/commands/core.py", line 648, in prepare</span>
7120 <span class="go"> await self._verify_checks(ctx)</span>
7121 <span class="go"> File "/Users/alex.ronquillo/.pyenv/versions/discord-venv/lib/python3.7/site-packages/discord/ext/commands/core.py", line 598, in _verify_checks</span>
7122 <span class="go"> raise CheckFailure('The check functions for command {0.qualified_name} failed.'.format(self))</span>
7123 <span class="go">discord.ext.commands.errors.CheckFailure: The check functions for command create-channel failed.</span>
7124 </pre></div>
7125
7126 <p>This <code>CheckFailure</code> says that <code>has_role('admin')</code> failed. Unfortunately, this error only prints to <code>stdout</code>. It would be better to report this to the user in the channel. To do so, add the following event:</p>
7127 <div class="highlight python"><pre><span></span><span class="nd">@bot</span><span class="o">.</span><span class="n">event</span>
7128 <span class="k">async</span> <span class="k">def</span> <span class="nf">on_command_error</span><span class="p">(</span><span class="n">ctx</span><span class="p">,</span> <span class="n">error</span><span class="p">):</span>
7129 <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">error</span><span class="p">,</span> <span class="n">commands</span><span class="o">.</span><span class="n">errors</span><span class="o">.</span><span class="n">CheckFailure</span><span class="p">):</span>
7130 <span class="k">await</span> <span class="n">ctx</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="s1">'You do not have the correct role for this command.'</span><span class="p">)</span>
7131 </pre></div>
7132
7133 <p>This event handles an error event from the command and sends an informative error message back to the original <code>Context</code> of the invoked <code>Command</code>.</p>
7134 <p>Try it all again, and you should see an error in the Discord channel:</p>
7135 <p><a href="https://files.realpython.com/media/discord-bot-role-error-message.adfe85fe76a9.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-role-error-message.adfe85fe76a9.png" width="3028" height="1768" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-role-error-message.adfe85fe76a9.png&w=757&sig=f4c4c8c318e4079b634fd089dfbdbc14524de8d8 757w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-role-error-message.adfe85fe76a9.png&w=1514&sig=0ac21290b0674b8954a40c6a92c0e14c04a8ca78 1514w, https://files.realpython.com/media/discord-bot-role-error-message.adfe85fe76a9.png 3028w" sizes="75vw" alt="Discord: Role Check Error"/></a></p>
7136 <p>Great! Now, to resolve the issue, you’ll need to give yourself the <em>admin</em> role:</p>
7137 <p><a href="https://files.realpython.com/media/discord-bot-role-granted.081c0c317834.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-role-granted.081c0c317834.png" width="3026" height="1767" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-role-granted.081c0c317834.png&w=756&sig=d2cb8f8765ffd8fc574808a810170c267e06069f 756w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-role-granted.081c0c317834.png&w=1513&sig=33fb00357b97007f92ee8fa88ecadfdcecdb09f9 1513w, https://files.realpython.com/media/discord-bot-role-granted.081c0c317834.png 3026w" sizes="75vw" alt="Discord: Grant Admin Role"/></a></p>
7138 <p>With the <em>admin</em> role, your user will pass the <code>Check</code> and will be able to create channels using the command.</p>
7139 <div class="alert alert-primary" role="alert">
7140 <p><strong>Note:</strong> Keep in mind that in order to assign a role, your user will have to have the correct permissions. The easiest way to ensure this is to sign in with the user that you created the guild with.</p>
7141 </div>
7142 <p>When you type <code>!create-channel</code> again, you’ll successfully create the channel <em>real-python</em>:</p>
7143 <p><a href="https://files.realpython.com/media/discord-bot-new-channel.43cd2889446c.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/discord-bot-new-channel.43cd2889446c.png" width="3026" height="1768" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-new-channel.43cd2889446c.png&w=756&sig=768abb1b230e675f2d0b864311f357eff67414b2 756w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/discord-bot-new-channel.43cd2889446c.png&w=1513&sig=ee91b1a9895a80fa1d326de58e6e5b8ac4fd1719 1513w, https://files.realpython.com/media/discord-bot-new-channel.43cd2889446c.png 3026w" sizes="75vw" alt="Discord: Navigate to New Channel"/></a></p>
7144 <p>Also, note that you can pass the optional <code>channel_name</code> argument to name the channel to whatever you want!</p>
7145 <p>With this last example, you combined a <code>Command</code>, an event, a <code>Check</code>, and even the <code>get()</code> utility to create a useful Discord bot!</p>
7146 <h2 id="conclusion">Conclusion</h2>
7147 <p>Congratulations! Now, you’ve learned how to make a Discord bot in Python. You’re able to build bots for interacting with users in guilds that you create or even bots that other users can invite to interact with their communities. Your bots will be able to respond to messages and commands and numerous other events.</p>
7148 <p>In this tutorial, you learned the basics of creating your own Discord bot. You now know:</p>
7149 <ul>
7150 <li>What Discord is</li>
7151 <li>Why <code>discord.py</code> is so valuable</li>
7152 <li>How to make a Discord bot in the Developer Portal</li>
7153 <li>How to create a Discord connection in Python</li>
7154 <li>How to handle events</li>
7155 <li>How to create a <code>Bot</code> connection</li>
7156 <li>How to use bot commands, checks, and converters</li>
7157 </ul>
7158 <p>To read more about the powerful <code>discord.py</code> library and take your bots to the next level, read through their extensive <a href="https://discordapp.com/developers/docs/intro">documentation</a>. Also, now that you’re familiar with Discord APIs in general, you have a better foundation for building other types of Discord applications.</p>
7159 <hr />
7160 <p><em>[ Improve Your Python With π Python Tricks π β Get a short & sweet Python Trick delivered to your inbox every couple of days. <a href="https://realpython.com/python-tricks/?utm_source=realpython&utm_medium=rss&utm_campaign=footer">>> Click here to learn more and see examples</a> ]</em></p>#
7161 datePublished: #Mon Aug 19 14:00:00 2019#
7162 dateUpdated: #Mon Aug 19 14:00:00 2019#
7163 # Person begin ####################
7164 name: #Real Python#
7165 # Person end ######################
7166 # Item end ########################
7167 # Item begin ######################
7168 id: #https://realpython.com/effective-python-environment/#
7169 title: #An Effective Python Environment: Making Yourself at Home#
7170 link: #https://realpython.com/effective-python-environment/#
7171 description: #This guide will walk you through the decisions you need to make when customizing your development environment for working with Python.#
7172 content: #<p>When you’re first learning a new programming language, a lot of your time and effort go into understanding the syntax, code style, and built-in tooling. This is just as true for Python as it is for any other language. Once you gain enough familiarity to be comfortable with the ins and outs of Python, you can start to invest time into building a Python environment that will foster your productivity.</p>
7173 <p>Your shell is more than a prebuilt program provided to you as-is. It’s a framework on which you can build an ecosystem. This ecosystem will come to fit your needs so that you can spend less time fiddling and more time thinking about the next big project you’re working on.</p>
7174 <p>Although no two developers have the same setup, there are a number of choices everyone faces when cultivating their Python environment. It’s important to understand each of these decisions and the options available to you!</p>
7175 <p><strong>By the end of this article, you’ll be able to answer questions like:</strong></p>
7176 <ul>
7177 <li>What shell should I use? What terminal should I use?</li>
7178 <li>What version(s) of Python can I use?</li>
7179 <li>How do I manage dependencies for different projects?</li>
7180 <li>How can I make my tools do some of the work for me?</li>
7181 </ul>
7182 <p>Once you’ve answered these questions for yourself, you can embark on the journey of creating a Python environment to call your very own. Let’s get started!</p>
7183 <div class="alert alert-warning" role="alert"><p><strong>Free Bonus:</strong> <a href="" class="alert-link" data-toggle="modal" data-target="#modal-dependency-pitfalls-email-course" data-focus="false">Click here to get access to a free 5-day class</a> that shows you how to avoid common dependency management issues with tools like Pip, PyPI, Virtualenv, and requirements files.</p></div>
7184
7185 <h2 id="shells">Shells</h2>
7186 <p>When you use a <a href="https://realpython.com/command-line-interfaces-python-argparse/#what-is-a-command-line-interface">command-line interface</a> (CLI), you execute commands and see their output. A <strong>shell</strong> is a program that provides this (usually text-based) interface to you. Shells often provide their own programming language that you can use to manipulate files, install software, and so on.</p>
7187 <p>There are more unique shells than could be reasonably listed here, so you’ll see a few prominent ones. Others differ in syntax or enhanced features, but they generally provide the same core functionality.</p>
7188 <h3 id="unix-shells">Unix Shells</h3>
7189 <p><a href="https://en.wikipedia.org/wiki/Unix">Unix</a> is a family of operating systems first developed in the early days of computing. Unix’s popularity has lasted through today, heavily inspiring Linux and macOS. The first shells were developed for use with Unix and Unix-like operating systems.</p>
7190 <h4 id="bourne-shell-sh">Bourne Shell (<code>sh</code>)</h4>
7191 <p>The Bourne shell—developed by Stephen Bourne for Bell Labs in 1979—was one of the first to incorporate the idea of environment variables, conditionals, and loops. It has provided a strong basis for many other shells in use today and is still available on most systems at <code>/bin/sh</code>.</p>
7192 <h4 id="bourne-again-shell-bash">Bourne-Again Shell (<code>bash</code>)</h4>
7193 <p>Built on the success of the original Bourne shell, <code>bash</code> introduced improved user-interaction features. With <code>bash</code>, you get <span class="keys"><kbd class="key-tab">Tab</kbd></span> completion, history, and wildcard searching for commands and paths. The <code>bash</code> programming language provides more data types, like arrays.</p>
7194 <h4 id="z-shell-zsh">Z Shell (<code>zsh</code>)</h4>
7195 <p><code>zsh</code> combines many of the best features from other shells along with a few of its own tricks into one experience. <code>zsh</code> offers autocorrection of misspelled commands, shorthand for manipulating multiple files, and advanced options for customizing your command prompt.</p>
7196 <p><code>zsh</code> also provides a framework for deep customization. The <a href="https://ohmyz.sh">Oh My Zsh</a> project supplies a rich set of themes and plugins, and is often used hand in hand with <code>zsh</code>.</p>
7197 <p><a href="https://support.apple.com/en-us/HT208050">macOS will ship with <code>zsh</code> as its default shell starting with Catalina</a>, speaking to the shell’s popularity. Consider acquainting yourself with <code>zsh</code> now so that you’ll be comfortable with it going forward.</p>
7198 <h4 id="xonsh">Xonsh</h4>
7199 <p>If you’re feeling particularly adventurous, you can give <a href="https://xon.sh">Xonsh</a> a try. Xonsh is a shell that combines some features of other Unix-like shells with the power of Python syntax. You can use the language you already know to accomplish tasks on your filesystem and so on.</p>
7200 <p>Although Xonsh is powerful, it lacks the compatibility other shells tend to share. You might not be able to run many existing shell scripts in Xonsh as a result. If you find that you like Xonsh, but compatibility is a concern, then you can use Xonsh as a supplement to your activities in a more widely used shell.</p>
7201 <h3 id="windows-shells">Windows Shells</h3>
7202 <p>Similarly to Unix-like operating systems, Windows also offers a number of options when it comes to shells. The shells offered in Windows vary in features and syntax, so you may need to try several to find one you like best.</p>
7203 <h4 id="cmd-cmdexe">CMD (<code>cmd.exe</code>)</h4>
7204 <p>CMD (short for “command”) is the default CLI shell for Windows. It’s the successor to COMMAND.COM, the shell built for DOS (disk operating system).</p>
7205 <p>Because DOS and Unix evolved independently, the commands and syntax in CMD are markedly different from shells built for Unix-like systems. However, CMD still provides the same core functionality for browsing and manipulating files, running commands, and viewing output.</p>
7206 <h4 id="powershell">PowerShell</h4>
7207 <p>PowerShell was released in 2006 and also ships with Windows. It provides Unix-like aliases for most commands, so if you’re coming to Windows from macOS or Linux or have to use both, then PowerShell might be great for you.</p>
7208 <p>PowerShell is vastly more powerful than CMD. With PowerShell you can:</p>
7209 <ul>
7210 <li>Pipe the output of one command to the input of another</li>
7211 <li>Automate tasks through the exposed Windows management features</li>
7212 <li>Use a scripting language to accomplish complex tasks</li>
7213 </ul>
7214 <h4 id="windows-subsystem-for-linux">Windows Subsystem for Linux</h4>
7215 <p>Microsoft has released a <a href="https://docs.microsoft.com/en-us/windows/wsl/install-win10">Windows subsystem for Linux</a> (WSL) for running Linux directly on Windows. If you install WSL, then you can use <code>zsh</code>, <code>bash</code>, or any other Unix-like shell. If you want strong compatibility across your Windows and macOS or Linux environments, then be sure to give WSL a try. You may also consider <a href="https://opensource.com/article/18/5/dual-boot-linux">dual-booting Linux and Windows</a> as an alternative.</p>
7216 <p>See this <a href="https://en.wikipedia.org/wiki/Comparison_of_command_shells">comparison of command shells</a> for exhaustive coverage.</p>
7217 <h2 id="terminal-emulators">Terminal Emulators</h2>
7218 <p>Early developers used <strong>terminals</strong> to interact with a central mainframe computer. These were devices with a keyboard and a screen or printer that would display computed output.</p>
7219 <p>Today, computers are portable and don’t require separate devices to interact with them, but the terminology still remains. Whereas a shell provides the prompt and interpreter you use to interface with text-based CLI tools, a terminal <strong>emulator</strong> (often shortened to <strong>terminal</strong>) is the graphical application you run to access the shell.</p>
7220 <p>Almost any terminal you encounter should support the same basic features:</p>
7221 <ul>
7222 <li><strong>Text colors</strong> for syntax highlighting in your code or distinguishing meaningful text in command output</li>
7223 <li><strong>Scrolling</strong> for viewing an earlier command or its output</li>
7224 <li><strong>Copy/paste</strong> for transferring text in or out of the shell from other programs</li>
7225 <li><strong>Tabs</strong> for running multiple programs at once or separating your work into different sessions</li>
7226 </ul>
7227 <h3 id="macos-terminals">macOS Terminals</h3>
7228 <p>The terminal options available for macOS are all full-featured, differing mostly in aesthetics and specific integrations with other tools.</p>
7229 <h4 id="terminal">Terminal</h4>
7230 <p>If you’re using a Mac, then you may have used the built-in <a href="https://support.apple.com/guide/terminal/welcome/mac">Terminal</a> app before. Terminal supports all the usual functionality, and you can also customize the color scheme and a few hotkeys. It’s a nice enough tool if you don’t need many bells and whistles. You can find the Terminal app in <em>Applications → Utilities → Terminal</em> on macOS.</p>
7231 <h4 id="iterm2">iTerm2</h4>
7232 <p>I’ve been a long-time user of <a href="https://iterm2.com">iTerm2</a>. It takes the developer experience on Mac a step further, offering a much wider palette of customization and productivity options that enable you to:</p>
7233 <ul>
7234 <li>Integrate with the shell to jump quickly to previously entered commands</li>
7235 <li>Create custom search term highlighting in the output from commands</li>
7236 <li>Open URLs and files displayed in the terminal with <span class="keys"><kbd class="key-command">Cmd</kbd><span>+</span><kbd>click</kbd></span></li>
7237 </ul>
7238 <p>A Python API ships with the latest versions of iTerm2, so you can even improve your Python chops by developing more intricate customizations!</p>
7239 <p>iTerm2 is popular enough to enjoy first-class integration with several other tools, and has a healthy community building plugins and so on. It’s a good choice because of its more frequent release cycle compared to Terminal, which only updates as often as macOS does.</p>
7240 <h4 id="hyper">Hyper</h4>
7241 <p>A relative newcomer, <a href="https://hyper.is/">Hyper</a> is a terminal built on <a href="https://electronjs.org/">Electron</a>, a framework for building desktop applications using web technologies. Electron apps are heavily customizable because they’re “just JavaScript” under the hood. You can create any functionality that you can write the JavaScript for.</p>
7242 <p>On the other hand, JavaScript is a high-level programming language and won’t always perform as well as low-level languages like Objective-C or Swift. Be mindful of the plugins you install or create!</p>
7243 <h3 id="windows-terminals">Windows Terminals</h3>
7244 <p>As with the shell options, Windows terminal options vary widely in utility. Some are tightly bound to a particular shell as well.</p>
7245 <h4 id="command-prompt">Command Prompt</h4>
7246 <p>Command Prompt is the graphical application you can use to work with CMD in Windows. Like CMD, it’s a bare-bones tool for getting a few small things done. Although Command Prompt and CMD provide fewer features than other alternatives, you can be confident that they’ll be available on nearly every Windows installation and in a consistent place.</p>
7247 <h4 id="cygwin">Cygwin</h4>
7248 <p>Cygwin is a third-party suite of tools for Windows that provides a Unix-like wrapper. This was my preferred setup when I was in Windows, but you may consider adopting the Windows Subsystem for Linux as it receives more traction and polish.</p>
7249 <h4 id="windows-terminal">Windows Terminal</h4>
7250 <p>Microsoft recently released an open source terminal for Windows 10 called <a href="https://github.com/Microsoft/Terminal">Windows Terminal</a>. It lets you work in CMD, PowerShell, and even the Windows Subsystem for Linux. If you need to do a fair amount of shell work in Windows, then Windows Terminal is probably your best bet! Windows Terminal is still in late beta, so it doesn’t ship with Windows yet. Check the documentation for instructions on getting access.</p>
7251 <h2 id="python-version-management">Python Version Management</h2>
7252 <p>With your choice of terminal and shell made, you can focus your attention on your Python environment specifically.</p>
7253 <p>Something you’ll eventually run into is the need to run multiple <strong>versions</strong> of Python. Projects you use may only run on certain versions, or you may be interested in creating a project that supports multiple Python versions. You can configure your Python environment to accommodate these needs.</p>
7254 <p>macOS and most Unix operating systems come with a version of Python installed by default. This is often called the <strong>system Python</strong>. The system Python works just fine, but it’s usually out of date. As of this writing, macOS High Sierra still ships with Python 2.7.10 as the system Python.</p>
7255 <div class="alert alert-primary" role="alert">
7256 <p><strong>Note</strong>: You’ll almost certainly want to install the latest version of Python at a minimum, so you’ll have at least two versions of Python already. </p>
7257 <p><strong>It’s important that you leave the system Python as the default</strong>, because many parts of the system rely on the default Python being a specific version. This is one of many great reasons to customize your Python environment!</p>
7258 </div>
7259 <p>How do you navigate this? Tooling is here to help.</p>
7260 <h3 id="pyenv"><code>pyenv</code></h3>
7261 <p><a href="https://github.com/pyenv/pyenv"><code>pyenv</code></a> is a mature tool for installing and managing multiple Python versions on macOS. I recommend <a href="https://github.com/pyenv/pyenv#homebrew-on-macos">installing it with Homebrew</a>. If you’re using Windows, you can use <a href="https://github.com/pyenv-win/pyenv-win#installation"><code>pyenv-win</code></a>. After you’ve got <code>pyenv</code> installed, you can install multiple versions of Python into your Python environment with a few short commands:</p>
7262 <div class="highlight sh"><pre><span></span><span class="gp">$</span> pyenv versions
7263 <span class="go">* system</span>
7264 <span class="gp">$</span> python --version
7265 <span class="go">Python 2.7.10</span>
7266 <span class="gp">$</span> pyenv install <span class="m">3</span>.7.3 <span class="c1"># This may take some time</span>
7267 <span class="gp">$</span> pyenv versions
7268 <span class="go">* system</span>
7269 <span class="go"> 3.7.3</span>
7270 </pre></div>
7271
7272 <p>You can manage which Python you’d like to use in your current session, globally, or on a per-project basis as well. <code>pyenv</code> will make the <code>python</code> command point to whichever Python you specify. Note that none of these overrides the default system Python for other applications, so you’re safe to use them however they work best for you within your Python environment:</p>
7273 <div class="highlight sh"><pre><span></span><span class="gp">$</span> pyenv global <span class="m">3</span>.7.3
7274 <span class="gp">$</span> pyenv versions
7275 <span class="go"> system</span>
7276 <span class="go">* 3.7.3 (set by /Users/dhillard/.pyenv/version)</span>
7277
7278 <span class="gp">$</span> pyenv <span class="nb">local</span> <span class="m">3</span>.7.3
7279 <span class="gp">$</span> pyenv versions
7280 <span class="go"> system</span>
7281 <span class="go">* 3.7.3 (set by /Users/dhillard/myproj/.python-version)</span>
7282
7283 <span class="gp">$</span> pyenv shell <span class="m">3</span>.7.3
7284 <span class="gp">$</span> pyenv versions
7285 <span class="go"> system</span>
7286 <span class="go">* 3.7.3 (set by PYENV_VERSION environment variable)</span>
7287
7288 <span class="gp">$</span> python --version
7289 <span class="go">Python 3.7.3</span>
7290 </pre></div>
7291
7292 <p>Because I use a specific version of Python for work, the latest version of Python for personal projects, and multiple versions for testing open source projects, <code>pyenv</code> has proven to be a fairly smooth way for me to manage all these different versions within my own Python environment. See <a href="https://realpython.com/intro-to-pyenv/">Managing Multiple Python Versions with <code>pyenv</code></a> for a detailed overview of the tool.</p>
7293 <h3 id="conda"><code>conda</code></h3>
7294 <p>If you’re in the data science community, you might already be using <a href="https://www.anaconda.com/distribution/">Anaconda</a> (or <a href="https://docs.conda.io/en/latest/miniconda.html">Miniconda</a>). Anaconda is a sort of one-stop shop for data science software that supports more than just Python.</p>
7295 <p>If you don’t need the data science packages or all the things that come pre-packaged with Anaconda, <code>pyenv</code> might be a better lightweight solution for you. Managing Python versions is pretty similar in each, though. You can install Python versions similarly to <code>pyenv</code>, using the <code>conda</code> command:</p>
7296 <div class="highlight sh"><pre><span></span><span class="gp">$</span> conda install <span class="nv">python</span><span class="o">=</span><span class="m">3</span>.7.3
7297 </pre></div>
7298
7299 <p>You’ll see a verbose list of all the dependent software <code>conda</code> will install, and it will ask you to confirm.</p>
7300 <p><code>conda</code> doesn’t have a way to set the “default” Python version or even a good way to see which versions of Python you’ve installed. Rather, it hinges on the concept of “environments,” which you can read more about in the following sections.</p>
7301 <h2 id="virtual-environments">Virtual Environments</h2>
7302 <p>Now you know how to manage multiple Python versions. Often, you’ll be working on multiple projects that need the <em>same</em> Python version.</p>
7303 <p>Because each project has its own set of dependencies, it’s a good practice to avoid mixing them. If all the dependencies are installed together in a single Python environment, then it will be difficult to discern where each one came from. In the worst cases, two different projects may depend on two different versions of a package, but with Python you can only have one version of a package installed at one time. What a mess!</p>
7304 <p>Enter <strong>virtual environments</strong>. You can think of a virtual environment as a carbon copy of a base version of Python. If you’ve installed Python 3.7.3, for example, then you can create many virtual environments based off of it. When you install a package in a virtual environment, you do it in isolation from other Python environments you may have. Each virtual environment has its own copy of the <code>python</code> executable.</p>
7305 <div class="alert alert-primary" role="alert">
7306 <p><strong>Tip</strong>: Most virtual environment tooling provides a way to update your shell’s command prompt to show the current active virtual environment. Make sure to do this if you frequently switch between projects so you’re sure you’re working inside the correct virtual environment.</p>
7307 </div>
7308 <h3 id="venv"><code>venv</code></h3>
7309 <p><a href="https://docs.python.org/3/library/venv.html"><code>venv</code></a> ships with Python versions 3.3+. You can create virtual environments just by passing it a path at which to store the environment’s <code>python</code>, installed packages, and so on:</p>
7310 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python -m venv ~/.virtualenvs/my-env
7311 </pre></div>
7312
7313 <p>You activate a virtual environment by sourcing its <code>activate</code> script:</p>
7314 <div class="highlight sh"><pre><span></span><span class="gp">$</span> <span class="nb">source</span> ~/.virtualenvs/my-env/bin/activate
7315 </pre></div>
7316
7317 <p>You exit the virtual environment using the <code>deactivate</code> command, which is made available when you activate the virtual environment:</p>
7318 <div class="highlight sh"><pre><span></span><span class="gp">(my-env)$</span> deactivate
7319 </pre></div>
7320
7321 <p><code>venv</code> is built on the wonderful work and successes of the independent <a href="https://virtualenv.pypa.io/en/stable/"><code>virtualenv</code></a> project. <code>virtualenv</code> still provides a few interesting features of its own, but <code>venv</code> is nice because it provides the utility of virtual environments without requiring you to install additional software. You can probably get pretty far with it if you’re working mostly in a single Python version in your Python environment.</p>
7322 <p>If you’re already managing multiple Python versions (or plan to), then it could make sense to integrate with that tooling to simplify the process of making new virtual environments with specific versions of Python. The <code>pyenv</code> and <code>conda</code> ecosystems both provide ways to specify the Python version to use when you create new virtual environments, covered in the following sections.</p>
7323 <h3 id="pyenv-virtualenv"><code>pyenv-virtualenv</code></h3>
7324 <p>If you’re using <code>pyenv</code>, then <a href="https://github.com/pyenv/pyenv-virtualenv"><code>pyenv-virtualenv</code></a> enhances <code>pyenv</code> with a subcommand for managing virtual environments:</p>
7325 <div class="highlight sh"><pre><span></span><span class="go">// Create virtual environment</span>
7326 <span class="gp">$</span> pyenv virtualenv <span class="m">3</span>.7.3 my-env
7327
7328 <span class="go">// Activate virtual environment</span>
7329 <span class="gp">$</span> pyenv activate my-env
7330
7331 <span class="go">// Exit virtual environment</span>
7332 <span class="gp">(my-env)$</span> pyenv deactivate
7333 </pre></div>
7334
7335 <p>I switch contexts between a large handful of projects on a day-to-day basis. As a result, I have at least a dozen distinct virtual environments to manage in my Python environment. What’s really nice about <code>pyenv-virtualenv</code> is that you can configure a virtual environment using the <code>pyenv local</code> command and have <code>pyenv-virtualenv</code> auto-activate the right environments as you switch to different directories:</p>
7336 <div class="highlight sh"><pre><span></span><span class="gp">$</span> pyenv virtualenv <span class="m">3</span>.7.3 proj1
7337 <span class="gp">$</span> pyenv virtualenv <span class="m">3</span>.7.3 proj2
7338 <span class="gp">$</span> <span class="nb">cd</span> /Users/dhillard/proj1
7339 <span class="gp">$</span> pyenv <span class="nb">local</span> proj1
7340 <span class="gp">(proj1)$</span> <span class="nb">cd</span> ../proj2
7341 <span class="gp">$</span> pyenv <span class="nb">local</span> proj2
7342 <span class="gp">(proj2)$</span> pyenv versions
7343 <span class="go"> system</span>
7344 <span class="go"> 3.7.3</span>
7345 <span class="go"> 3.7.3/envs/proj1</span>
7346 <span class="go"> 3.7.3/envs/proj2</span>
7347 <span class="go"> proj1</span>
7348 <span class="go">* proj2 (set by /Users/dhillard/proj2/.python-version)</span>
7349 </pre></div>
7350
7351 <p><code>pyenv</code> and <code>pyenv-virtualenv</code> have provided a particularly fluid workflow in my Python environment.</p>
7352 <h3 id="conda_1"><code>conda</code></h3>
7353 <p>You saw earlier that <code>conda</code> treats environments, rather than Python versions, as the main method of working. <a href="https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html"><code>conda</code> has built-in support for managing virtual environments</a>:</p>
7354 <div class="highlight sh"><pre><span></span><span class="go">// Create virtual environment</span>
7355 <span class="gp">$</span> conda create --name my-env <span class="nv">python</span><span class="o">=</span><span class="m">3</span>.7.3
7356
7357 <span class="go">// Activate virtual environment</span>
7358 <span class="gp">$</span> conda activate my-env
7359
7360 <span class="go">// Exit virtual environment</span>
7361 <span class="gp">(my-env)$</span> conda deactivate
7362 </pre></div>
7363
7364 <p><code>conda</code> will install the specified version of Python if it isn’t already installed, so you don’t have to run <code>conda install python=3.7.3</code> first.</p>
7365 <h3 id="pipenv"><code>pipenv</code></h3>
7366 <p><a href="https://docs.pipenv.org/en/latest/"><code>pipenv</code></a> is a relatively new tool that seeks to combine package management (more on this in a moment) with virtual environment management. It mostly abstracts the virtual environment management from you, which can be great as long as things go smoothly:</p>
7367 <div class="highlight sh"><pre><span></span><span class="gp">$</span> <span class="nb">cd</span> /Users/dhillard/myproj
7368
7369 <span class="go">// Create virtual environment</span>
7370 <span class="gp">$</span> pipenv install
7371 <span class="go">Creating a virtualenv for this projectβ¦</span>
7372 <span class="go">Pipfile: /Users/dhillard/myproj/Pipfile</span>
7373 <span class="go">Using /path/to/pipenv/python3.7 (3.7.3) to create virtualenvβ¦</span>
7374 <span class="go">β Successfully created virtual environment!</span>
7375 <span class="go">Virtualenv location: /Users/dhillard/.local/share/virtualenvs/myproj-nAbMEAt0</span>
7376 <span class="go">Creating a Pipfile for this projectβ¦</span>
7377 <span class="go">Pipfile.lock not found, creatingβ¦</span>
7378 <span class="go">Locking [dev-packages] dependenciesβ¦</span>
7379 <span class="go">Locking [packages] dependenciesβ¦</span>
7380 <span class="go">Updated Pipfile.lock (a65489)!</span>
7381 <span class="go">Installing dependencies from Pipfile.lock (a65489)β¦</span>
7382 <span class="go"> π ββββββββββββββββββββββββββββββββ 0/0 β 00:00:00</span>
7383 <span class="go">To activate this project's virtualenv, run pipenv shell.</span>
7384 <span class="go">Alternatively, run a command inside the virtualenv with pipenv run.</span>
7385
7386 <span class="go">// Activate virtual environment (uses a subshell)</span>
7387 <span class="gp">$</span> pipenv shell
7388 <span class="go">Launching subshell in virtual environmentβ¦</span>
7389 <span class="go"> . /Users/dhillard/.local/share/virtualenvs/test-nAbMEAt0/bin/activate</span>
7390
7391 <span class="go">// Exit virtual environment (by exiting subshell)</span>
7392 <span class="gp">(myproj-nAbMEAt0)$</span> <span class="nb">exit</span>
7393 </pre></div>
7394
7395 <p><code>pipenv</code> does all the heavy lifting of creating a virtual environment and activating it for you. If you look carefully, you can see that it also creates a file called <code>Pipfile</code>. After you first run <code>pipenv install</code>, this file contains just a few things:</p>
7396 <div class="highlight ini"><pre><span></span><span class="k">[[source]]</span>
7397 <span class="na">name</span> <span class="o">=</span> <span class="s">"pypi"</span>
7398 <span class="na">url</span> <span class="o">=</span> <span class="s">"https://pypi.org/simple"</span>
7399 <span class="na">verify_ssl</span> <span class="o">=</span> <span class="s">true</span>
7400
7401 <span class="k">[dev-packages]</span>
7402
7403 <span class="k">[packages]</span>
7404
7405 <span class="k">[requires]</span>
7406 <span class="na">python_version</span> <span class="o">=</span> <span class="s">"3.7"</span>
7407 </pre></div>
7408
7409 <p>In particular, note that it shows <code>python_version = "3.7"</code>. By default, <code>pipenv</code> creates a virtual Python environment using the same Python version it was installed under. If you want to use a different Python version, then you can create the <code>Pipfile</code> yourself before running <code>pipenv install</code> and specify the version you want. If you have <code>pyenv</code> installed, then <code>pipenv</code> will use it to install the specified Python version if necessary.</p>
7410 <p>Abstracting virtual environment management is a noble goal of <code>pipenv</code>, but it does get hung up with hard-to-read errors occasionally. Give it a try, but don’t worry if you feel confused or overwhelmed by it. The tool, documentation, and community will grow and improve around it as it matures.</p>
7411 <p>To get an in-depth introduction to virtual environments, be sure to read <a href="https://realpython.com/python-virtual-environments-a-primer">Python Virtual Environments: A Primer</a>.</p>
7412 <h2 id="package-management">Package Management</h2>
7413 <p>For many of the projects you work on, you’ll probably need some number of third-party packages. Those packages may have their own dependencies in turn. In the early days of Python, using packages involved manually downloading files and pointing Python at them. Today, we’re fortunate to have a variety of package management tools available to us.</p>
7414 <p>Most package managers work in tandem with virtual environments, isolating the packages you install in one Python environment from another. Using the two together is where you really start to see the power of the tools available to you.</p>
7415 <h3 id="pip"><code>pip</code></h3>
7416 <p><code>pip</code> (<strong>p</strong>ip <strong>i</strong>nstalls <strong>p</strong>ackages) has been the de facto standard for package management in Python for several years. It was heavily inspired by an earlier tool called <code>easy_install</code>. Python incorporated <code>pip</code> into the standard distribution starting in version 3.4. <code>pip</code> automates the process of downloading packages and making Python aware of them.</p>
7417 <p>If you have multiple virtual environments, then you can see that they’re isolated by installing a few packages in one:</p>
7418 <div class="highlight sh"><pre><span></span><span class="gp">$</span> pyenv virtualenv <span class="m">3</span>.7.3 proj1
7419 <span class="gp">$</span> pyenv activate proj1
7420 <span class="gp">(proj1)$</span> pip list
7421 <span class="go">Package Version</span>
7422 <span class="go">---------- ---------</span>
7423 <span class="go">pip 19.1.1</span>
7424 <span class="go">setuptools 40.8.0</span>
7425
7426 <span class="gp">(proj1)$</span> python -m pip install requests
7427 <span class="go">Collecting requests</span>
7428 <span class="go"> Downloading .../requests-2.22.0-py2.py3-none-any.whl (57kB)</span>
7429 <span class="go"> 100% |ββββββββββββββββββββββββββββββββ| 61kB 2.2MB/s</span>
7430 <span class="go">Collecting chardet<3.1.0,>=3.0.2 (from requests)</span>
7431 <span class="go"> Downloading .../chardet-3.0.4-py2.py3-none-any.whl (133kB)</span>
7432 <span class="go"> 100% |ββββββββββββββββββββββββββββββββ| 143kB 1.7MB/s</span>
7433 <span class="go">Collecting certifi>=2017.4.17 (from requests)</span>
7434 <span class="go"> Downloading .../certifi-2019.6.16-py2.py3-none-any.whl (157kB)</span>
7435 <span class="go"> 100% |ββββββββββββββββββββββββββββββββ| 163kB 6.0MB/s</span>
7436 <span class="go">Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests)</span>
7437 <span class="go"> Downloading .../urllib3-1.25.3-py2.py3-none-any.whl (150kB)</span>
7438 <span class="go"> 100% |ββββββββββββββββββββββββββββββββ| 153kB 1.7MB/s</span>
7439 <span class="go">Collecting idna<2.9,>=2.5 (from requests)</span>
7440 <span class="go"> Downloading .../idna-2.8-py2.py3-none-any.whl (58kB)</span>
7441 <span class="go"> 100% |ββββββββββββββββββββββββββββββββ| 61kB 26.6MB/s</span>
7442 <span class="go">Installing collected packages: chardet, certifi, urllib3, idna, requests</span>
7443 <span class="go">Successfully installed packages</span>
7444
7445 <span class="gp">$</span> pip list
7446 <span class="go">Package Version</span>
7447 <span class="go">---------- ---------</span>
7448 <span class="go">certifi 2019.6.16</span>
7449 <span class="go">chardet 3.0.4</span>
7450 <span class="go">idna 2.8</span>
7451 <span class="go">pip 19.1.1</span>
7452 <span class="go">requests 2.22.0</span>
7453 <span class="go">setuptools 40.8.0</span>
7454 <span class="go">urllib3 1.25.3</span>
7455 </pre></div>
7456
7457 <p><code>pip</code> installed <code>requests</code>, along with several packages it depends on. <code>pip list</code> shows you all the currently installed packages and their versions.</p>
7458 <div class="alert alert-primary" role="alert">
7459 <p><strong>Warning</strong>: You can uninstall packages using <code>pip uninstall requests</code>, for example, but this will <em>only</em> uninstall <code>requests</code>—not any of its dependencies.</p>
7460 </div>
7461 <p>A common way to specify project dependencies for <code>pip</code> is with a <code>requirements.txt</code> file. Each line in the file specifies a package name and, optionally, the version to install:</p>
7462 <div class="highlight pyreq"><pre><span></span><span class="n">scipy</span><span class="o">==</span><span class="mf">1.3</span><span class="o">.</span><span class="mi">0</span>
7463 <span class="n">requests</span><span class="o">==</span><span class="mf">2.22</span><span class="o">.</span><span class="mi">0</span>
7464 </pre></div>
7465
7466 <p>You can then run <code>python -m pip install -r requirements.txt</code> to install all of the specified dependencies at once. For more on <code>pip</code>, see <a href="https://realpython.com/what-is-pip/">What is Pip? A Guide for New Pythonistas</a>.</p>
7467 <h3 id="pipenv_1"><code>pipenv</code></h3>
7468 <p><a href="https://docs.pipenv.org/en/latest/"><code>pipenv</code></a> has most of the same basic operations as <code>pip</code> but thinks about packages a bit differently. Remember the <code>Pipfile</code> that <code>pipenv</code> creates? When you install a package, <code>pipenv</code> adds that package to <code>Pipfile</code> and also adds more detailed information to a new <strong>lock file</strong> called <code>Pipfile.lock</code>. Lock files act as a snapshot of the precise set of packages installed, including direct dependencies as well as their sub-dependencies.</p>
7469 <p>You can see <code>pipenv</code> sorting out the package management when you install a package:</p>
7470 <div class="highlight sh"><pre><span></span><span class="gp">$</span> pipenv install requests
7471 <span class="go">Installing requestsβ¦</span>
7472 <span class="go">Adding requests to Pipfile's [packages]β¦</span>
7473 <span class="go">β Installation Succeeded</span>
7474 <span class="go">Pipfile.lock (444a6d) out of date, updating to (a65489)β¦</span>
7475 <span class="go">Locking [dev-packages] dependenciesβ¦</span>
7476 <span class="go">Locking [packages] dependenciesβ¦</span>
7477 <span class="go">β Success!</span>
7478 <span class="go">Updated Pipfile.lock (444a6d)!</span>
7479 <span class="go">Installing dependencies from Pipfile.lock (444a6d)β¦</span>
7480 <span class="go"> π ββββββββββββββββββββββββββββββββ 5/5 β 00:00:00</span>
7481 </pre></div>
7482
7483 <p><code>pipenv</code> will use this lock file, if present, to install the same set of packages. You can ensure that you always have the same set of working dependencies in any Python environment you create using this approach.</p>
7484 <p><code>pipenv</code> also distinguishes between <strong>development dependencies</strong> and <strong>production (regular) dependencies</strong>. You may need some tools during development, such as <a href="https://github.com/python/black"><code>black</code></a> or <a href="http://flake8.pycqa.org/en/latest/"><code>flake8</code></a>, that you don’t need when you run your application in production. You can specify that a package is for development when you install it:</p>
7485 <div class="highlight sh"><pre><span></span><span class="gp">$</span> pipenv install --dev flake8
7486 <span class="go">Installing flake8β¦</span>
7487 <span class="go">Adding flake8 to Pipfile's [dev-packages]β¦</span>
7488 <span class="go">β Installation Succeeded</span>
7489 <span class="go">...</span>
7490 </pre></div>
7491
7492 <p><code>pipenv install</code> (without any arguments) will only install your production packages by default, but you can tell it to install development dependencies as well with <code>pipenv install --dev</code>.</p>
7493 <h3 id="poetry"><code>poetry</code></h3>
7494 <p><a href="https://poetry.eustace.io"><code>poetry</code></a> addresses additional facets of package management, including creating and publishing your own packages. After installing <code>poetry</code>, you can use it to create a new project:</p>
7495 <div class="highlight sh"><pre><span></span><span class="gp">$</span> poetry new myproj
7496 <span class="go">Created package myproj in myproj</span>
7497 <span class="gp">$</span> ls myproj/
7498 <span class="go">README.rst myproj pyproject.toml tests</span>
7499 </pre></div>
7500
7501 <p>Similarly to how <code>pipenv</code> creates the <code>Pipfile</code>, <code>poetry</code> creates a <code>pyproject.toml</code> file. This <a href="https://www.python.org/dev/peps/pep-0518/#file-format">recent standard</a> contains metadata about the project as well as dependency versions:</p>
7502 <div class="highlight ini"><pre><span></span><span class="k">[tool.poetry]</span>
7503 <span class="na">name</span> <span class="o">=</span> <span class="s">"myproj"</span>
7504 <span class="na">version</span> <span class="o">=</span> <span class="s">"0.1.0"</span>
7505 <span class="na">description</span> <span class="o">=</span> <span class="s">""</span>
7506 <span class="na">authors</span> <span class="o">=</span> <span class="s">["Dane Hillard <github@danehillard.com>"]</span>
7507
7508 <span class="k">[tool.poetry.dependencies]</span>
7509 <span class="na">python</span> <span class="o">=</span> <span class="s">"^3.7"</span>
7510
7511 <span class="k">[tool.poetry.dev-dependencies]</span>
7512 <span class="na">pytest</span> <span class="o">=</span> <span class="s">"^3.0"</span>
7513
7514 <span class="k">[build-system]</span>
7515 <span class="na">requires</span> <span class="o">=</span> <span class="s">["poetry>=0.12"]</span>
7516 <span class="na">build-backend</span> <span class="o">=</span> <span class="s">"poetry.masonry.api"</span>
7517 </pre></div>
7518
7519 <p>You can install packages with <code>poetry add</code> (or as development dependencies with <code>poetry add --dev</code>):</p>
7520 <div class="highlight sh"><pre><span></span><span class="gp">$</span> poetry add requests
7521 <span class="go">Using version ^2.22 for requests</span>
7522
7523 <span class="go">Updating dependencies</span>
7524 <span class="go">Resolving dependencies... (0.2s)</span>
7525
7526 <span class="go">Writing lock file</span>
7527
7528
7529 <span class="go">Package operations: 5 installs, 0 updates, 0 removals</span>
7530
7531 <span class="go"> - Installing certifi (2019.6.16)</span>
7532 <span class="go"> - Installing chardet (3.0.4)</span>
7533 <span class="go"> - Installing idna (2.8)</span>
7534 <span class="go"> - Installing urllib3 (1.25.3)</span>
7535 <span class="go"> - Installing requests (2.22.0)</span>
7536 </pre></div>
7537
7538 <p><code>poetry</code> also maintains a lock file, and it has a benefit over <code>pipenv</code> because it keeps track of which packages are subdependencies. As a result, you can uninstall <code>requests</code> <em>and</em> its dependencies with <code>poetry remove requests</code>.</p>
7539 <h3 id="conda_2"><code>conda</code></h3>
7540 <p>With <code>conda</code>, you can use <code>pip</code> to install packages as usual, but you can also use <code>conda install</code> to install packages from different <strong>channels </strong>, which are collections of packages provided by Anaconda or other providers. To install <code>requests</code> from the <code>conda-forge</code> channel, you can run <code>conda install -c conda-forge requests</code>.</p>
7541 <p>Learn more about package management in <code>conda</code> in <a href="https://realpython.com/python-windows-machine-learning-setup/">Setting Up Python for Machine Learning on Windows</a>.</p>
7542 <h2 id="python-interpreters">Python Interpreters</h2>
7543 <p>If you’re interested in further customization of your Python environment, you can choose the command line experience you have when interacting with Python. The Python interpreter provides a <strong>read-eval-print loop</strong> (REPL), which is what comes up when you type <code>python</code> with no arguments in your shell:</p>
7544 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="go">Python 3.7.3 (default, Jun 17 2019, 14:09:05)</span>
7545 <span class="go">[Clang 10.0.1 (clang-1001.0.46.4)] on darwin</span>
7546 <span class="go">Type "help", "copyright", "credits" or "license" for more information.</span>
7547 <span class="gp">>>> </span><span class="mi">2</span> <span class="o">+</span> <span class="mi">2</span>
7548 <span class="go">4</span>
7549 <span class="gp">>>> </span><span class="n">exit</span><span class="p">()</span>
7550 </pre></div>
7551
7552 <p>The REPL <strong>reads</strong> what you type, <strong>evaluates</strong> it as Python code, and <strong>prints</strong> the result. Then it waits to do it all over again. This is about as much as the default Python REPL provides, which is sufficient for a good portion of typical work.</p>
7553 <h3 id="ipython">IPython</h3>
7554 <p>Like Anaconda, <a href="https://ipython.org/">IPython</a> is a suite of tools supporting more than just Python, but one of its main features is an alternative Python REPL. IPython’s REPL numbers each command and explicitly labels each command’s input and output. After installing IPython (<code>python -m pip install ipython</code>), you can run the <code>ipython</code> command in place of the <code>python</code> command to use the IPython REPL:</p>
7555 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="go">Python 3.7.3</span>
7556 <span class="go">Type 'copyright', 'credits' or 'license' for more information</span>
7557 <span class="go">IPython 6.0.0.dev -- An enhanced Interactive Python. Type '?' for help.</span>
7558
7559 <span class="gp">In [1]: </span><span class="mi">2</span> <span class="o">+</span> <span class="mi">2</span>
7560 <span class="gh">Out[1]: </span><span class="go">4</span>
7561
7562 <span class="gp">In [2]: </span><span class="nb">print</span><span class="p">(</span><span class="s2">"Hello!"</span><span class="p">)</span>
7563 <span class="gh">Out[2]: </span><span class="go">Hello!</span>
7564 </pre></div>
7565
7566 <p>IPython also supports <span class="keys"><kbd class="key-tab">Tab</kbd></span> completion, more powerful help features, and strong integration with other tooling such as <a href="https://matplotlib.org/"><code>matplotlib</code></a> for graphing. IPython provided the foundation for <a href="https://jupyter.org/">Jupyter</a>, and both have been used extensively in the data science community because of their integration with other tools.</p>
7567 <p>The IPython REPL is <a href="https://ipython.readthedocs.io/en/stable/config/intro.html">highly configurable</a> too, so while it falls just shy of being a full development environment, it can still be a boon to your productivity. Its built-in and customizable <a href="https://ipython.org/ipython-doc/3/interactive/tutorial.html#magic-functions">magic commands</a> are worth checking out.</p>
7568 <h3 id="bpython"><code>bpython</code></h3>
7569 <p><a href="https://bpython-interpreter.org"><code>bpython</code></a> is another alternative REPL that provides inline syntax highlighting, tab completion, and even auto-suggestions as you type. It provides quite a few of the quick benefits of IPython without altering the interface much. Without the weight of the integrations and so on, <code>bpython</code> might be good to add to your repertoire for a while to see how it improves your use of the REPL.</p>
7570 <h2 id="text-editors">Text Editors</h2>
7571 <p>You spend a third of your life sleeping, so it makes sense to invest in a great bed. As a developer, you spend a great deal of your time reading and writing code, so it follows that you should invest time in setting up your Python environment’s text editor just the way you like it.</p>
7572 <p>Each editor offers a different set of key bindings and model for manipulating text. Some require a mouse to interact with them effectively, whereas others can be controlled with only the keyboard. Some people consider their choice of text editor and customizations some of the most personal decisions they make!</p>
7573 <p>There are so many options to choose from in this arena, so I won’t attempt to cover it in detail here. Check out <a href="https://realpython.com/python-ides-code-editors-guide/">Python IDEs and Code Editors (Guide)</a> for a broad overview. A good strategy is to find a simple, small text editor for quick changes and a full-featured IDE for more involved work. <a href="https://www.vim.org/">Vim</a> and <a href="https://www.jetbrains.com/pycharm/">PyCharm</a>, respectively, are my editors of choice.</p>
7574 <h2 id="python-environment-tips-and-tricks">Python Environment Tips and Tricks</h2>
7575 <p>Once you’ve made the big decisions about your Python environment, the rest of the road is paved with little tweaks to make your life a little easier. These tweaks each save minutes or seconds alone, but they collectively save you hours of time.</p>
7576 <p>Making a certain activity easier reduces your cognitive load so you can focus on the task at hand instead of the logistics surrounding it. If you notice yourself performing an action over and over, then consider automating it. Use <a href="https://xkcd.com/1205/">this wonderful chart</a> from XKCD to determine if it’s worth automating a particular task.</p>
7577 <p>Here are a few final tips.</p>
7578 <p><strong>Know your current virtual environment</strong></p>
7579 <p>As mentioned earlier, it’s a great idea to display the active Python version or virtual environment in your command prompt. Most tools will do this for you, but if not (or if you want to customize the prompt), the value is usually contained in the <code>VIRTUAL_ENV</code> environment variable.</p>
7580 <p><strong>Disable unnecessary, temporary files</strong></p>
7581 <p>Have you ever noticed <code>*.pyc</code> files all over your project directories? These files are pre-compiled Python bytecode—they help Python start your application faster. In production, these are a great idea because they’ll give you some performance gain. During local development, however, they’re rarely useful. Set <code>PYTHONDONTWRITEBYTECODE=1</code> to disable this behavior. If you find use cases for them later, then you can easily remove this from your Python environment.</p>
7582 <p><strong>Customize your Python interpreter</strong></p>
7583 <p>You can affect how the REPL behaves using a <strong>startup file</strong>. Python will read this startup file and execute the code it contains before entering the REPL. Set the <code>PYTHONSTARTUP</code> environment variable to the path of your startup file. (Mine’s at <code>~/.pystartup</code>.) If you’d like to hit <span class="keys"><kbd class="key-arrow-up">Up</kbd></span> for command history and <span class="keys"><kbd class="key-tab">Tab</kbd></span> for completion like your shell provides, then give <a href="https://github.com/daneah/dotfiles/blob/master/source/pystartup">this startup file</a> a try.</p>
7584 <h2 id="conclusion">Conclusion</h2>
7585 <p>You learned about many facets of the typical Python environment. Armed with this knowledge, you can:</p>
7586 <ul>
7587 <li>Choose a terminal with the aesthetics and enhanced features you like</li>
7588 <li>Choose a shell with as many (or as few) customization options as you need</li>
7589 <li>Manage multiple versions of Python on your system</li>
7590 <li>Manage multiple projects that use a single version of Python, using virtual Python environments</li>
7591 <li>Install packages in your virtual environments</li>
7592 <li>Choose a REPL that suits your interactive coding needs</li>
7593 </ul>
7594 <p>When you’ve got your Python environment just so, I hope you’ll share screenshots, screencasts, or blog posts about your perfect setup β¨</p>
7595 <hr />
7596 <p><em>[ Improve Your Python With π Python Tricks π β Get a short & sweet Python Trick delivered to your inbox every couple of days. <a href="https://realpython.com/python-tricks/?utm_source=realpython&utm_medium=rss&utm_campaign=footer">>> Click here to learn more and see examples</a> ]</em></p>#
7597 datePublished: #Wed Aug 14 14:00:00 2019#
7598 dateUpdated: #Wed Aug 14 14:00:00 2019#
7599 # Person begin ####################
7600 name: #Real Python#
7601 # Person end ######################
7602 # Item end ########################
7603 # Item begin ######################
7604 id: #https://realpython.com/courses/traditional-face-detection-python/#
7605 title: #Traditional Face Detection With Python#
7606 link: #https://realpython.com/courses/traditional-face-detection-python/#
7607 description: #In this course on face detection with Python, you'll learn about a historically important algorithm for object detection that can be successfully applied to finding the location of a human face within an image.#
7608 content: #<p><strong>Computer vision</strong> is an exciting and growing field. There are tons of interesting problems to solve! One of them is face detection: the ability of a computer to recognize that a photograph contains a human face, and tell you where it is located. In this course, you’ll learn about <strong>face detection</strong> with Python.</p>
7609 <p>To detect any object in an image, it is necessary to understand how images are represented inside a computer, and how that object differs <em>visually</em> from any other object.</p>
7610 <p>Once that is done, the process of scanning an image and looking for those visual cues needs to be automated and optimized. All these steps come together to form a fast and reliable computer vision algorithm.</p>
7611 <p><strong>In this course, you’ll learn:</strong></p>
7612 <ul>
7613 <li>What face detection is</li>
7614 <li>How computers understand features in images</li>
7615 <li>How to quickly analyze many different features to reach a decision</li>
7616 <li>How to use a minimal Python solution for detecting human faces in images</li>
7617 </ul>
7618 <hr />
7619 <p><em>[ Improve Your Python With π Python Tricks π β Get a short & sweet Python Trick delivered to your inbox every couple of days. <a href="https://realpython.com/python-tricks/?utm_source=realpython&utm_medium=rss&utm_campaign=footer">>> Click here to learn more and see examples</a> ]</em></p>#
7620 datePublished: #Tue Aug 13 14:00:00 2019#
7621 dateUpdated: #Tue Aug 13 14:00:00 2019#
7622 # Person begin ####################
7623 name: #Real Python#
7624 # Person end ######################
7625 # Item end ########################
7626 # Item begin ######################
7627 id: #https://realpython.com/python-print/#
7628 title: #Your Guide to the Python Print Function#
7629 link: #https://realpython.com/python-print/#
7630 description: #In this step-by-step tutorial, you'll learn about the print() function in Python and discover some of its lesser-known features. Avoid common mistakes, take your "hello world" to the next level, and know when to use a better alternative.#
7631 content: #<p>If you’re like most Python users, including me, then you probably started your Python journey by learning about <code>print()</code>. It helped you write your very own <code>hello world</code> one-liner. You can use it to display formatted messages onto the screen and perhaps find some bugs. But if you think that’s all there is to know about Python’s <code>print()</code> function, then you’re missing out on a lot!</p>
7632 <p>Keep reading to take full advantage of this seemingly boring and unappreciated little function. This tutorial will get you up to speed with using Python <code>print()</code> effectively. However, prepare for a deep dive as you go through the sections. You may be surprised how much <code>print()</code> has to offer!</p>
7633 <p><strong>By the end of this tutorial, you’ll know how to:</strong></p>
7634 <ul>
7635 <li>Avoid common mistakes with Python’s <code>print()</code></li>
7636 <li>Deal with newlines, character encodings, and buffering</li>
7637 <li>Write text to files</li>
7638 <li>Mock <code>print()</code> in unit tests</li>
7639 <li>Build advanced user interfaces in the terminal</li>
7640 </ul>
7641 <p>If you’re a complete beginner, then you’ll benefit most from reading the first part of this tutorial, which illustrates the essentials of printing in Python. Otherwise, feel free to skip that part and jump around as you see fit.</p>
7642 <div class="alert alert-primary" role="alert">
7643 <p><strong>Note:</strong> <code>print()</code> was a major addition to Python 3, in which it replaced the old <code>print</code> statement available in Python 2.</p>
7644 <p>There were a number of good reasons for that, as you’ll see shortly. Although this tutorial focuses on Python 3, it does show the old way of printing in Python for reference.</p>
7645 </div>
7646 <div class="alert alert-warning" role="alert"><p><strong>Free Bonus:</strong> <a href="" class="alert-link" data-toggle="modal" data-target="#modal-python-cheat-sheet-experiment" data-focus="false">Click here to get our free Python Cheat Sheet</a> that shows you the basics of Python 3, like working with data types, dictionaries, lists, and Python functions.</p></div>
7647
7648 <h2 id="printing-in-a-nutshell">Printing in a Nutshell</h2>
7649 <p>Let’s jump in by looking at a few real-life examples of printing in Python. By the end of this section, you’ll know every possible way of calling <code>print()</code>. Or, in programmer lingo, you’d say you’ll be familiar with the <strong>function signature</strong>.</p>
7650 <h3 id="calling-print">Calling Print</h3>
7651 <p>The simplest example of using Python <code>print()</code> requires just a few keystrokes:</p>
7652 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="nb">print</span><span class="p">()</span>
7653 </pre></div>
7654
7655 <p>You don’t pass any arguments, but you still need to put empty parentheses at the end, which tell Python to actually <a href="https://realpython.com/lessons/example-function/">execute the function</a> rather than just refer to it by name.</p>
7656 <p>This will produce an invisible newline character, which in turn will cause a blank line to appear on your screen. You can call <code>print()</code> multiple times like this to add vertical space. It’s just as if you were hitting <span class="keys"><kbd class="key-enter">Enter</kbd></span> on your keyboard in a word processor.</p>
7657 <div class="card mb-3" id="collapse_card5787b1">
7658 <div class="card-header border-0"><p class="m-0"><button class="btn" data-toggle="collapse" data-target="#collapse5787b1" aria-expanded="false" aria-controls="collapse5787b1">Newline Character</button> <button class="btn btn-link float-right" data-toggle="collapse" data-target="#collapse5787b1" aria-expanded="false" aria-controls="collapse5787b1">Show/Hide</button></p></div>
7659 <div id="collapse5787b1" class="collapse" data-parent="#collapse_card5787b1"><div class="card-body" markdown="1">
7660
7661 <p>A <strong>newline character</strong> is a special control character used to indicate the end of a line (EOL). It usually doesn’t have a visible representation on the screen, but some text editors can display such non-printable characters with little graphics.</p>
7662 <p>The word “character” is somewhat of a misnomer in this case, because a newline is often more than one character long. For example, the Windows operating system, as well as the HTTP protocol, represent newlines with a pair of characters. Sometimes you need to take those differences into account to design truly portable programs.</p>
7663 <p>To find out what constitutes a newline in your operating system, use Python’s built-in <code>os</code> module.</p>
7664 <p>This will immediately tell you that <strong>Windows</strong> and <strong>DOS</strong> represent the newline as a sequence of <code>\r</code> followed by <code>\n</code>:</p>
7665 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">os</span>
7666 <span class="gp">>>> </span><span class="n">os</span><span class="o">.</span><span class="n">linesep</span>
7667 <span class="go">'\r\n'</span>
7668 </pre></div>
7669
7670 <p>On <strong>Unix</strong>, <strong>Linux</strong>, and recent versions of <strong>macOS</strong>, it’s a single <code>\n</code> character:</p>
7671 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">os</span>
7672 <span class="gp">>>> </span><span class="n">os</span><span class="o">.</span><span class="n">linesep</span>
7673 <span class="go">'\n'</span>
7674 </pre></div>
7675
7676 <p>The classic <strong>Mac OS X</strong>, however, sticks to its own “think different” philosophy by choosing yet another representation:</p>
7677 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">os</span>
7678 <span class="gp">>>> </span><span class="n">os</span><span class="o">.</span><span class="n">linesep</span>
7679 <span class="go">'\r'</span>
7680 </pre></div>
7681
7682 <p>Notice how these characters appear in string literals. They use special syntax with a preceding backslash (<code>\</code>) to denote the start of an <strong>escape character sequence</strong>. Such sequences allow for representing control characters, which would be otherwise invisible on screen.</p>
7683 <p>Most programming languages come with a predefined set of escape sequences for special characters such as these:</p>
7684 <ul>
7685 <li><strong><code>\\</code>:</strong> backslash</li>
7686 <li><strong><code>\b</code>:</strong> backspace</li>
7687 <li><strong><code>\t</code>:</strong> tab</li>
7688 <li><strong><code>\r</code>:</strong> carriage return (CR)</li>
7689 <li><strong><code>\n</code>:</strong> newline, also known as line feed (LF)</li>
7690 </ul>
7691 <p>The last two are reminiscent of mechanical typewriters, which required two separate commands to insert a newline. The first command would move the carriage back to the beginning of the current line, while the second one would advance the roll to the next line.</p>
7692 <p>By comparing the corresponding <strong>ASCII character codes</strong>, you’ll see that putting a backslash in front of a character changes its meaning completely. However, not all characters allow for this–only the special ones.</p>
7693 <p>To compare ASCII character codes, you may want to use the built-in <code>ord()</code> function:</p>
7694 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="nb">ord</span><span class="p">(</span><span class="s1">'r'</span><span class="p">)</span>
7695 <span class="go">114</span>
7696 <span class="gp">>>> </span><span class="nb">ord</span><span class="p">(</span><span class="s1">'</span><span class="se">\r</span><span class="s1">'</span><span class="p">)</span>
7697 <span class="go">13</span>
7698 </pre></div>
7699
7700 <p>Keep in mind that, in order to form a correct escape sequence, there must be no space between the backslash character and a letter!</p>
7701 </div></div>
7702
7703 </div>
7704 <p>As you just saw, calling <code>print()</code> without arguments results in a <strong>blank line</strong>, which is a line comprised solely of the newline character. Don’t confuse this with an <strong>empty line</strong>, which doesn’t contain any characters at all, not even the newline!</p>
7705 <p>You can use Python’s <a href="https://realpython.com/python-strings/">string</a> literals to visualize these two:</p>
7706 <div class="highlight python"><pre><span></span><span class="s1">'</span><span class="se">\n</span><span class="s1">'</span> <span class="c1"># Blank line</span>
7707 <span class="s1">''</span> <span class="c1"># Empty line</span>
7708 </pre></div>
7709
7710 <p>The first one is one character long, whereas the second one has no content.</p>
7711 <div class="alert alert-primary" role="alert">
7712 <p><strong>Note:</strong> To remove the newline character from a string in Python, use its <code>.rstrip()</code> method, like this:</p>
7713 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="s1">'A line of text.</span><span class="se">\n</span><span class="s1">'</span><span class="o">.</span><span class="n">rstrip</span><span class="p">()</span>
7714 <span class="go">'A line of text.'</span>
7715 </pre></div>
7716
7717 <p>This strips any trailing whitespace from the right edge of the string of characters.</p>
7718 </div>
7719 <p>In a more common scenario, you’d want to communicate some message to the end user. There are a few ways to achieve this.</p>
7720 <p>First, you may pass a string literal directly to <code>print()</code>:</p>
7721 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="s1">'Please wait while the program is loading...'</span><span class="p">)</span>
7722 </pre></div>
7723
7724 <p>This will print the message verbatim onto the screen.</p>
7725 <div class="card mb-3" id="collapse_cardf395ab">
7726 <div class="card-header border-0"><p class="m-0"><button class="btn" data-toggle="collapse" data-target="#collapsef395ab" aria-expanded="false" aria-controls="collapsef395ab">String Literals</button> <button class="btn btn-link float-right" data-toggle="collapse" data-target="#collapsef395ab" aria-expanded="false" aria-controls="collapsef395ab">Show/Hide</button></p></div>
7727 <div id="collapsef395ab" class="collapse" data-parent="#collapse_cardf395ab"><div class="card-body" markdown="1">
7728
7729 <p><strong>String literals</strong> in Python can be enclosed either in single quotes (<code>'</code>) or double quotes (<code>"</code>). According to the official <a href="https://www.python.org/dev/peps/pep-0008/#string-quotes">PEP 8</a> style guide, you should just pick one and keep using it consistently. There’s no difference, unless you need to nest one in another.</p>
7730 <p>For example, you can’t use double quotes for the literal and also include double quotes inside of it, because that’s ambiguous for the Python interpreter:</p>
7731 <div class="highlight python"><pre><span></span><span class="s2">"My favorite book is "</span><span class="n">Python</span> <span class="n">Tricks</span><span class="s2">""</span> <span class="c1"># Wrong!</span>
7732 </pre></div>
7733
7734 <p>What you want to do is enclose the text, which contains double quotes, within single quotes:</p>
7735 <div class="highlight python"><pre><span></span><span class="s1">'My favorite book is "Python Tricks"'</span>
7736 </pre></div>
7737
7738 <p>The same trick would work the other way around:</p>
7739 <div class="highlight python"><pre><span></span><span class="s2">"My favorite book is 'Python Tricks'"</span>
7740 </pre></div>
7741
7742 <p>Alternatively, you could use escape character sequences mentioned earlier, to make Python treat those internal double quotes literally as part of the string literal:</p>
7743 <div class="highlight python"><pre><span></span><span class="s2">"My favorite book is </span><span class="se">\"</span><span class="s2">Python Tricks</span><span class="se">\"</span><span class="s2">"</span>
7744 </pre></div>
7745
7746 <p>Escaping is fine and dandy, but it can sometimes get in the way. Specifically, when you need your string to contain relatively many backslash characters in literal form.</p>
7747 <p>One classic example is a file path on Windows:</p>
7748 <div class="highlight python"><pre><span></span><span class="s1">'C:\Users\jdoe'</span> <span class="c1"># Wrong!</span>
7749 <span class="s1">'C:</span><span class="se">\\</span><span class="s1">Users</span><span class="se">\\</span><span class="s1">jdoe'</span>
7750 </pre></div>
7751
7752 <p>Notice how each backslash character needs to be escaped with yet another backslash.</p>
7753 <p>This is even more prominent with regular expressions, which quickly get convoluted due to the heavy use of special characters:</p>
7754 <div class="highlight python"><pre><span></span><span class="s1">'^</span><span class="se">\\</span><span class="s1">w:</span><span class="se">\\\\</span><span class="s1">(?:(?:(?:[^</span><span class="se">\\\\</span><span class="s1">]+)?|(?:[^</span><span class="se">\\\\</span><span class="s1">]+)</span><span class="se">\\\\</span><span class="s1">[^</span><span class="se">\\\\</span><span class="s1">]+)*)$'</span>
7755 </pre></div>
7756
7757 <p>Fortunately, you can turn off character escaping entirely with the help of raw-string literals. Simply prepend an <code>r</code> or <code>R</code> before the opening quote, and now you end up with this:</p>
7758 <div class="highlight python"><pre><span></span><span class="sa">r</span><span class="s1">'C:\Users\jdoe'</span>
7759 <span class="sa">r</span><span class="s1">'^\w:</span><span class="se">\\</span><span class="s1">(?:(?:(?:[^</span><span class="se">\\</span><span class="s1">]+)?|(?:[^</span><span class="se">\\</span><span class="s1">]+)</span><span class="se">\\</span><span class="s1">[^</span><span class="se">\\</span><span class="s1">]+)*)$'</span>
7760 </pre></div>
7761
7762 <p>That’s much better, isn’t it?</p>
7763 <p>There are a few more prefixes that give special meaning to string literals in Python, but you won’t get into them here.</p>
7764 <p>Lastly, you can define multi-line string literals by enclosing them between <code>'''</code> or <code>"""</code>, which are often used as <strong>docstrings</strong>.</p>
7765 <p>Here’s an example:</p>
7766 <div class="highlight python"><pre><span></span><span class="sd">"""</span>
7767 <span class="sd">This is an example</span>
7768 <span class="sd">of a multi-line string</span>
7769 <span class="sd">in Python.</span>
7770 <span class="sd">"""</span>
7771 </pre></div>
7772
7773 <p>To prevent an initial newline, simply put the text right after the opening <code>"""</code>:</p>
7774 <div class="highlight python"><pre><span></span><span class="sd">"""This is an example</span>
7775 <span class="sd">of a multi-line string</span>
7776 <span class="sd">in Python.</span>
7777 <span class="sd">"""</span>
7778 </pre></div>
7779
7780 <p>You can also use a backslash to get rid of the newline:</p>
7781 <div class="highlight python"><pre><span></span><span class="sd">"""\</span>
7782 <span class="sd">This is an example</span>
7783 <span class="sd">of a multi-line string</span>
7784 <span class="sd">in Python.</span>
7785 <span class="sd">"""</span>
7786 </pre></div>
7787
7788 <p>To remove indentation from a multi-line string, you might take advantage of the built-in <code>textwrap</code> module:</p>
7789 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">textwrap</span>
7790 <span class="gp">>>> </span><span class="n">paragraph</span> <span class="o">=</span> <span class="s1">'''</span>
7791 <span class="gp">... </span><span class="s1"> This is an example</span>
7792 <span class="gp">... </span><span class="s1"> of a multi-line string</span>
7793 <span class="gp">... </span><span class="s1"> in Python.</span>
7794 <span class="gp">... </span><span class="s1"> '''</span>
7795 <span class="gp">...</span>
7796 <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">paragraph</span><span class="p">)</span>
7797
7798 <span class="go"> This is an example</span>
7799 <span class="go"> of a multi-line string</span>
7800 <span class="go"> in Python.</span>
7801
7802 <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">textwrap</span><span class="o">.</span><span class="n">dedent</span><span class="p">(</span><span class="n">paragraph</span><span class="p">)</span><span class="o">.</span><span class="n">strip</span><span class="p">())</span>
7803 <span class="go">This is an example</span>
7804 <span class="go">of a multi-line string</span>
7805 <span class="go">in Python.</span>
7806 </pre></div>
7807
7808 <p>This will take care of unindenting paragraphs for you. There are also a few other useful functions in <code>textwrap</code> for text alignment you’d find in a word processor.</p>
7809 </div></div>
7810
7811 </div>
7812 <p>Secondly, you could extract that message into its own variable with a meaningful name to enhance readability and promote code reuse:</p>
7813 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">message</span> <span class="o">=</span> <span class="s1">'Please wait while the program is loading...'</span>
7814 <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">message</span><span class="p">)</span>
7815 </pre></div>
7816
7817 <p>Lastly, you could pass an expression, like <a href="https://realpython.com/lessons/concatenating-joining-strings-python/">string concatenation</a>, to be evaluated before printing the result:</p>
7818 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">os</span>
7819 <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="s1">'Hello, '</span> <span class="o">+</span> <span class="n">os</span><span class="o">.</span><span class="n">getlogin</span><span class="p">()</span> <span class="o">+</span> <span class="s1">'! How are you?'</span><span class="p">)</span>
7820 <span class="go">Hello, jdoe! How are you?</span>
7821 </pre></div>
7822
7823 <p>In fact, there are a dozen ways to format messages in Python. I highly encourage you to take a look at <a href="https://realpython.com/python-f-strings/">f-strings</a>, introduced in Python 3.6, because they offer the most concise syntax of them all:</p>
7824 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">os</span>
7825 <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Hello, {os.getlogin()}! How are you?'</span><span class="p">)</span>
7826 </pre></div>
7827
7828 <p>Moreover, f-strings will prevent you from making a common mistake, which is forgetting to type cast concatenated operands. Python is a strongly typed language, which means it won’t allow you to do this:</p>
7829 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="s1">'My age is '</span> <span class="o">+</span> <span class="mi">42</span>
7830 <span class="gt">Traceback (most recent call last):</span>
7831 File <span class="nb">"<input>"</span>, line <span class="m">1</span>, in <span class="n"><module></span>
7832 <span class="s1">'My age is '</span> <span class="o">+</span> <span class="mi">42</span>
7833 <span class="gr">TypeError</span>: <span class="n">can only concatenate str (not "int") to str</span>
7834 </pre></div>
7835
7836 <p>That’s wrong because adding numbers to strings doesn’t make sense. You need to explicitly convert the number to string first, in order to join them together:</p>
7837 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="s1">'My age is '</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="mi">42</span><span class="p">)</span>
7838 <span class="go">'My age is 42'</span>
7839 </pre></div>
7840
7841 <p>Unless you <a href="https://realpython.com/courses/python-exceptions-101/">handle such errors</a> yourself, the Python interpreter will let you know about a problem by showing a <a href="https://realpython.com/python-traceback/">traceback</a>.</p>
7842 <div class="alert alert-primary" role="alert">
7843 <p><strong>Note:</strong> <code>str()</code> is a global built-in function that converts an object into its string representation.</p>
7844 <p>You can call it directly on any object, for example, a number:</p>
7845 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="nb">str</span><span class="p">(</span><span class="mf">3.14</span><span class="p">)</span>
7846 <span class="go">'3.14'</span>
7847 </pre></div>
7848
7849 <p>Built-in data types have a predefined string representation out of the box, but later in this article, you’ll find out how to provide one for your custom classes.</p>
7850 </div>
7851 <p>As with any function, it doesn’t matter whether you pass a literal, a variable, or an expression. Unlike many other functions, however, <code>print()</code> will accept anything regardless of its type.</p>
7852 <p>So far, you only looked at the string, but how about other data types? Let’s try literals of different built-in types and see what comes out:</p>
7853 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="mi">42</span><span class="p">)</span> <span class="c1"># <class 'int'></span>
7854 <span class="go">42</span>
7855 <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="mf">3.14</span><span class="p">)</span> <span class="c1"># <class 'float'></span>
7856 <span class="go">3.14</span>
7857 <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="mi">1</span> <span class="o">+</span> <span class="mi">2</span><span class="n">j</span><span class="p">)</span> <span class="c1"># <class 'complex'></span>
7858 <span class="go">(1+2j)</span>
7859 <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="kc">True</span><span class="p">)</span> <span class="c1"># <class 'bool'></span>
7860 <span class="go">True</span>
7861 <span class="gp">>>> </span><span class="nb">print</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">])</span> <span class="c1"># <class 'list'></span>
7862 <span class="go">[1, 2, 3]</span>
7863 <span class="gp">>>> </span><span class="nb">print</span><span class="p">((</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">))</span> <span class="c1"># <class 'tuple'></span>
7864 <span class="go">(1, 2, 3)</span>
7865 <span class="gp">>>> </span><span class="nb">print</span><span class="p">({</span><span class="s1">'red'</span><span class="p">,</span> <span class="s1">'green'</span><span class="p">,</span> <span class="s1">'blue'</span><span class="p">})</span> <span class="c1"># <class 'set'></span>
7866 <span class="go">{'red', 'green', 'blue'}</span>
7867 <span class="gp">>>> </span><span class="nb">print</span><span class="p">({</span><span class="s1">'name'</span><span class="p">:</span> <span class="s1">'Alice'</span><span class="p">,</span> <span class="s1">'age'</span><span class="p">:</span> <span class="mi">42</span><span class="p">})</span> <span class="c1"># <class 'dict'></span>
7868 <span class="go">{'name': 'Alice', 'age': 42}</span>
7869 <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="s1">'hello'</span><span class="p">)</span> <span class="c1"># <class 'str'></span>
7870 <span class="go">hello</span>
7871 </pre></div>
7872
7873 <p>Watch out for the <code>None</code> constant, though. Despite being used to indicate an absence of a value, it will show up as <code>'None'</code> rather than an empty string:</p>
7874 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="kc">None</span><span class="p">)</span>
7875 <span class="go">None</span>
7876 </pre></div>
7877
7878 <p>How does <code>print()</code> know how to work with all these different types? Well, the short answer is that it doesn’t. It implicitly calls <code>str()</code> behind the scenes to type cast any object into a string. Afterward, it treats strings in a uniform way.</p>
7879 <p>Later in this tutorial, you’ll learn how to use this mechanism for printing custom data types such as your classes.</p>
7880 <p>Okay, you’re now able to call <code>print()</code> with a single argument or without any arguments. You know how to print fixed or formatted messages onto the screen. The next subsection will expand on message formatting a little bit.</p>
7881 <div class="card mb-3" id="collapse_card09ba8e">
7882 <div class="card-header border-0"><p class="m-0"><button class="btn" data-toggle="collapse" data-target="#collapse09ba8e" aria-expanded="false" aria-controls="collapse09ba8e">Syntax in Python 2</button> <button class="btn btn-link float-right" data-toggle="collapse" data-target="#collapse09ba8e" aria-expanded="false" aria-controls="collapse09ba8e">Show/Hide</button></p></div>
7883 <div id="collapse09ba8e" class="collapse" data-parent="#collapse_card09ba8e"><div class="card-body" markdown="1">
7884
7885 <p>To achieve the same result in the previous language generation, you’d normally want to drop the parentheses enclosing the text:</p>
7886 <div class="highlight python"><pre><span></span><span class="c1"># Python 2</span>
7887 <span class="k">print</span>
7888 <span class="k">print</span> <span class="s1">'Please wait...'</span>
7889 <span class="k">print</span> <span class="s1">'Hello, </span><span class="si">%s</span><span class="s1">! How are you?'</span> <span class="o">%</span> <span class="n">os</span><span class="o">.</span><span class="n">getlogin</span><span class="p">()</span>
7890 <span class="k">print</span> <span class="s1">'Hello, </span><span class="si">%s</span><span class="s1">. Your age is </span><span class="si">%d</span><span class="s1">.'</span> <span class="o">%</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">age</span><span class="p">)</span>
7891 </pre></div>
7892
7893 <p>That’s because <code>print</code> wasn’t a function back then, as you’ll see in the <a href="#understanding-python-print">next section</a>. Note, however, that in some cases parentheses in Python are redundant. It wouldn’t harm to include them as they’d just get ignored. Does that mean you should be using the <code>print</code> statement as if it were a function? Absolutely not!</p>
7894 <p>For example, parentheses enclosing a single expression or a literal are optional. Both instructions produce the same result in Python 2:</p>
7895 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="c1"># Python 2</span>
7896 <span class="gp">>>> </span><span class="k">print</span> <span class="s1">'Please wait...'</span>
7897 <span class="go">Please wait...</span>
7898 <span class="gp">>>> </span><span class="k">print</span><span class="p">(</span><span class="s1">'Please wait...'</span><span class="p">)</span>
7899 <span class="go">Please wait...</span>
7900 </pre></div>
7901
7902 <p>Round brackets are actually part of the expression rather than the <code>print</code> statement. If your expression happens to contain only one item, then it’s as if you didn’t include the brackets at all.</p>
7903 <p>On the other hand, putting parentheses around multiple items forms a <a href="https://realpython.com/python-lists-tuples/#python-tuples">tuple</a>:</p>
7904 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="c1"># Python 2</span>
7905 <span class="gp">>>> </span><span class="k">print</span> <span class="s1">'My name is'</span><span class="p">,</span> <span class="s1">'John'</span>
7906 <span class="go">My name is John</span>
7907 <span class="gp">>>> </span><span class="k">print</span><span class="p">(</span><span class="s1">'My name is'</span><span class="p">,</span> <span class="s1">'John'</span><span class="p">)</span>
7908 <span class="go">('My name is', 'John')</span>
7909 </pre></div>
7910
7911 <p>This is a known source of confusion. In fact, you’d also get a tuple by appending a trailing comma to the only item surrounded by parentheses:</p>
7912 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="c1"># Python 2</span>
7913 <span class="gp">>>> </span><span class="k">print</span><span class="p">(</span><span class="s1">'Please wait...'</span><span class="p">)</span>
7914 <span class="go">Please wait...</span>
7915 <span class="gp">>>> </span><span class="k">print</span><span class="p">(</span><span class="s1">'Please wait...'</span><span class="p">,)</span> <span class="c1"># Notice the comma</span>
7916 <span class="go">('Please wait...',)</span>
7917 </pre></div>
7918
7919 <p>The bottom line is that you shouldn’t call <code>print</code> with brackets in Python 2. Although, to be completely accurate, you can work around this with the help of a <code>__future__</code> import, which you’ll read more about in the relevant section.</p>
7920 </div></div>
7921
7922 </div>
7923 <h3 id="separating-multiple-arguments">Separating Multiple Arguments</h3>
7924 <p>You saw <code>print()</code> called without any arguments to produce a blank line and then called with a single argument to display either a fixed or a formatted message.</p>
7925 <p>However, it turns out that this function can accept any number of <strong>positional arguments</strong>, including zero, one, or more arguments. That’s very handy in a common case of message formatting, where you’d want to join a few elements together.</p>
7926 <div class="card mb-3" id="collapse_card9620b8">
7927 <div class="card-header border-0"><p class="m-0"><button class="btn" data-toggle="collapse" data-target="#collapse9620b8" aria-expanded="false" aria-controls="collapse9620b8">Positional Arguments</button> <button class="btn btn-link float-right" data-toggle="collapse" data-target="#collapse9620b8" aria-expanded="false" aria-controls="collapse9620b8">Show/Hide</button></p></div>
7928 <div id="collapse9620b8" class="collapse" data-parent="#collapse_card9620b8"><div class="card-body" markdown="1">
7929
7930 <p>Arguments can be passed to a function in one of several ways. One way is by explicitly naming the arguments when you’re calling the function, like this:</p>
7931 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">def</span> <span class="nf">div</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">):</span>
7932 <span class="gp">... </span> <span class="k">return</span> <span class="n">a</span> <span class="o">/</span> <span class="n">b</span>
7933 <span class="gp">...</span>
7934 <span class="gp">>>> </span><span class="n">div</span><span class="p">(</span><span class="n">a</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span> <span class="n">b</span><span class="o">=</span><span class="mi">4</span><span class="p">)</span>
7935 <span class="go">0.75</span>
7936 </pre></div>
7937
7938 <p>Since arguments can be uniquely identified by name, their order doesn’t matter. Swapping them out will still give the same result:</p>
7939 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">div</span><span class="p">(</span><span class="n">b</span><span class="o">=</span><span class="mi">4</span><span class="p">,</span> <span class="n">a</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span>
7940 <span class="go">0.75</span>
7941 </pre></div>
7942
7943 <p>Conversely, arguments passed without names are identified by their position. That’s why <strong>positional arguments</strong> need to follow strictly the order imposed by the function signature:</p>
7944 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">div</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">)</span>
7945 <span class="go">0.75</span>
7946 <span class="gp">>>> </span><span class="n">div</span><span class="p">(</span><span class="mi">4</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span>
7947 <span class="go">1.3333333333333333</span>
7948 </pre></div>
7949
7950 <p><code>print()</code> allows an <a href="https://docs.python.org/dev/tutorial/controlflow.html#arbitrary-argument-lists">arbitrary number of positional arguments</a> thanks to the <code>*args</code> parameter.</p>
7951 </div></div>
7952
7953 </div>
7954 <p>Let’s have a look at this example:</p>
7955 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">os</span>
7956 <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="s1">'My name is'</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">getlogin</span><span class="p">(),</span> <span class="s1">'and I am'</span><span class="p">,</span> <span class="mi">42</span><span class="p">)</span>
7957 <span class="go">My name is jdoe and I am 42</span>
7958 </pre></div>
7959
7960 <p><code>print()</code> concatenated all four arguments passed to it, and it inserted a single space between them so that you didn’t end up with a squashed message like <code>'My name isjdoeand I am42'</code>.</p>
7961 <p>Notice that it also took care of proper type casting by implicitly calling <code>str()</code> on each argument before joining them together. If you recall from the previous subsection, a naΓ―ve concatenation may easily result in an error due to incompatible types:</p>
7962 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="s1">'My age is: '</span> <span class="o">+</span> <span class="mi">42</span><span class="p">)</span>
7963 <span class="gt">Traceback (most recent call last):</span>
7964 File <span class="nb">"<input>"</span>, line <span class="m">1</span>, in <span class="n"><module></span>
7965 <span class="nb">print</span><span class="p">(</span><span class="s1">'My age is: '</span> <span class="o">+</span> <span class="mi">42</span><span class="p">)</span>
7966 <span class="gr">TypeError</span>: <span class="n">can only concatenate str (not "int") to str</span>
7967 </pre></div>
7968
7969 <p>Apart from accepting a variable number of positional arguments, <code>print()</code> defines four named or <strong>keyword arguments</strong>, which are optional since they all have default values. You can view their brief documentation by calling <code>help(print)</code> from the interactive interpreter.</p>
7970 <p>Let’s focus on <code>sep</code> just for now. It stands for <strong>separator</strong> and is assigned a single space (<code>' '</code>) by default. It determines the value to join elements with.</p>
7971 <p>It has to be either a string or <code>None</code>, but the latter has the same effect as the default space:</p>
7972 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="s1">'hello'</span><span class="p">,</span> <span class="s1">'world'</span><span class="p">,</span> <span class="n">sep</span><span class="o">=</span><span class="kc">None</span><span class="p">)</span>
7973 <span class="go">hello world</span>
7974 <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="s1">'hello'</span><span class="p">,</span> <span class="s1">'world'</span><span class="p">,</span> <span class="n">sep</span><span class="o">=</span><span class="s1">' '</span><span class="p">)</span>
7975 <span class="go">hello world</span>
7976 <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="s1">'hello'</span><span class="p">,</span> <span class="s1">'world'</span><span class="p">)</span>
7977 <span class="go">hello world</span>
7978 </pre></div>
7979
7980 <p>If you wanted to suppress the separator completely, you’d have to pass an empty string (<code>''</code>) instead:</p>
7981 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="s1">'hello'</span><span class="p">,</span> <span class="s1">'world'</span><span class="p">,</span> <span class="n">sep</span><span class="o">=</span><span class="s1">''</span><span class="p">)</span>
7982 <span class="go">helloworld</span>
7983 </pre></div>
7984
7985 <p>You may want <code>print()</code> to join its arguments as separate lines. In that case, simply pass the escaped newline character described earlier:</p>
7986 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="s1">'hello'</span><span class="p">,</span> <span class="s1">'world'</span><span class="p">,</span> <span class="n">sep</span><span class="o">=</span><span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">)</span>
7987 <span class="go">hello</span>
7988 <span class="go">world</span>
7989 </pre></div>
7990
7991 <p>A more useful example of the <code>sep</code> parameter would be printing something like file paths:</p>
7992 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="s1">'home'</span><span class="p">,</span> <span class="s1">'user'</span><span class="p">,</span> <span class="s1">'documents'</span><span class="p">,</span> <span class="n">sep</span><span class="o">=</span><span class="s1">'/'</span><span class="p">)</span>
7993 <span class="go">home/user/documents</span>
7994 </pre></div>
7995
7996 <p>Remember that the separator comes between the elements, not around them, so you need to account for that in one way or another:</p>
7997 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="s1">'/home'</span><span class="p">,</span> <span class="s1">'user'</span><span class="p">,</span> <span class="s1">'documents'</span><span class="p">,</span> <span class="n">sep</span><span class="o">=</span><span class="s1">'/'</span><span class="p">)</span>
7998 <span class="go">/home/user/documents</span>
7999 <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="s1">''</span><span class="p">,</span> <span class="s1">'home'</span><span class="p">,</span> <span class="s1">'user'</span><span class="p">,</span> <span class="s1">'documents'</span><span class="p">,</span> <span class="n">sep</span><span class="o">=</span><span class="s1">'/'</span><span class="p">)</span>
8000 <span class="go">/home/user/documents</span>
8001 </pre></div>
8002
8003 <p>Specifically, you can insert a slash character (<code>/</code>) into the first positional argument, or use an empty string as the first argument to enforce the leading slash.</p>
8004 <div class="alert alert-primary" role="alert">
8005 <p><strong>Note:</strong> Be careful about joining elements of a list or tuple.</p>
8006 <p>Doing it manually will result in a well-known <code>TypeError</code> if at least one of the elements isn’t a string:</p>
8007 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="s1">' '</span><span class="o">.</span><span class="n">join</span><span class="p">([</span><span class="s1">'jdoe is'</span><span class="p">,</span> <span class="mi">42</span><span class="p">,</span> <span class="s1">'years old'</span><span class="p">]))</span>
8008 <span class="gt">Traceback (most recent call last):</span>
8009 File <span class="nb">"<input>"</span>, line <span class="m">1</span>, in <span class="n"><module></span>
8010 <span class="nb">print</span><span class="p">(</span><span class="s1">','</span><span class="o">.</span><span class="n">join</span><span class="p">([</span><span class="s1">'jdoe is'</span><span class="p">,</span> <span class="mi">42</span><span class="p">,</span> <span class="s1">'years old'</span><span class="p">]))</span>
8011 <span class="gr">TypeError</span>: <span class="n">sequence item 1: expected str instance, int found</span>
8012 </pre></div>
8013
8014 <p>It’s safer to just unpack the sequence with the star operator (<code>*</code>) and let <code>print()</code> handle type casting:</p>
8015 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="o">*</span><span class="p">[</span><span class="s1">'jdoe is'</span><span class="p">,</span> <span class="mi">42</span><span class="p">,</span> <span class="s1">'years old'</span><span class="p">])</span>
8016 <span class="go">jdoe is 42 years old</span>
8017 </pre></div>
8018
8019 <p>Unpacking is effectively the same as calling <code>print()</code> with individual elements of the list.</p>
8020 </div>
8021 <p>One more interesting example could be exporting data to a <a href="https://realpython.com/courses/reading-and-writing-csv-files/">comma-separated values</a> (CSV) format:</p>
8022 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="s1">'Python Tricks'</span><span class="p">,</span> <span class="s1">'Dan Bader'</span><span class="p">,</span> <span class="n">sep</span><span class="o">=</span><span class="s1">','</span><span class="p">)</span>
8023 <span class="go">1,Python Tricks,Dan Bader</span>
8024 </pre></div>
8025
8026 <p>This wouldn’t handle edge cases such as escaping commas correctly, but for simple use cases, it should do. The line above would show up in your terminal window. In order to save it to a file, you’d have to redirect the output. Later in this section, you’ll see how to use <code>print()</code> to write text to files straight from Python.</p>
8027 <p>Finally, the <code>sep</code> parameter isn’t constrained to a single character only. You can join elements with strings of any length:</p>
8028 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="s1">'node'</span><span class="p">,</span> <span class="s1">'child'</span><span class="p">,</span> <span class="s1">'child'</span><span class="p">,</span> <span class="n">sep</span><span class="o">=</span><span class="s1">' -> '</span><span class="p">)</span>
8029 <span class="go">node -> child -> child</span>
8030 </pre></div>
8031
8032 <p>In the upcoming subsections, you’ll explore the remaining keyword arguments of the <code>print()</code> function.</p>
8033 <div class="card mb-3" id="collapse_cardab50a5">
8034 <div class="card-header border-0"><p class="m-0"><button class="btn" data-toggle="collapse" data-target="#collapseab50a5" aria-expanded="false" aria-controls="collapseab50a5">Syntax in Python 2</button> <button class="btn btn-link float-right" data-toggle="collapse" data-target="#collapseab50a5" aria-expanded="false" aria-controls="collapseab50a5">Show/Hide</button></p></div>
8035 <div id="collapseab50a5" class="collapse" data-parent="#collapse_cardab50a5"><div class="card-body" markdown="1">
8036
8037 <p>To print multiple elements in Python 2, you must drop the parentheses around them, just like before:</p>
8038 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="c1"># Python 2</span>
8039 <span class="gp">>>> </span><span class="kn">import</span> <span class="nn">os</span>
8040 <span class="gp">>>> </span><span class="k">print</span> <span class="s1">'My name is'</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">getlogin</span><span class="p">(),</span> <span class="s1">'and I am'</span><span class="p">,</span> <span class="mi">42</span>
8041 <span class="go">My name is jdoe and I am 42</span>
8042 </pre></div>
8043
8044 <p>If you kept them, on the other hand, you’d be passing a single tuple element to the <code>print</code> statement:</p>
8045 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="c1"># Python 2</span>
8046 <span class="gp">>>> </span><span class="kn">import</span> <span class="nn">os</span>
8047 <span class="gp">>>> </span><span class="k">print</span><span class="p">(</span><span class="s1">'My name is'</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">getlogin</span><span class="p">(),</span> <span class="s1">'and I am'</span><span class="p">,</span> <span class="mi">42</span><span class="p">)</span>
8048 <span class="go">('My name is', 'jdoe', 'and I am', 42)</span>
8049 </pre></div>
8050
8051 <p>Moreover, there’s no way of altering the default separator of joined elements in Python 2, so one workaround is to use string interpolation like so:</p>
8052 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="c1"># Python 2</span>
8053 <span class="gp">>>> </span><span class="kn">import</span> <span class="nn">os</span>
8054 <span class="gp">>>> </span><span class="k">print</span> <span class="s1">'My name is </span><span class="si">%s</span><span class="s1"> and I am </span><span class="si">%d</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getlogin</span><span class="p">(),</span> <span class="mi">42</span><span class="p">)</span>
8055 <span class="go">My name is jdoe and I am 42</span>
8056 </pre></div>
8057
8058 <p>That was the default way of formatting strings until the <code>.format()</code> method got backported from Python 3.</p>
8059 </div></div>
8060
8061 </div>
8062 <h3 id="preventing-line-breaks">Preventing Line Breaks</h3>
8063 <p>Sometimes you don’t want to end your message with a trailing newline so that subsequent calls to <code>print()</code> will continue on the same line. Classic examples include updating the progress of a long-running operation or prompting the user for input. In the latter case, you want the user to type in the answer on the same line:</p>
8064 <div class="highlight text"><pre><span></span>Are you sure you want to do this? [y/n] y
8065 </pre></div>
8066
8067 <p>Many programming languages expose functions similar to <code>print()</code> through their standard libraries, but they let you decide whether to add a newline or not. For example, in Java and C#, you have two distinct functions, while other languages require you to explicitly append <code>\n</code> at the end of a string literal.</p>
8068 <p>Here are a few examples of syntax in such languages:</p>
8069 <div class="table-responsive">
8070 <table class="table table-hover">
8071 <thead>
8072 <tr>
8073 <th>Language</th>
8074 <th>Example</th>
8075 </tr>
8076 </thead>
8077 <tbody>
8078 <tr>
8079 <td>Perl</td>
8080 <td><code>print "hello world\n"</code></td>
8081 </tr>
8082 <tr>
8083 <td>C</td>
8084 <td><code>printf("hello world\n");</code></td>
8085 </tr>
8086 <tr>
8087 <td>C++</td>
8088 <td><code>std::cout << "hello world" << std::endl;</code></td>
8089 </tr>
8090 </tbody>
8091 </table>
8092 </div>
8093 <p>In contrast, Python’s <code>print()</code> function always adds <code>\n</code> without asking, because that’s what you want in most cases. To disable it, you can take advantage of yet another keyword argument, <code>end</code>, which dictates what to end the line with.</p>
8094 <p>In terms of semantics, the <code>end</code> parameter is almost identical to the <code>sep</code> one that you saw earlier:</p>
8095 <ul>
8096 <li>It must be a string or <code>None</code>.</li>
8097 <li>It can be arbitrarily long.</li>
8098 <li>It has a default value of <code>'\n'</code>.</li>
8099 <li>If equal to <code>None</code>, it’ll have the same effect as the default value.</li>
8100 <li>If equal to an empty string (<code>''</code>), it’ll suppress the newline.</li>
8101 </ul>
8102 <p>Now you understand what’s happening under the hood when you’re calling <code>print()</code> without arguments. Since you don’t provide any positional arguments to the function, there’s nothing to be joined, and so the default separator isn’t used at all. However, the default value of <code>end</code> still applies, and a blank line shows up.</p>
8103 <div class="alert alert-primary" role="alert">
8104 <p><strong>Note:</strong> You may be wondering why the <code>end</code> parameter has a fixed default value rather than whatever makes sense on your operating system.</p>
8105 <p>Well, you don’t have to worry about newline representation across different operating systems when printing, because <code>print()</code> will handle the conversion automatically. Just remember to always use the <code>\n</code> escape sequence in string literals.</p>
8106 <p>This is currently the most portable way of printing a newline character in Python:</p>
8107 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="s1">'line1</span><span class="se">\n</span><span class="s1">line2</span><span class="se">\n</span><span class="s1">line3'</span><span class="p">)</span>
8108 <span class="go">line1</span>
8109 <span class="go">line2</span>
8110 <span class="go">line3</span>
8111 </pre></div>
8112
8113 <p>If you were to try to forcefully print a Windows-specific newline character on a Linux machine, for example, you’d end up with broken output:</p>
8114 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="s1">'line1</span><span class="se">\r\n</span><span class="s1">line2</span><span class="se">\r\n</span><span class="s1">line3'</span><span class="p">)</span>
8115
8116
8117 <span class="go">line3</span>
8118 </pre></div>
8119
8120 <p>On the flip side, when you open a file for reading with <code>open()</code>, you don’t need to care about newline representation either. The function will translate any system-specific newline it encounters into a universal <code>'\n'</code>. At the same time, you have control over how the newlines should be treated both on input and output if you really need that.</p>
8121 </div>
8122 <p>To disable the newline, you must specify an empty string through the <code>end</code> keyword argument:</p>
8123 <div class="highlight python"><pre><span></span><span class="nb">print</span><span class="p">(</span><span class="s1">'Checking file integrity...'</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s1">''</span><span class="p">)</span>
8124 <span class="c1"># (...)</span>
8125 <span class="nb">print</span><span class="p">(</span><span class="s1">'ok'</span><span class="p">)</span>
8126 </pre></div>
8127
8128 <p>Even though these are two separate <code>print()</code> calls, which can execute a long time apart, you’ll eventually see only one line. First, it’ll look like this:</p>
8129 <div class="highlight text"><pre><span></span>Checking file integrity...
8130 </pre></div>
8131
8132 <p>However, after the second call to <code>print()</code>, the same line will appear on the screen as:</p>
8133 <div class="highlight text"><pre><span></span>Checking file integrity...ok
8134 </pre></div>
8135
8136 <p>As with <code>sep</code>, you can use <code>end</code> to join individual pieces into a big blob of text with a custom separator. Instead of joining multiple arguments, however, it’ll append text from each function call to the same line:</p>
8137 <div class="highlight python"><pre><span></span><span class="nb">print</span><span class="p">(</span><span class="s1">'The first sentence'</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s1">'. '</span><span class="p">)</span>
8138 <span class="nb">print</span><span class="p">(</span><span class="s1">'The second sentence'</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s1">'. '</span><span class="p">)</span>
8139 <span class="nb">print</span><span class="p">(</span><span class="s1">'The last sentence.'</span><span class="p">)</span>
8140 </pre></div>
8141
8142 <p>These three instructions will output a single line of text:</p>
8143 <div class="highlight text"><pre><span></span>The first sentence. The second sentence. The last sentence.
8144 </pre></div>
8145
8146 <p>You can mix the two keyword arguments:</p>
8147 <div class="highlight python"><pre><span></span><span class="nb">print</span><span class="p">(</span><span class="s1">'Mercury'</span><span class="p">,</span> <span class="s1">'Venus'</span><span class="p">,</span> <span class="s1">'Earth'</span><span class="p">,</span> <span class="n">sep</span><span class="o">=</span><span class="s1">', '</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s1">', '</span><span class="p">)</span>
8148 <span class="nb">print</span><span class="p">(</span><span class="s1">'Mars'</span><span class="p">,</span> <span class="s1">'Jupiter'</span><span class="p">,</span> <span class="s1">'Saturn'</span><span class="p">,</span> <span class="n">sep</span><span class="o">=</span><span class="s1">', '</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s1">', '</span><span class="p">)</span>
8149 <span class="nb">print</span><span class="p">(</span><span class="s1">'Uranus'</span><span class="p">,</span> <span class="s1">'Neptune'</span><span class="p">,</span> <span class="s1">'Pluto'</span><span class="p">,</span> <span class="n">sep</span><span class="o">=</span><span class="s1">', '</span><span class="p">)</span>
8150 </pre></div>
8151
8152 <p>Not only do you get a single line of text, but all items are separated with a comma:</p>
8153 <div class="highlight text"><pre><span></span>Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune, Pluto
8154 </pre></div>
8155
8156 <p>There’s nothing to stop you from using the newline character with some extra padding around it:</p>
8157 <div class="highlight python"><pre><span></span><span class="nb">print</span><span class="p">(</span><span class="s1">'Printing in a Nutshell'</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s1">'</span><span class="se">\n</span><span class="s1"> * '</span><span class="p">)</span>
8158 <span class="nb">print</span><span class="p">(</span><span class="s1">'Calling Print'</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s1">'</span><span class="se">\n</span><span class="s1"> * '</span><span class="p">)</span>
8159 <span class="nb">print</span><span class="p">(</span><span class="s1">'Separating Multiple Arguments'</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s1">'</span><span class="se">\n</span><span class="s1"> * '</span><span class="p">)</span>
8160 <span class="nb">print</span><span class="p">(</span><span class="s1">'Preventing Line Breaks'</span><span class="p">)</span>
8161 </pre></div>
8162
8163 <p>It would print out the following piece of text:</p>
8164 <div class="highlight text"><pre><span></span>Printing in a Nutshell
8165 * Calling Print
8166 * Separating Multiple Arguments
8167 * Preventing Line Breaks
8168 </pre></div>
8169
8170 <p>As you can see, the <code>end</code> keyword argument will accept arbitrary strings.</p>
8171 <div class="alert alert-primary" role="alert">
8172 <p><strong>Note:</strong> Looping over lines in a text file preserves their own newline characters, which combined with the <code>print()</code> function’s default behavior will result in a redundant newline character:</p>
8173 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'file.txt'</span><span class="p">)</span> <span class="k">as</span> <span class="n">file_object</span><span class="p">:</span>
8174 <span class="gp">... </span> <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">file_object</span><span class="p">:</span>
8175 <span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
8176 <span class="gp">...</span>
8177 <span class="go">Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod</span>
8178
8179 <span class="go">tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,</span>
8180
8181 <span class="go">quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo</span>
8182 </pre></div>
8183
8184 <p>There are two newlines after each line of text. You want to strip one of the them, as shown earlier in this article, before printing the line:</p>
8185 <div class="highlight python"><pre><span></span><span class="nb">print</span><span class="p">(</span><span class="n">line</span><span class="o">.</span><span class="n">rstrip</span><span class="p">())</span>
8186 </pre></div>
8187
8188 <p>Alternatively, you can keep the newline in the content but suppress the one appended by <code>print()</code> automatically. You’d use the <code>end</code> keyword argument to do that:</p>
8189 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'file.txt'</span><span class="p">)</span> <span class="k">as</span> <span class="n">file_object</span><span class="p">:</span>
8190 <span class="gp">... </span> <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">file_object</span><span class="p">:</span>
8191 <span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">line</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s1">''</span><span class="p">)</span>
8192 <span class="gp">...</span>
8193 <span class="go">Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod</span>
8194 <span class="go">tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,</span>
8195 <span class="go">quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo</span>
8196 </pre></div>
8197
8198 <p>By ending a line with an empty string, you effectively disable one of the newlines.</p>
8199 </div>
8200 <p>You’re getting more acquainted with printing in Python, but there’s still a lot of useful information ahead. In the upcoming subsection, you’ll learn how to intercept and redirect the <code>print()</code> function’s output.</p>
8201 <div class="card mb-3" id="collapse_card37ad87">
8202 <div class="card-header border-0"><p class="m-0"><button class="btn" data-toggle="collapse" data-target="#collapse37ad87" aria-expanded="false" aria-controls="collapse37ad87">Syntax in Python 2</button> <button class="btn btn-link float-right" data-toggle="collapse" data-target="#collapse37ad87" aria-expanded="false" aria-controls="collapse37ad87">Show/Hide</button></p></div>
8203 <div id="collapse37ad87" class="collapse" data-parent="#collapse_card37ad87"><div class="card-body" markdown="1">
8204
8205 <p>Preventing a line break in Python 2 requires that you append a trailing comma to the expression:</p>
8206 <div class="highlight python"><pre><span></span><span class="k">print</span> <span class="s1">'hello world'</span><span class="p">,</span>
8207 </pre></div>
8208
8209 <p>However, that’s not ideal because it also adds an unwanted space, which would translate to <code>end=' '</code> instead of <code>end=''</code> in Python 3. You can test this with the following code snippet:</p>
8210 <div class="highlight python"><pre><span></span><span class="k">print</span> <span class="s1">'BEFORE'</span>
8211 <span class="k">print</span> <span class="s1">'hello'</span><span class="p">,</span>
8212 <span class="k">print</span> <span class="s1">'AFTER'</span>
8213 </pre></div>
8214
8215 <p>Notice there’s a space between the words <code>hello</code> and <code>AFTER</code>:</p>
8216 <div class="highlight text"><pre><span></span>BEFORE
8217 hello AFTER
8218 </pre></div>
8219
8220 <p>In order to get the expected result, you’d need to use one of the tricks explained later, which is either importing the <code>print()</code> function from <code>__future__</code> or falling back to the <code>sys</code> module:</p>
8221 <div class="highlight python"><pre><span></span><span class="kn">import</span> <span class="nn">sys</span>
8222 <span class="k">print</span> <span class="s1">'BEFORE'</span>
8223 <span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'hello'</span><span class="p">)</span>
8224 <span class="k">print</span> <span class="s1">'AFTER'</span>
8225 </pre></div>
8226
8227 <p>This will print the correct output without extra space:</p>
8228 <div class="highlight text"><pre><span></span>BEFORE
8229 helloAFTER
8230 </pre></div>
8231
8232 <p>While using the <code>sys</code> module gives you control over what gets printed to the standard output, the code becomes a little bit more cluttered.</p>
8233 </div></div>
8234
8235 </div>
8236 <h3 id="printing-to-a-file">Printing to a File</h3>
8237 <p>Believe it or not, <code>print()</code> doesn’t know how to turn messages into text on your screen, and frankly it doesn’t need to. That’s a job for lower-level layers of code, which understand bytes and know how to push them around.</p>
8238 <p><code>print()</code> is an abstraction over these layers, providing a convenient interface that merely delegates the actual printing to a stream or <strong>file-like object</strong>. A stream can be any file on your disk, a network socket, or perhaps an in-memory buffer.</p>
8239 <p>In addition to this, there are three standard streams provided by the operating system:</p>
8240 <ol>
8241 <li><strong><code>stdin</code>:</strong> standard input</li>
8242 <li><strong><code>stdout</code>:</strong> standard output</li>
8243 <li><strong><code>stderr</code>:</strong> standard error</li>
8244 </ol>
8245 <div class="card mb-3" id="collapse_cardd994b2">
8246 <div class="card-header border-0"><p class="m-0"><button class="btn" data-toggle="collapse" data-target="#collapsed994b2" aria-expanded="false" aria-controls="collapsed994b2">Standard Streams</button> <button class="btn btn-link float-right" data-toggle="collapse" data-target="#collapsed994b2" aria-expanded="false" aria-controls="collapsed994b2">Show/Hide</button></p></div>
8247 <div id="collapsed994b2" class="collapse" data-parent="#collapse_cardd994b2"><div class="card-body" markdown="1">
8248
8249 <p><strong>Standard output</strong> is what you see in the terminal when you run various command-line programs including your own <a href="https://realpython.com/run-python-scripts/">Python scripts</a>:</p>
8250 <div class="highlight sh"><pre><span></span><span class="gp">$</span> cat hello.py
8251 <span class="go">print('This will appear on stdout')</span>
8252 <span class="gp">$</span> python hello.py
8253 <span class="go">This will appear on stdout</span>
8254 </pre></div>
8255
8256 <p>Unless otherwise instructed, <code>print()</code> will default to writing to standard output. However, you can tell your operating system to temporarily swap out <code>stdout</code> for a file stream, so that any output ends up in that file rather than the screen:</p>
8257 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python hello.py > file.txt
8258 <span class="gp">$</span> cat file.txt
8259 <span class="go">This will appear on stdout</span>
8260 </pre></div>
8261
8262 <p>That’s called stream redirection.</p>
8263 <p>The standard error is similar to <code>stdout</code> in that it also shows up on the screen. Nonetheless, it’s a separate stream, whose purpose is to log error messages for diagnostics. By redirecting one or both of them, you can keep things clean.</p>
8264 <div class="alert alert-primary" role="alert">
8265 <p><strong>Note:</strong> To redirect <code>stderr</code>, you need to know about <strong>file descriptors</strong>, also known as <strong>file handles</strong>.</p>
8266 <p>They’re arbitrary, albeit constant, numbers associated with standard streams. Below, you’ll find a summary of the file descriptors for a family of POSIX-compliant operating systems:</p>
8267 <div class="table-responsive">
8268 <table class="table table-hover">
8269 <thead>
8270 <tr>
8271 <th>Stream</th>
8272 <th>File Descriptor</th>
8273 </tr>
8274 </thead>
8275 <tbody>
8276 <tr>
8277 <td><code>stdin</code></td>
8278 <td>0</td>
8279 </tr>
8280 <tr>
8281 <td><code>stdout</code></td>
8282 <td>1</td>
8283 </tr>
8284 <tr>
8285 <td><code>stderr</code></td>
8286 <td>2</td>
8287 </tr>
8288 </tbody>
8289 </table>
8290 </div>
8291 <p>Knowing those descriptors allows you to redirect one or more streams at a time:</p>
8292 <div class="table-responsive">
8293 <table class="table table-hover">
8294 <thead>
8295 <tr>
8296 <th>Command</th>
8297 <th>Description</th>
8298 </tr>
8299 </thead>
8300 <tbody>
8301 <tr>
8302 <td><code>./program > out.txt</code></td>
8303 <td>Redirect <code>stdout</code></td>
8304 </tr>
8305 <tr>
8306 <td><code>./program 2> err.txt</code></td>
8307 <td>Redirect <code>stderr</code></td>
8308 </tr>
8309 <tr>
8310 <td><code>./program > out.txt 2> err.txt</code></td>
8311 <td>Redirect <code>stdout</code> and <code>stderr</code> to separate files</td>
8312 </tr>
8313 <tr>
8314 <td><code>./program &> out_err.txt</code></td>
8315 <td>Redirect <code>stdout</code> and <code>stderr</code> to the same file</td>
8316 </tr>
8317 </tbody>
8318 </table>
8319 </div>
8320 <p>Note that <code>></code> is the same as <code>1></code>.</p>
8321 </div>
8322 <p>Some programs use different coloring to distinguish between messages printed to <code>stdout</code> and <code>stderr</code>:</p>
8323 <figure class="figure mx-auto d-block"><a href="https://files.realpython.com/media/pycharm-console-streams.69affb3462e4.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/pycharm-console-streams.69affb3462e4.png" width="999" height="286" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-console-streams.69affb3462e4.png&w=249&sig=6f17bc7db6fe5ea5fb93f3575f95a4d46eb88af0 249w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/pycharm-console-streams.69affb3462e4.png&w=499&sig=ccfce4ba25553f534f07f65b483f9af2b106109b 499w, https://files.realpython.com/media/pycharm-console-streams.69affb3462e4.png 999w" sizes="75vw" alt="The output of a program executed in PyCharm"/></a><figcaption class="figure-caption text-center">Run Tool Window in PyCharm</figcaption></figure>
8324
8325
8326 <p>While both <code>stdout</code> and <code>stderr</code> are write-only, <code>stdin</code> is read-only. You can think of standard input as your keyboard, but just like with the other two, you can swap out <code>stdin</code> for a file to read data from.</p>
8327 </div></div>
8328
8329 </div>
8330 <p>In Python, you can access all standard streams through the built-in <code>sys</code> module:</p>
8331 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">sys</span>
8332 <span class="gp">>>> </span><span class="n">sys</span><span class="o">.</span><span class="n">stdin</span>
8333 <span class="go"><_io.TextIOWrapper name='<stdin>' mode='r' encoding='UTF-8'></span>
8334 <span class="gp">>>> </span><span class="n">sys</span><span class="o">.</span><span class="n">stdin</span><span class="o">.</span><span class="n">fileno</span><span class="p">()</span>
8335 <span class="go">0</span>
8336 <span class="gp">>>> </span><span class="n">sys</span><span class="o">.</span><span class="n">stdout</span>
8337 <span class="go"><_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'></span>
8338 <span class="gp">>>> </span><span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">fileno</span><span class="p">()</span>
8339 <span class="go">1</span>
8340 <span class="gp">>>> </span><span class="n">sys</span><span class="o">.</span><span class="n">stderr</span>
8341 <span class="go"><_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'></span>
8342 <span class="gp">>>> </span><span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="o">.</span><span class="n">fileno</span><span class="p">()</span>
8343 <span class="go">2</span>
8344 </pre></div>
8345
8346 <p>As you can see, these predefined values resemble file-like objects with <code>mode</code> and <code>encoding</code> attributes as well as <code>.read()</code> and <code>.write()</code> methods among many others.</p>
8347 <p>By default, <code>print()</code> is bound to <code>sys.stdout</code> through its <code>file</code> argument, but you can change that. Use that keyword argument to indicate a file that was open in write or append mode, so that messages go straight to it:</p>
8348 <div class="highlight python"><pre><span></span><span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'file.txt'</span><span class="p">,</span> <span class="n">mode</span><span class="o">=</span><span class="s1">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">file_object</span><span class="p">:</span>
8349 <span class="nb">print</span><span class="p">(</span><span class="s1">'hello world'</span><span class="p">,</span> <span class="n">file</span><span class="o">=</span><span class="n">file_object</span><span class="p">)</span>
8350 </pre></div>
8351
8352 <p>This will make your code immune to stream redirection at the operating system level, which might or might not be desired.</p>
8353 <p>For more information on <a href="https://realpython.com/working-with-files-in-python/">working with files in Python</a>, you can check out <a href="https://realpython.com/read-write-files-python">Reading and Writing Files in Python (Guide)</a>.</p>
8354 <div class="alert alert-primary" role="alert">
8355 <p><strong>Note:</strong> Don’t try using <code>print()</code> for writing binary data as it’s only well suited for text.</p>
8356 <p>Just call the binary file’s <code>.write()</code> directly:</p>
8357 <div class="highlight python"><pre><span></span><span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'file.dat'</span><span class="p">,</span> <span class="s1">'wb'</span><span class="p">)</span> <span class="k">as</span> <span class="n">file_object</span><span class="p">:</span>
8358 <span class="n">file_object</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="nb">bytes</span><span class="p">(</span><span class="mi">4</span><span class="p">))</span>
8359 <span class="n">file_object</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="sa">b</span><span class="s1">'</span><span class="se">\xff</span><span class="s1">'</span><span class="p">)</span>
8360 </pre></div>
8361
8362 <p>If you wanted to write raw bytes on the standard output, then this will fail too because <code>sys.stdout</code> is a character stream:</p>
8363 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">sys</span>
8364 <span class="gp">>>> </span><span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="nb">bytes</span><span class="p">(</span><span class="mi">4</span><span class="p">))</span>
8365 <span class="gt">Traceback (most recent call last):</span>
8366 File <span class="nb">"<stdin>"</span>, line <span class="m">1</span>, in <span class="n"><module></span>
8367 <span class="gr">TypeError</span>: <span class="n">write() argument must be str, not bytes</span>
8368 </pre></div>
8369
8370 <p>You must dig deeper to get a handle of the underlying byte stream instead:</p>
8371 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">sys</span>
8372 <span class="gp">>>> </span><span class="n">num_bytes_written</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">buffer</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="sa">b</span><span class="s1">'</span><span class="se">\x41\x0a</span><span class="s1">'</span><span class="p">)</span>
8373 <span class="go">A</span>
8374 </pre></div>
8375
8376 <p>This prints an uppercase letter <code>A</code> and a newline character, which correspond to decimal values of 65 and 10 in ASCII. However, they’re encoded using hexadecimal notation in the bytes literal.</p>
8377 </div>
8378 <p>Note that <code>print()</code> has no control over <a href="https://realpython.com/python-encodings-guide/">character encoding</a>. It’s the stream’s responsibility to encode received Unicode strings into bytes correctly. In most cases, you won’t set the encoding yourself, because the default UTF-8 is what you want. If you really need to, perhaps for legacy systems, you can use the <code>encoding</code> argument of <code>open()</code>:</p>
8379 <div class="highlight python"><pre><span></span><span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'file.txt'</span><span class="p">,</span> <span class="n">mode</span><span class="o">=</span><span class="s1">'w'</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s1">'iso-8859-1'</span><span class="p">)</span> <span class="k">as</span> <span class="n">file_object</span><span class="p">:</span>
8380 <span class="nb">print</span><span class="p">(</span><span class="s1">'ΓΌber naΓ―ve cafΓ©'</span><span class="p">,</span> <span class="n">file</span><span class="o">=</span><span class="n">file_object</span><span class="p">)</span>
8381 </pre></div>
8382
8383 <p>Instead of a real file existing somewhere in your file system, you can provide a fake one, which would reside in your computer’s memory. You’ll use this technique later for mocking <code>print()</code> in unit tests:</p>
8384 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">io</span>
8385 <span class="gp">>>> </span><span class="n">fake_file</span> <span class="o">=</span> <span class="n">io</span><span class="o">.</span><span class="n">StringIO</span><span class="p">()</span>
8386 <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="s1">'hello world'</span><span class="p">,</span> <span class="n">file</span><span class="o">=</span><span class="n">fake_file</span><span class="p">)</span>
8387 <span class="gp">>>> </span><span class="n">fake_file</span><span class="o">.</span><span class="n">getvalue</span><span class="p">()</span>
8388 <span class="go">'hello world\n'</span>
8389 </pre></div>
8390
8391 <p>If you got to this point, then you’re left with only one keyword argument in <code>print()</code>, which you’ll see in the next subsection. It’s probably the least used of them all. Nevertheless, there are times when it’s absolutely necessary.</p>
8392 <div class="card mb-3" id="collapse_card80f59e">
8393 <div class="card-header border-0"><p class="m-0"><button class="btn" data-toggle="collapse" data-target="#collapse80f59e" aria-expanded="false" aria-controls="collapse80f59e">Syntax in Python 2</button> <button class="btn btn-link float-right" data-toggle="collapse" data-target="#collapse80f59e" aria-expanded="false" aria-controls="collapse80f59e">Show/Hide</button></p></div>
8394 <div id="collapse80f59e" class="collapse" data-parent="#collapse_card80f59e"><div class="card-body" markdown="1">
8395
8396 <p>There’s a special syntax in Python 2 for replacing the default <code>sys.stdout</code> with a custom file in the <code>print</code> statement:</p>
8397 <div class="highlight python"><pre><span></span><span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'file.txt'</span><span class="p">,</span> <span class="n">mode</span><span class="o">=</span><span class="s1">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">file_object</span><span class="p">:</span>
8398 <span class="k">print</span> <span class="o">>></span> <span class="n">file_object</span><span class="p">,</span> <span class="s1">'hello world'</span>
8399 </pre></div>
8400
8401 <p>Because strings and bytes are represented with the same <code>str</code> type in Python 2, the <code>print</code> statement can handle binary data just fine:</p>
8402 <div class="highlight python"><pre><span></span><span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'file.dat'</span><span class="p">,</span> <span class="n">mode</span><span class="o">=</span><span class="s1">'wb'</span><span class="p">)</span> <span class="k">as</span> <span class="n">file_object</span><span class="p">:</span>
8403 <span class="k">print</span> <span class="o">>></span> <span class="n">file_object</span><span class="p">,</span> <span class="s1">'</span><span class="se">\x41\x0a</span><span class="s1">'</span>
8404 </pre></div>
8405
8406 <p>Although, there’s a problem with character encoding. The <code>open()</code> function in Python 2 lacks the <code>encoding</code> parameter, which would often result in the dreadful <code>UnicodeEncodeError</code>:</p>
8407 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'file.txt'</span><span class="p">,</span> <span class="n">mode</span><span class="o">=</span><span class="s1">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">file_object</span><span class="p">:</span>
8408 <span class="gp">... </span> <span class="n">unicode_text</span> <span class="o">=</span> <span class="sa">u</span><span class="s1">'</span><span class="se">\xfc</span><span class="s1">ber na</span><span class="se">\xef</span><span class="s1">ve caf</span><span class="se">\xe9</span><span class="s1">'</span>
8409 <span class="gp">... </span> <span class="k">print</span> <span class="o">>></span> <span class="n">file_object</span><span class="p">,</span> <span class="n">unicode_text</span>
8410 <span class="gp">... </span>
8411 <span class="gt">Traceback (most recent call last):</span>
8412 File <span class="nb">"<stdin>"</span>, line <span class="m">3</span>, in <span class="n"><module></span>
8413 <span class="gr">UnicodeEncodeError</span>: <span class="n">'ascii' codec can't encode character u'\xfc'...</span>
8414 </pre></div>
8415
8416 <p>Notice how non-Latin characters must be escaped in both Unicode and string literals to avoid a syntax error. Take a look at this example:</p>
8417 <div class="highlight python"><pre><span></span><span class="n">unicode_literal</span> <span class="o">=</span> <span class="sa">u</span><span class="s1">'</span><span class="se">\xfc</span><span class="s1">ber na</span><span class="se">\xef</span><span class="s1">ve caf</span><span class="se">\xe9</span><span class="s1">'</span>
8418 <span class="n">string_literal</span> <span class="o">=</span> <span class="s1">'</span><span class="se">\xc3\xbc</span><span class="s1">ber na</span><span class="se">\xc3\xaf</span><span class="s1">ve caf</span><span class="se">\xc3\xa9</span><span class="s1">'</span>
8419 </pre></div>
8420
8421 <p>Alternatively, you could specify source code encoding according to <a href="https://www.python.org/dev/peps/pep-0263/">PEP 263</a> at the top of the file, but that wasn’t the best practice due to portability issues:</p>
8422 <div class="highlight python"><pre><span></span><span class="ch">#!/usr/bin/env python2</span>
8423 <span class="c1"># -*- coding: utf-8 -*-</span>
8424
8425 <span class="n">unescaped_unicode_literal</span> <span class="o">=</span> <span class="sa">u</span><span class="s1">'ΓΌber naΓ―ve cafΓ©'</span>
8426 <span class="n">unescaped_string_literal</span> <span class="o">=</span> <span class="s1">'ΓΌber naΓ―ve cafΓ©'</span>
8427 </pre></div>
8428
8429 <p>Your best bet is to encode the Unicode string just before printing it. You can do this manually:</p>
8430 <div class="highlight python"><pre><span></span><span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'file.txt'</span><span class="p">,</span> <span class="n">mode</span><span class="o">=</span><span class="s1">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">file_object</span><span class="p">:</span>
8431 <span class="n">unicode_text</span> <span class="o">=</span> <span class="sa">u</span><span class="s1">'</span><span class="se">\xfc</span><span class="s1">ber na</span><span class="se">\xef</span><span class="s1">ve caf</span><span class="se">\xe9</span><span class="s1">'</span>
8432 <span class="n">encoded_text</span> <span class="o">=</span> <span class="n">unicode_text</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">)</span>
8433 <span class="k">print</span> <span class="o">>></span> <span class="n">file_object</span><span class="p">,</span> <span class="n">encoded_text</span>
8434 </pre></div>
8435
8436 <p>However, a more convenient option is to use the built-in <code>codecs</code> module:</p>
8437 <div class="highlight python"><pre><span></span><span class="kn">import</span> <span class="nn">codecs</span>
8438
8439 <span class="k">with</span> <span class="n">codecs</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="s1">'file.txt'</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s1">'utf-8'</span><span class="p">)</span> <span class="k">as</span> <span class="n">file_object</span><span class="p">:</span>
8440 <span class="n">unicode_text</span> <span class="o">=</span> <span class="sa">u</span><span class="s1">'</span><span class="se">\xfc</span><span class="s1">ber na</span><span class="se">\xef</span><span class="s1">ve caf</span><span class="se">\xe9</span><span class="s1">'</span>
8441 <span class="k">print</span> <span class="o">>></span> <span class="n">file_object</span><span class="p">,</span> <span class="n">unicode_text</span>
8442 </pre></div>
8443
8444 <p>It’ll take care of making appropriate conversions when you need to read or write files.</p>
8445 </div></div>
8446
8447 </div>
8448 <h3 id="buffering-print-calls">Buffering Print Calls</h3>
8449 <p>In the previous subsection, you learned that <code>print()</code> delegates printing to a file-like object such as <code>sys.stdout</code>. Some streams, however, buffer certain I/O operations to enhance performance, which can get in the way. Let’s take a look at an example.</p>
8450 <p>Imagine you were writing a countdown timer, which should append the remaining time to the same line every second:</p>
8451 <div class="highlight text"><pre><span></span>3...2...1...Go!
8452 </pre></div>
8453
8454 <p>Your first attempt may look something like this:</p>
8455 <div class="highlight python"><pre><span></span><span class="kn">import</span> <span class="nn">time</span>
8456
8457 <span class="n">num_seconds</span> <span class="o">=</span> <span class="mi">3</span>
8458 <span class="k">for</span> <span class="n">countdown</span> <span class="ow">in</span> <span class="nb">reversed</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="n">num_seconds</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)):</span>
8459 <span class="k">if</span> <span class="n">countdown</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span>
8460 <span class="nb">print</span><span class="p">(</span><span class="n">countdown</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s1">'...'</span><span class="p">)</span>
8461 <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
8462 <span class="k">else</span><span class="p">:</span>
8463 <span class="nb">print</span><span class="p">(</span><span class="s1">'Go!'</span><span class="p">)</span>
8464 </pre></div>
8465
8466 <p>As long as the <code>countdown</code> variable is greater than zero, the code keeps appending text without a trailing newline and then goes to sleep for one second. Finally, when the countdown is finished, it prints <code>Go!</code> and terminates the line.</p>
8467 <p>Unexpectedly, instead of counting down every second, the program idles wastefully for three seconds, and then suddenly prints the entire line at once:</p>
8468 <p><a href="https://files.realpython.com/media/print_countdown.ba38eb242915.gif" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/print_countdown.ba38eb242915.gif" width="576" height="236" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/print_countdown.ba38eb242915.gif&w=144&sig=1084be0d7eb723df32fe7d0e16f06724e2666c04 144w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/print_countdown.ba38eb242915.gif&w=288&sig=0618ab48bb5a9e8b729d8a97e4736f78f9b3560f 288w, https://files.realpython.com/media/print_countdown.ba38eb242915.gif 576w" sizes="75vw" alt="Terminal with buffered output"/></a></p>
8469 <p>That’s because the operating system buffers subsequent writes to the standard output in this case. You need to know that there are three kinds of streams with respect to buffering:</p>
8470 <ol>
8471 <li>Unbuffered</li>
8472 <li>Line-buffered</li>
8473 <li>Block-buffered</li>
8474 </ol>
8475 <p><strong>Unbuffered</strong> is self-explanatory, that is, no buffering is taking place, and all writes have immediate effect. A <strong>line-buffered</strong> stream waits before firing any I/O calls until a line break appears somewhere in the buffer, whereas a <strong>block-buffered</strong> one simply allows the buffer to fill up to a certain size regardless of its content. Standard output is both <strong>line-buffered</strong> and <strong>block-buffered</strong>, depending on which event comes first.</p>
8476 <p>Buffering helps to reduce the number of expensive I/O calls. Think about sending messages over a high-latency network, for example. When you connect to a remote server to execute commands over the SSH protocol, each of your keystrokes may actually produce an individual data packet, which is orders of magnitude bigger than its payload. What an overhead! It would make sense to wait until at least a few characters are typed and then send them together. That’s where buffering steps in.</p>
8477 <p>On the other hand, buffering can sometimes have undesired effects as you just saw with the countdown example. To fix it, you can simply tell <code>print()</code> to forcefully flush the stream without waiting for a newline character in the buffer using its <code>flush</code> flag:</p>
8478 <div class="highlight python"><pre><span></span><span class="nb">print</span><span class="p">(</span><span class="n">countdown</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s1">'...'</span><span class="p">,</span> <span class="n">flush</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
8479 </pre></div>
8480
8481 <p>That’s all. Your countdown should work as expected now, but don’t take my word for it. Go ahead and test it to see the difference.</p>
8482 <p>Congratulations! At this point, you’ve seen examples of calling <code>print()</code> that cover all of its parameters. You know their purpose and when to use them. Understanding the signature is only the beginning, however. In the upcoming sections, you’ll see why.</p>
8483 <div class="card mb-3" id="collapse_cardec4147">
8484 <div class="card-header border-0"><p class="m-0"><button class="btn" data-toggle="collapse" data-target="#collapseec4147" aria-expanded="false" aria-controls="collapseec4147">Syntax in Python 2</button> <button class="btn btn-link float-right" data-toggle="collapse" data-target="#collapseec4147" aria-expanded="false" aria-controls="collapseec4147">Show/Hide</button></p></div>
8485 <div id="collapseec4147" class="collapse" data-parent="#collapse_cardec4147"><div class="card-body" markdown="1">
8486
8487 <p>There isn’t an easy way to flush the stream in Python 2, because the <code>print</code> statement doesn’t allow for it by itself. You need to get a handle of its lower-level layer, which is the standard output, and call it directly:</p>
8488 <div class="highlight python"><pre><span></span><span class="kn">import</span> <span class="nn">time</span>
8489 <span class="kn">import</span> <span class="nn">sys</span>
8490
8491 <span class="n">num_seconds</span> <span class="o">=</span> <span class="mi">3</span>
8492 <span class="k">for</span> <span class="n">countdown</span> <span class="ow">in</span> <span class="nb">reversed</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="n">num_seconds</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)):</span>
8493 <span class="k">if</span> <span class="n">countdown</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span>
8494 <span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'</span><span class="si">%s</span><span class="s1">...'</span> <span class="o">%</span> <span class="n">countdown</span><span class="p">)</span>
8495 <span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
8496 <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
8497 <span class="k">else</span><span class="p">:</span>
8498 <span class="k">print</span> <span class="s1">'Go!'</span>
8499 </pre></div>
8500
8501 <p>Alternatively, you could disable buffering of the standard streams either by providing the <code>-u</code> flag to the Python interpreter or by setting up the <code>PYTHONUNBUFFERED</code> environment variable:</p>
8502 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python2 -u countdown.py
8503 <span class="gp">$</span> <span class="nv">PYTHONUNBUFFERED</span><span class="o">=</span><span class="m">1</span> python2 countdown.py
8504 </pre></div>
8505
8506 <p>Note that <code>print()</code> was backported to Python 2 and made available through the <code>__future__</code> module. Unfortunately, it doesn’t come with the <code>flush</code> parameter:</p>
8507 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">print_function</span>
8508 <span class="gp">>>> </span><span class="n">help</span><span class="p">(</span><span class="k">print</span><span class="p">)</span>
8509 <span class="go">Help on built-in function print in module __builtin__:</span>
8510
8511 <span class="go">print(...)</span>
8512 <span class="go"> print(value, ..., sep=' ', end='\n', file=sys.stdout)</span>
8513 </pre></div>
8514
8515 <p>What you’re seeing here is a <strong>docstring</strong> of the <code>print()</code> function. You can display docstrings of various objects in Python using the built-in <code>help()</code> function.</p>
8516 </div></div>
8517
8518 </div>
8519 <h3 id="printing-custom-data-types">Printing Custom Data Types</h3>
8520 <p>Up until now, you only dealt with built-in data types such as strings and numbers, but you’ll often want to print your own abstract data types. Let’s have a look at different ways of defining them.</p>
8521 <p>For simple objects without any logic, whose purpose is to carry data, you’ll typically take advantage of <a href="https://docs.python.org/3/library/collections.html#collections.namedtuple"><code>namedtuple</code></a>, which is available in the standard library. Named tuples have a neat textual representation out of the box:</p>
8522 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">collections</span> <span class="k">import</span> <span class="n">namedtuple</span>
8523 <span class="gp">>>> </span><span class="n">Person</span> <span class="o">=</span> <span class="n">namedtuple</span><span class="p">(</span><span class="s1">'Person'</span><span class="p">,</span> <span class="s1">'name age'</span><span class="p">)</span>
8524 <span class="gp">>>> </span><span class="n">jdoe</span> <span class="o">=</span> <span class="n">Person</span><span class="p">(</span><span class="s1">'John Doe'</span><span class="p">,</span> <span class="mi">42</span><span class="p">)</span>
8525 <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">jdoe</span><span class="p">)</span>
8526 <span class="go">Person(name='John Doe', age=42)</span>
8527 </pre></div>
8528
8529 <p>That’s great as long as holding data is enough, but in order to add behaviors to the <code>Person</code> type, you’ll eventually need to define a class. Take a look at this example:</p>
8530 <div class="highlight python"><pre><span></span><span class="k">class</span> <span class="nc">Person</span><span class="p">:</span>
8531 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">age</span><span class="p">):</span>
8532 <span class="bp">self</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">age</span> <span class="o">=</span> <span class="n">name</span><span class="p">,</span> <span class="n">age</span>
8533 </pre></div>
8534
8535 <p>If you now create an instance of the <code>Person</code> class and try to print it, you’ll get this bizarre output, which is quite different from the equivalent <code>namedtuple</code>:</p>
8536 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">jdoe</span> <span class="o">=</span> <span class="n">Person</span><span class="p">(</span><span class="s1">'John Doe'</span><span class="p">,</span> <span class="mi">42</span><span class="p">)</span>
8537 <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">jdoe</span><span class="p">)</span>
8538 <span class="go"><__main__.Person object at 0x7fcac3fed1d0></span>
8539 </pre></div>
8540
8541 <p>It’s the default representation of objects, which comprises their address in memory, the corresponding class name and a module in which they were defined. You’ll fix that in a bit, but just for the record, as a quick workaround you could combine <code>namedtuple</code> and a custom class through <a href="https://realpython.com/inheritance-composition-python/">inheritance</a>:</p>
8542 <div class="highlight python"><pre><span></span><span class="kn">from</span> <span class="nn">collections</span> <span class="k">import</span> <span class="n">namedtuple</span>
8543
8544 <span class="k">class</span> <span class="nc">Person</span><span class="p">(</span><span class="n">namedtuple</span><span class="p">(</span><span class="s1">'Person'</span><span class="p">,</span> <span class="s1">'name age'</span><span class="p">)):</span>
8545 <span class="k">pass</span>
8546 </pre></div>
8547
8548 <p>Your <code>Person</code> class has just become a specialized kind of <code>namedtuple</code> with two attributes, which you can customize.</p>
8549 <div class="alert alert-primary" role="alert">
8550 <p><strong>Note:</strong> In Python 3, the <code>pass</code> statement can be replaced with the <a href="https://docs.python.org/dev/library/constants.html#Ellipsis">ellipsis</a> (<code>...</code>) literal to indicate a placeholder:</p>
8551 <div class="highlight python"><pre><span></span><span class="k">def</span> <span class="nf">delta</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="n">c</span><span class="p">):</span>
8552 <span class="o">...</span>
8553 </pre></div>
8554
8555 <p>This prevents the interpreter from raising <code>IndentationError</code> due to missing indented block of code.</p>
8556 </div>
8557 <p>That’s better than a plain <code>namedtuple</code>, because not only do you get printing right for free, but you can also add custom methods and properties to the class. However, it solves one problem while introducing another. Remember that tuples, including named tuples, are immutable in Python, so they can’t change their values once created.</p>
8558 <p>It’s true that designing immutable data types is desirable, but in many cases, you’ll want them to allow for change, so you’re back with regular classes again.</p>
8559 <div class="alert alert-primary" role="alert">
8560 <p><strong>Note:</strong> Following other languages and frameworks, Python 3.7 introduced <a href="https://realpython.com/python-data-classes/">data classes</a>, which you can think of as mutable tuples. This way, you get the best of both worlds:</p>
8561 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">dataclasses</span> <span class="k">import</span> <span class="n">dataclass</span>
8562 <span class="gp">>>> </span><span class="nd">@dataclass</span>
8563 <span class="gp">... </span><span class="k">class</span> <span class="nc">Person</span><span class="p">:</span>
8564 <span class="gp">... </span> <span class="n">name</span><span class="p">:</span> <span class="nb">str</span>
8565 <span class="gp">... </span> <span class="n">age</span><span class="p">:</span> <span class="nb">int</span>
8566 <span class="gp">... </span>
8567 <span class="gp">... </span> <span class="k">def</span> <span class="nf">celebrate_birthday</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
8568 <span class="gp">... </span> <span class="bp">self</span><span class="o">.</span><span class="n">age</span> <span class="o">+=</span> <span class="mi">1</span>
8569 <span class="gp">... </span>
8570 <span class="gp">>>> </span><span class="n">jdoe</span> <span class="o">=</span> <span class="n">Person</span><span class="p">(</span><span class="s1">'John Doe'</span><span class="p">,</span> <span class="mi">42</span><span class="p">)</span>
8571 <span class="gp">>>> </span><span class="n">jdoe</span><span class="o">.</span><span class="n">celebrate_birthday</span><span class="p">()</span>
8572 <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">jdoe</span><span class="p">)</span>
8573 <span class="go">Person(name='John Doe', age=43)</span>
8574 </pre></div>
8575
8576 <p>The syntax for <a href="https://www.python.org/dev/peps/pep-0526/">variable annotations</a>, which is required to specify class fields with their corresponding types, was defined in Python 3.6.</p>
8577 </div>
8578 <p>From earlier subsections, you already know that <code>print()</code> implicitly calls the built-in <code>str()</code> function to convert its positional arguments into strings. Indeed, calling <code>str()</code> manually against an instance of the regular <code>Person</code> class yields the same result as printing it:</p>
8579 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">jdoe</span> <span class="o">=</span> <span class="n">Person</span><span class="p">(</span><span class="s1">'John Doe'</span><span class="p">,</span> <span class="mi">42</span><span class="p">)</span>
8580 <span class="gp">>>> </span><span class="nb">str</span><span class="p">(</span><span class="n">jdoe</span><span class="p">)</span>
8581 <span class="go">'<__main__.Person object at 0x7fcac3fed1d0>'</span>
8582 </pre></div>
8583
8584 <p><code>str()</code>, in turn, looks for one of two <strong>magic methods</strong> within the class body, which you typically implement. If it doesn’t find one, then it falls back to the ugly default representation. Those magic methods are, in order of search:</p>
8585 <ol>
8586 <li><code>def __str__(self)</code></li>
8587 <li><code>def __repr__(self)</code></li>
8588 </ol>
8589 <p>The first one is recommended to return a short, human-readable text, which includes information from the most relevant attributes. After all, you don’t want to expose sensitive data, such as user passwords, when printing objects.</p>
8590 <p>However, the other one should provide complete information about an object, to allow for restoring its state from a string. Ideally, it should return valid Python code, so that you can pass it directly to <code>eval()</code>:</p>
8591 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="nb">repr</span><span class="p">(</span><span class="n">jdoe</span><span class="p">)</span>
8592 <span class="go">"Person(name='John Doe', age=42)"</span>
8593 <span class="gp">>>> </span><span class="nb">type</span><span class="p">(</span><span class="nb">eval</span><span class="p">(</span><span class="nb">repr</span><span class="p">(</span><span class="n">jdoe</span><span class="p">)))</span>
8594 <span class="go"><class '__main__.Person'></span>
8595 </pre></div>
8596
8597 <p>Notice the use of another built-in function, <code>repr()</code>, which always tries to call <code>.__repr__()</code> in an object, but falls back to the default representation if it doesn’t find that method.</p>
8598 <div class="alert alert-primary" role="alert">
8599 <p><strong>Note:</strong> Even though <code>print()</code> itself uses <code>str()</code> for type casting, some compound data types delegate that call to <code>repr()</code> on their members. This happens to lists and tuples, for example.</p>
8600 <p>Consider this class with both magic methods, which return alternative string representations of the same object:</p>
8601 <div class="highlight python"><pre><span></span><span class="k">class</span> <span class="nc">User</span><span class="p">:</span>
8602 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">login</span><span class="p">,</span> <span class="n">password</span><span class="p">):</span>
8603 <span class="bp">self</span><span class="o">.</span><span class="n">login</span> <span class="o">=</span> <span class="n">login</span>
8604 <span class="bp">self</span><span class="o">.</span><span class="n">password</span> <span class="o">=</span> <span class="n">password</span>
8605
8606 <span class="k">def</span> <span class="nf">__str__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
8607 <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">login</span>
8608
8609 <span class="k">def</span> <span class="nf">__repr__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
8610 <span class="k">return</span> <span class="n">f</span><span class="s2">"User('</span><span class="si">{self.login}</span><span class="s2">', '</span><span class="si">{self.password}</span><span class="s2">')"</span>
8611 </pre></div>
8612
8613 <p>If you print a single object of the <code>User</code> class, then you won’t see the password, because <code>print(user)</code> will call <code>str(user)</code>, which eventually will invoke <code>user.__str__()</code>:</p>
8614 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">user</span> <span class="o">=</span> <span class="n">User</span><span class="p">(</span><span class="s1">'jdoe'</span><span class="p">,</span> <span class="s1">'s3cret'</span><span class="p">)</span>
8615 <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">user</span><span class="p">)</span>
8616 <span class="go">jdoe</span>
8617 </pre></div>
8618
8619 <p>However, if you put the same <code>user</code> variable inside a list by wrapping it in square brackets, then the password will become clearly visible:</p>
8620 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="nb">print</span><span class="p">([</span><span class="n">user</span><span class="p">])</span>
8621 <span class="go">[User('jdoe', 's3cret')]</span>
8622 </pre></div>
8623
8624 <p>That’s because sequences, such as lists and tuples, implement their <code>.__str__()</code> method so that all of their elements are first converted with <code>repr()</code>.</p>
8625 </div>
8626 <p>Python gives you a lot of freedom when it comes to defining your own data types if none of the built-in ones meet your needs. Some of them, such as named tuples and data classes, offer string representations that look good without requiring any work on your part. Still, for the most flexibility, you’ll have to define a class and override its magic methods described above.</p>
8627 <div class="card mb-3" id="collapse_card996300">
8628 <div class="card-header border-0"><p class="m-0"><button class="btn" data-toggle="collapse" data-target="#collapse996300" aria-expanded="false" aria-controls="collapse996300">Syntax in Python 2</button> <button class="btn btn-link float-right" data-toggle="collapse" data-target="#collapse996300" aria-expanded="false" aria-controls="collapse996300">Show/Hide</button></p></div>
8629 <div id="collapse996300" class="collapse" data-parent="#collapse_card996300"><div class="card-body" markdown="1">
8630
8631 <p>The semantics of <code>.__str__()</code> and <code>.__repr__()</code> didn’t change since Python 2, but you must remember that strings were nothing more than glorified byte arrays back then. To convert your objects into proper Unicode, which was a separate data type, you’d have to provide yet another magic method: <code>.__unicode__()</code>.</p>
8632 <p>Here’s an example of the same <code>User</code> class in Python 2:</p>
8633 <div class="highlight python"><pre><span></span><span class="k">class</span> <span class="nc">User</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
8634 <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">login</span><span class="p">,</span> <span class="n">password</span><span class="p">):</span>
8635 <span class="bp">self</span><span class="o">.</span><span class="n">login</span> <span class="o">=</span> <span class="n">login</span>
8636 <span class="bp">self</span><span class="o">.</span><span class="n">password</span> <span class="o">=</span> <span class="n">password</span>
8637
8638 <span class="k">def</span> <span class="fm">__unicode__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
8639 <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">login</span>
8640
8641 <span class="k">def</span> <span class="fm">__str__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
8642 <span class="k">return</span> <span class="nb">unicode</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">)</span>
8643
8644 <span class="k">def</span> <span class="fm">__repr__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
8645 <span class="n">user</span> <span class="o">=</span> <span class="sa">u</span><span class="s2">"User('</span><span class="si">%s</span><span class="s2">', '</span><span class="si">%s</span><span class="s2">')"</span> <span class="o">%</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">login</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">password</span><span class="p">)</span>
8646 <span class="k">return</span> <span class="n">user</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'unicode_escape'</span><span class="p">)</span>
8647 </pre></div>
8648
8649 <p>As you can see, this implementation delegates some work to avoid duplication by calling the built-in <code>unicode()</code> function on itself.</p>
8650 <p>Both <code>.__str__()</code> and <code>.__repr__()</code> methods must return strings, so they encode Unicode characters into specific byte representations called <strong>character sets</strong>. UTF-8 is the most widespread and safest encoding, while <code>unicode_escape</code> is a special constant to express funky characters, such as <code>Γ©</code>, as escape sequences in plain ASCII, such as <code>\xe9</code>.</p>
8651 <p>The <code>print</code> statement is looking for the magic <code>.__str__()</code> method in the class, so the chosen <strong>charset</strong> must correspond to the one used by the terminal. For example, default encoding in DOS and Windows is CP 852 rather than UTF-8, so running this can result in a <code>UnicodeEncodeError</code> or even garbled output:</p>
8652 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">user</span> <span class="o">=</span> <span class="n">User</span><span class="p">(</span><span class="sa">u</span><span class="s1">'</span><span class="se">\u043d\u0438\u043a\u0438\u0442\u0430</span><span class="s1">'</span><span class="p">,</span> <span class="sa">u</span><span class="s1">'s3cret'</span><span class="p">)</span>
8653 <span class="gp">>>> </span><span class="k">print</span> <span class="n">user</span>
8654 <span class="go">ΔΕ»ΔΕΔβΔΕΔΓ©Δβ</span>
8655 </pre></div>
8656
8657 <p>However, if you ran the same code on a system with UTF-8 encoding, then you’d get the proper spelling of a popular Russian name:</p>
8658 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">user</span> <span class="o">=</span> <span class="n">User</span><span class="p">(</span><span class="sa">u</span><span class="s1">'</span><span class="se">\u043d\u0438\u043a\u0438\u0442\u0430</span><span class="s1">'</span><span class="p">,</span> <span class="sa">u</span><span class="s1">'s3cret'</span><span class="p">)</span>
8659 <span class="gp">>>> </span><span class="k">print</span> <span class="n">user</span>
8660 <span class="go">Π½ΠΈΠΊΠΈΡΠ°</span>
8661 </pre></div>
8662
8663 <p>It’s recommended to convert strings to Unicode as early as possible, for example, when you’re reading data from a file, and use it consistently everywhere in your code. At the same time, you should encode Unicode back to the chosen character set right before presenting it to the user.</p>
8664 <p>It seems as if you have more control over string representation of objects in Python 2 because there’s no magic <code>.__unicode__()</code> method in Python 3 anymore. You may be asking yourself if it’s possible to convert an object to its byte string representation rather than a Unicode string in Python 3. It’s possible, with a special <code>.__bytes__()</code> method that does just that:</p>
8665 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">class</span> <span class="nc">User</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
8666 <span class="gp">... </span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">login</span><span class="p">,</span> <span class="n">password</span><span class="p">):</span>
8667 <span class="gp">... </span> <span class="bp">self</span><span class="o">.</span><span class="n">login</span> <span class="o">=</span> <span class="n">login</span>
8668 <span class="gp">... </span> <span class="bp">self</span><span class="o">.</span><span class="n">password</span> <span class="o">=</span> <span class="n">password</span>
8669 <span class="gp">... </span>
8670 <span class="gp">... </span> <span class="k">def</span> <span class="nf">__bytes__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="c1"># Python 3</span>
8671 <span class="gp">... </span> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">login</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">)</span>
8672 <span class="gp">...</span>
8673 <span class="gp">>>> </span><span class="n">user</span> <span class="o">=</span> <span class="n">User</span><span class="p">(</span><span class="sa">u</span><span class="s1">'</span><span class="se">\u043d\u0438\u043a\u0438\u0442\u0430</span><span class="s1">'</span><span class="p">,</span> <span class="sa">u</span><span class="s1">'s3cret'</span><span class="p">)</span>
8674 <span class="gp">>>> </span><span class="nb">bytes</span><span class="p">(</span><span class="n">user</span><span class="p">)</span>
8675 <span class="go">b'\xd0\xbd\xd0\xb8\xd0\xba\xd0\xb8\xd1\x82\xd0\xb0'</span>
8676 </pre></div>
8677
8678 <p>Using the built-in <code>bytes()</code> function on an instance delegates the call to its <code>__bytes__()</code> method defined in the corresponding class.</p>
8679 </div></div>
8680
8681 </div>
8682 <h2 id="understanding-python-print">Understanding Python Print</h2>
8683 <p>You know <strong>how</strong> to use <code>print()</code> quite well at this point, but knowing <strong>what</strong> it is will allow you to use it even more effectively and consciously. After reading this section, you’ll understand how printing in Python has improved over the years.</p>
8684 <h3 id="print-is-a-function-in-python-3">Print Is a Function in Python 3</h3>
8685 <p>You’ve seen that <code>print()</code> is a function in Python 3. More specifically, it’s a built-in function, which means that you don’t need to import it from anywhere:</p>
8686 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="nb">print</span>
8687 <span class="go"><built-in function print></span>
8688 </pre></div>
8689
8690 <p>It’s always available in the global namespace so that you can call it directly, but you can also access it through a module from the standard library:</p>
8691 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">builtins</span>
8692 <span class="gp">>>> </span><span class="n">builtins</span><span class="o">.</span><span class="n">print</span>
8693 <span class="go"><built-in function print></span>
8694 </pre></div>
8695
8696 <p>This way, you can avoid name collisions with custom functions. Let’s say you wanted to <strong>redefine</strong> <code>print()</code> so that it doesn’t append a trailing newline. At the same time, you wanted to rename the original function to something like <code>println()</code>:</p>
8697 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">builtins</span>
8698 <span class="gp">>>> </span><span class="n">println</span> <span class="o">=</span> <span class="n">builtins</span><span class="o">.</span><span class="n">print</span>
8699 <span class="gp">>>> </span><span class="k">def</span> <span class="nf">print</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
8700 <span class="gp">... </span> <span class="n">builtins</span><span class="o">.</span><span class="n">print</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s1">''</span><span class="p">)</span>
8701 <span class="gp">...</span>
8702 <span class="gp">>>> </span><span class="n">println</span><span class="p">(</span><span class="s1">'hello'</span><span class="p">)</span>
8703 <span class="go">hello</span>
8704 <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="s1">'hello</span><span class="se">\n</span><span class="s1">'</span><span class="p">)</span>
8705 <span class="go">hello</span>
8706 </pre></div>
8707
8708 <p>Now you have two separate printing functions just like in the Java programming language. You’ll define custom <code>print()</code> functions in the <a href="#mocking-python-print-in-unit-tests">mocking section</a> later as well. Also, note that you wouldn’t be able to overwrite <code>print()</code> in the first place if it wasn’t a function.</p>
8709 <p>On the other hand, <code>print()</code> isn’t a function in the mathematical sense, because it doesn’t return any meaningful value other than the implicit <code>None</code>:</p>
8710 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">value</span> <span class="o">=</span> <span class="nb">print</span><span class="p">(</span><span class="s1">'hello world'</span><span class="p">)</span>
8711 <span class="go">hello world</span>
8712 <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
8713 <span class="go">None</span>
8714 </pre></div>
8715
8716 <p>Such functions are, in fact, procedures or subroutines that you call to achieve some kind of side-effect, which ultimately is a change of a global state. In the case of <code>print()</code>, that side-effect is showing a message on the standard output or writing to a file.</p>
8717 <p>Because <code>print()</code> is a function, it has a well-defined signature with known attributes. You can quickly find its <strong>documentation</strong> using the editor of your choice, without having to remember some weird syntax for performing a certain task.</p>
8718 <p>Besides, functions are easier to <strong>extend</strong>. Adding a new feature to a function is as easy as adding another keyword argument, whereas changing the language to support that new feature is much more cumbersome. Think of stream redirection or buffer flushing, for example.</p>
8719 <p>Another benefit of <code>print()</code> being a function is <strong>composability</strong>. Functions are so-called <a href="https://realpython.com/lessons/functions-first-class-objects-python/">first-class objects</a> or <a href="https://realpython.com/lessons/functions-are-first-class-citizens-python/">first-class citizens</a> in Python, which is a fancy way of saying they’re values just like strings or numbers. This way, you can assign a function to a variable, pass it to another function, or even return one from another. <code>print()</code> isn’t different in this regard. For instance, you can take advantage of it for dependency injection:</p>
8720 <div class="highlight python"><pre><span></span><span class="k">def</span> <span class="nf">download</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">log</span><span class="o">=</span><span class="nb">print</span><span class="p">):</span>
8721 <span class="n">log</span><span class="p">(</span><span class="n">f</span><span class="s1">'Downloading </span><span class="si">{url}</span><span class="s1">'</span><span class="p">)</span>
8722 <span class="c1"># ...</span>
8723
8724 <span class="k">def</span> <span class="nf">custom_print</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">):</span>
8725 <span class="k">pass</span> <span class="c1"># Do not print anything</span>
8726
8727 <span class="n">download</span><span class="p">(</span><span class="s1">'/js/app.js'</span><span class="p">,</span> <span class="n">log</span><span class="o">=</span><span class="n">custom_print</span><span class="p">)</span>
8728 </pre></div>
8729
8730 <p>Here, the <code>log</code> parameter lets you inject a callback function, which defaults to <code>print()</code> but can be any callable. In this example, printing is completely disabled by substituting <code>print()</code> with a dummy function that does nothing.</p>
8731 <div class="alert alert-primary" role="alert">
8732 <p><strong>Note:</strong> A <strong>dependency</strong> is any piece of code required by another bit of code.</p>
8733 <p><strong>Dependency injection</strong> is a technique used in code design to make it more testable, reusable, and open for extension. You can achieve it by referring to dependencies indirectly through abstract interfaces and by providing them in a <strong>push</strong> rather than <strong>pull</strong> fashion.</p>
8734 <p>There’s a funny explanation of dependency injection circulating on the Internet:</p>
8735 <blockquote>
8736 <p>Dependency injection for five-year-olds</p>
8737 <p>When you go and get things out of the refrigerator for yourself, you can cause problems. You might leave the door open, you might get something Mommy or Daddy doesn’t want you to have. You might even be looking for something we don’t even have or which has expired.</p>
8738 <p>What you should be doing is stating a need, “I need something to drink with lunch,” and then we will make sure you have something when you sit down to eat.</p>
8739 <p>— <em>John Munsch, 28 October 2009.</em> (<a href="https://stackoverflow.com/a/1638961">Source</a>)</p>
8740 </blockquote>
8741 </div>
8742 <p>Composition allows you to combine a few functions into a new one of the same kind. Let’s see this in action by specifying a custom <code>error()</code> function that prints to the standard error stream and prefixes all messages with a given log level:</p>
8743 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">functools</span> <span class="k">import</span> <span class="n">partial</span>
8744 <span class="gp">>>> </span><span class="kn">import</span> <span class="nn">sys</span>
8745 <span class="gp">>>> </span><span class="n">redirect</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">function</span><span class="p">,</span> <span class="n">stream</span><span class="p">:</span> <span class="n">partial</span><span class="p">(</span><span class="n">function</span><span class="p">,</span> <span class="n">file</span><span class="o">=</span><span class="n">stream</span><span class="p">)</span>
8746 <span class="gp">>>> </span><span class="n">prefix</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">function</span><span class="p">,</span> <span class="n">prefix</span><span class="p">:</span> <span class="n">partial</span><span class="p">(</span><span class="n">function</span><span class="p">,</span> <span class="n">prefix</span><span class="p">)</span>
8747 <span class="gp">>>> </span><span class="n">error</span> <span class="o">=</span> <span class="n">prefix</span><span class="p">(</span><span class="n">redirect</span><span class="p">(</span><span class="nb">print</span><span class="p">,</span> <span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="p">),</span> <span class="s1">'[ERROR]'</span><span class="p">)</span>
8748 <span class="gp">>>> </span><span class="n">error</span><span class="p">(</span><span class="s1">'Something went wrong'</span><span class="p">)</span>
8749 <span class="go">[ERROR] Something went wrong</span>
8750 </pre></div>
8751
8752 <p>This custom function uses <strong>partial functions</strong> to achieve the desired effect. It’s an advanced concept borrowed from the <a href="https://realpython.com/courses/functional-programming-python/">functional programming</a> paradigm, so you don’t need to go too deep into that topic for now. However, if you’re interested in this topic, I recommend taking a look at the <a href="https://pymotw.com/3/functools/"><code>functools</code></a> module.</p>
8753 <p>Unlike statements, functions are values. That means you can mix them with <strong>expressions</strong>, in particular, <a href="https://realpython.com/python-lambda/"><strong>lambda</strong> expressions</a>. Instead of defining a full-blown function to replace <code>print()</code> with, you can make an anonymous lambda expression that calls it:</p>
8754 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">download</span><span class="p">(</span><span class="s1">'/js/app.js'</span><span class="p">,</span> <span class="k">lambda</span> <span class="n">msg</span><span class="p">:</span> <span class="nb">print</span><span class="p">(</span><span class="s1">'[INFO]'</span><span class="p">,</span> <span class="n">msg</span><span class="p">))</span>
8755 <span class="go">[INFO] Downloading /js/app.js</span>
8756 </pre></div>
8757
8758 <p>However, because a lambda expression is defined in place, there’s no way of referring to it elsewhere in the code.</p>
8759 <div class="alert alert-primary" role="alert">
8760 <p><strong>Note:</strong> In Python, you can’t put statements, such as assignments, conditional statements, loops, and so on, in an <strong>anonymous lambda function</strong>. It has to be a single expression!</p>
8761 </div>
8762 <p>Another kind of expression is a ternary conditional expression:</p>
8763 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">user</span> <span class="o">=</span> <span class="s1">'jdoe'</span>
8764 <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="s1">'Hi!'</span><span class="p">)</span> <span class="k">if</span> <span class="n">user</span> <span class="ow">is</span> <span class="kc">None</span> <span class="k">else</span> <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Hi, </span><span class="si">{user}</span><span class="s1">.'</span><span class="p">)</span>
8765 <span class="go">Hi, jdoe.</span>
8766 </pre></div>
8767
8768 <p>Python has both <a href="https://realpython.com/python-conditional-statements/">conditional statements</a> and <a href="https://realpython.com/python-conditional-statements/#conditional-expressions-pythons-ternary-operator">conditional expressions</a>. The latter is evaluated to a single value that can be assigned to a variable or passed to a function. In the example above, you’re interested in the side-effect rather than the value, which evaluates to <code>None</code>, so you simply ignore it.</p>
8769 <p>As you can see, functions allow for an elegant and extensible solution, which is consistent with the rest of the language. In the next subsection, you’ll discover how not having <code>print()</code> as a function caused a lot of headaches.</p>
8770 <h3 id="print-was-a-statement-in-python-2">Print Was a Statement in Python 2</h3>
8771 <p>A <strong>statement</strong> is an instruction that may evoke a side-effect when executed but never evaluates to a value. In other words, you wouldn’t be able to print a statement or assign it to a variable like this:</p>
8772 <div class="highlight python"><pre><span></span><span class="n">result</span> <span class="o">=</span> <span class="k">print</span> <span class="s1">'hello world'</span>
8773 </pre></div>
8774
8775 <p>That’s a syntax error in Python 2.</p>
8776 <p>Here are a few more examples of statements in Python:</p>
8777 <ul>
8778 <li><strong>assignment:</strong> <code>=</code></li>
8779 <li><strong>conditional:</strong> <code>if</code></li>
8780 <li><strong>loop:</strong> <code>while</code></li>
8781 <li><strong>assertion</strong>: <code>assert</code></li>
8782 </ul>
8783 <div class="alert alert-primary" role="alert">
8784 <p><strong>Note:</strong> Python 3.8 brings a controversial <strong>walrus operator</strong> (<code>:=</code>), which is an <a href="https://www.python.org/dev/peps/pep-0572/">assignment expression</a>. With it, you can evaluate an expression and assign the result to a variable at the same time, even within another expression!</p>
8785 <p>Take a look at this example, which calls an expensive function once and then reuses the result for further computation:</p>
8786 <div class="highlight python"><pre><span></span><span class="c1"># Python 3.8+</span>
8787 <span class="n">values</span> <span class="o">=</span> <span class="p">[</span><span class="n">y</span> <span class="p">:</span><span class="o">=</span> <span class="n">f</span><span class="p">(</span><span class="n">x</span><span class="p">),</span> <span class="n">y</span><span class="o">**</span><span class="mi">2</span><span class="p">,</span> <span class="n">y</span><span class="o">**</span><span class="mi">3</span><span class="p">]</span>
8788 </pre></div>
8789
8790 <p>This is useful for simplifying the code without losing its efficiency. Typically, performant code tends to be more verbose:</p>
8791 <div class="highlight python"><pre><span></span><span class="n">y</span> <span class="o">=</span> <span class="n">f</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
8792 <span class="n">values</span> <span class="o">=</span> <span class="p">[</span><span class="n">y</span><span class="p">,</span> <span class="n">y</span><span class="o">**</span><span class="mi">2</span><span class="p">,</span> <span class="n">y</span><span class="o">**</span><span class="mi">3</span><span class="p">]</span>
8793 </pre></div>
8794
8795 <p>The controversy behind this new piece of syntax caused a lot of argument. An abundance of negative comments and heated debates eventually led Guido van Rossum to step down from the <strong>Benevolent Dictator For Life</strong> or BDFL position.</p>
8796 </div>
8797 <p>Statements are usually comprised of reserved keywords such as <code>if</code>, <code>for</code>, or <code>print</code> that have fixed meaning in the language. You can’t use them to name your variables or other symbols. That’s why redefining or mocking the <code>print</code> statement isn’t possible in Python 2. You’re stuck with what you get.</p>
8798 <p>Furthermore, you can’t print from anonymous functions, because statements aren’t accepted in lambda expressions:</p>
8799 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">lambda</span><span class="p">:</span> <span class="k">print</span> <span class="s1">'hello world'</span>
8800 File <span class="nb">"<stdin>"</span>, line <span class="m">1</span>
8801 <span class="k">lambda</span><span class="p">:</span> <span class="k">print</span> <span class="s1">'hello world'</span>
8802 <span class="o">^</span>
8803 <span class="gr">SyntaxError</span>: <span class="n">invalid syntax</span>
8804 </pre></div>
8805
8806 <p>The syntax of the <code>print</code> statement is ambiguous. Sometimes you can add parentheses around the message, and they’re completely optional:</p>
8807 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">print</span> <span class="s1">'Please wait...'</span>
8808 <span class="go">Please wait...</span>
8809 <span class="gp">>>> </span><span class="k">print</span><span class="p">(</span><span class="s1">'Please wait...'</span><span class="p">)</span>
8810 <span class="go">Please wait...</span>
8811 </pre></div>
8812
8813 <p>At other times they change how the message is printed:</p>
8814 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">print</span> <span class="s1">'My name is'</span><span class="p">,</span> <span class="s1">'John'</span>
8815 <span class="go">My name is John</span>
8816 <span class="gp">>>> </span><span class="k">print</span><span class="p">(</span><span class="s1">'My name is'</span><span class="p">,</span> <span class="s1">'John'</span><span class="p">)</span>
8817 <span class="go">('My name is', 'John')</span>
8818 </pre></div>
8819
8820 <p>String concatenation can raise a <code>TypeError</code> due to incompatible types, which you have to handle manually, for example:</p>
8821 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">values</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'jdoe'</span><span class="p">,</span> <span class="s1">'is'</span><span class="p">,</span> <span class="mi">42</span><span class="p">,</span> <span class="s1">'years old'</span><span class="p">]</span>
8822 <span class="gp">>>> </span><span class="k">print</span> <span class="s1">' '</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="nb">map</span><span class="p">(</span><span class="nb">str</span><span class="p">,</span> <span class="n">values</span><span class="p">))</span>
8823 <span class="go">jdoe is 42 years old</span>
8824 </pre></div>
8825
8826 <p>Compare this with similar code in Python 3, which leverages sequence unpacking:</p>
8827 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">values</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'jdoe'</span><span class="p">,</span> <span class="s1">'is'</span><span class="p">,</span> <span class="mi">42</span><span class="p">,</span> <span class="s1">'years old'</span><span class="p">]</span>
8828 <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="o">*</span><span class="n">values</span><span class="p">)</span> <span class="c1"># Python 3</span>
8829 <span class="go">jdoe is 42 years old</span>
8830 </pre></div>
8831
8832 <p>There aren’t any keyword arguments for common tasks such as flushing the buffer or stream redirection. You need to remember the quirky syntax instead. Even the built-in <code>help()</code> function isn’t that helpful with regards to the <code>print</code> statement:</p>
8833 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">help</span><span class="p">(</span><span class="k">print</span><span class="p">)</span>
8834 File <span class="nb">"<stdin>"</span>, line <span class="m">1</span>
8835 <span class="n">help</span><span class="p">(</span><span class="k">print</span><span class="p">)</span>
8836 <span class="o">^</span>
8837 <span class="gr">SyntaxError</span>: <span class="n">invalid syntax</span>
8838 </pre></div>
8839
8840 <p>Trailing newline removal doesn’t work quite right, because it adds an unwanted space. You can’t compose multiple <code>print</code> statements together, and, on top of that, you have to be extra diligent about character encoding.</p>
8841 <p>The list of problems goes on and on. If you’re curious, you can jump back to the <a href="#printing-in-a-nutshell">previous section</a> and look for more detailed explanations of the syntax in Python 2.</p>
8842 <p>However, you can mitigate some of those problems with a much simpler approach. It turns out the <code>print()</code> function was backported to ease the migration to Python 3. You can import it from a special <code>__future__</code> module, which exposes a selection of language features released in later Python versions.</p>
8843 <div class="alert alert-primary" role="alert">
8844 <p><strong>Note:</strong> You may import future functions as well as baked-in language constructs such as the <code>with</code> statement.</p>
8845 <p>To find out exactly what features are available to you, inspect the module:</p>
8846 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">__future__</span>
8847 <span class="gp">>>> </span><span class="n">__future__</span><span class="o">.</span><span class="n">all_feature_names</span>
8848 <span class="go">['nested_scopes',</span>
8849 <span class="go"> 'generators',</span>
8850 <span class="go"> 'division',</span>
8851 <span class="go"> 'absolute_import',</span>
8852 <span class="go"> 'with_statement',</span>
8853 <span class="go"> 'print_function',</span>
8854 <span class="go"> 'unicode_literals']</span>
8855 </pre></div>
8856
8857 <p>You could also call <code>dir(__future__)</code>, but that would show a lot of uninteresting internal details of the module.</p>
8858 </div>
8859 <p>To enable the <code>print()</code> function in Python 2, you need to add this import statement at the beginning of your source code:</p>
8860 <div class="highlight python"><pre><span></span><span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">print_function</span>
8861 </pre></div>
8862
8863 <p>From now on the <code>print</code> statement is no longer available, but you have the <code>print()</code> function at your disposal. Note that it isn’t the same function like the one in Python 3, because it’s missing the <code>flush</code> keyword argument, but the rest of the arguments are the same.</p>
8864 <p>Other than that, it doesn’t spare you from managing character encodings properly.</p>
8865 <p>Here’s an example of calling the <code>print()</code> function in Python 2:</p>
8866 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">print_function</span>
8867 <span class="gp">>>> </span><span class="kn">import</span> <span class="nn">sys</span>
8868 <span class="gp">>>> </span><span class="k">print</span><span class="p">(</span><span class="s1">'I am a function in Python'</span><span class="p">,</span> <span class="n">sys</span><span class="o">.</span><span class="n">version_info</span><span class="o">.</span><span class="n">major</span><span class="p">)</span>
8869 <span class="go">I am a function in Python 2</span>
8870 </pre></div>
8871
8872 <p>You now have an idea of how printing in Python evolved and, most importantly, understand why these backward-incompatible changes were necessary. Knowing this will surely help you become a better Python programmer.</p>
8873 <h2 id="printing-with-style">Printing With Style</h2>
8874 <p>If you thought that printing was only about lighting pixels up on the screen, then technically you’d be right. However, there are ways to make it look cool. In this section, you’ll find out how to format complex data structures, add colors and other decorations, build interfaces, use animation, and even play sounds with text! </p>
8875 <h3 id="pretty-printing-nested-data-structures">Pretty-Printing Nested Data Structures</h3>
8876 <p>Computer languages allow you to represent data as well as executable code in a structured way. Unlike Python, however, most languages give you a lot of freedom in using whitespace and formatting. This can be useful, for example in compression, but it sometimes leads to less readable code.</p>
8877 <p>Pretty-printing is about making a piece of data or code look more appealing to the human eye so that it can be understood more easily. This is done by indenting certain lines, inserting newlines, reordering elements, and so forth.</p>
8878 <p>Python comes with the <code>pprint</code> module in its standard library, which will help you in pretty-printing large data structures that don’t fit on a single line. Because it prints in a more human-friendly way, many popular <a href="https://realpython.com/interacting-with-python/">REPL</a> tools, including <a href="https://realpython.com/jupyter-notebook-introduction/">JupyterLab and IPython</a>, use it by default in place of the regular <code>print()</code> function.</p>
8879 <div class="alert alert-primary" role="alert">
8880 <p><strong>Note:</strong> To toggle pretty printing in IPython, issue the following command:</p>
8881 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">In [1]: </span><span class="o">%</span><span class="k">pprint</span>
8882 <span class="go">Pretty printing has been turned OFF</span>
8883 <span class="gp">In [2]: </span><span class="o">%</span><span class="k">pprint</span>
8884 <span class="go">Pretty printing has been turned ON</span>
8885 </pre></div>
8886
8887 <p>This is an example of <strong>Magic</strong> in IPython. There are a lot of built-in commands that start with a percent sign (<code>%</code>), but you can find more on <a href="https://pypi.org/">PyPI</a>, or even create your own.</p>
8888 </div>
8889 <p>If you don’t care about not having access to the original <code>print()</code> function, then you can replace it with <code>pprint()</code> in your code using import renaming:</p>
8890 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pprint</span> <span class="k">import</span> <span class="n">pprint</span> <span class="k">as</span> <span class="nb">print</span>
8891 <span class="gp">>>> </span><span class="nb">print</span>
8892 <span class="go"><function pprint at 0x7f7a775a3510></span>
8893 </pre></div>
8894
8895 <p>Personally, I like to have both functions at my fingertips, so I’d rather use something like <code>pp</code> as a short alias:</p>
8896 <div class="highlight python"><pre><span></span><span class="kn">from</span> <span class="nn">pprint</span> <span class="k">import</span> <span class="n">pprint</span> <span class="k">as</span> <span class="n">pp</span>
8897 </pre></div>
8898
8899 <p>At first glance, there’s hardly any difference between the two functions, and in some cases there’s virtually none:</p>
8900 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="mi">42</span><span class="p">)</span>
8901 <span class="go">42</span>
8902 <span class="gp">>>> </span><span class="n">pp</span><span class="p">(</span><span class="mi">42</span><span class="p">)</span>
8903 <span class="go">42</span>
8904 <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="s1">'hello'</span><span class="p">)</span>
8905 <span class="go">hello</span>
8906 <span class="gp">>>> </span><span class="n">pp</span><span class="p">(</span><span class="s1">'hello'</span><span class="p">)</span>
8907 <span class="go">'hello' # Did you spot the difference?</span>
8908 </pre></div>
8909
8910 <p>That’s because <code>pprint()</code> calls <code>repr()</code> instead of the usual <code>str()</code> for type casting, so that you may evaluate its output as Python code if you want to. The differences become apparent as you start feeding it more complex data structures:</p>
8911 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">data</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'powers'</span><span class="p">:</span> <span class="p">[</span><span class="n">x</span><span class="o">**</span><span class="mi">10</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">)]}</span>
8912 <span class="gp">>>> </span><span class="n">pp</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
8913 <span class="go">{'powers': [0,</span>
8914 <span class="go"> 1,</span>
8915 <span class="go"> 1024,</span>
8916 <span class="go"> 59049,</span>
8917 <span class="go"> 1048576,</span>
8918 <span class="go"> 9765625,</span>
8919 <span class="go"> 60466176,</span>
8920 <span class="go"> 282475249,</span>
8921 <span class="go"> 1073741824,</span>
8922 <span class="go"> 3486784401]}</span>
8923 </pre></div>
8924
8925 <p>The function applies reasonable formatting to improve readability, but you can customize it even further with a couple of parameters. For example, you may limit a deeply nested hierarchy by showing an ellipsis below a given level:</p>
8926 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">cities</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'USA'</span><span class="p">:</span> <span class="p">{</span><span class="s1">'Texas'</span><span class="p">:</span> <span class="p">{</span><span class="s1">'Dallas'</span><span class="p">:</span> <span class="p">[</span><span class="s1">'Irving'</span><span class="p">]}}}</span>
8927 <span class="gp">>>> </span><span class="n">pp</span><span class="p">(</span><span class="n">cities</span><span class="p">,</span> <span class="n">depth</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span>
8928 <span class="go">{'USA': {'Texas': {'Dallas': [...]}}}</span>
8929 </pre></div>
8930
8931 <p>The ordinary <code>print()</code> also uses ellipses but for displaying recursive data structures, which form a cycle, to avoid stack overflow error:</p>
8932 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">items</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span>
8933 <span class="gp">>>> </span><span class="n">items</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">items</span><span class="p">)</span>
8934 <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">items</span><span class="p">)</span>
8935 <span class="go">[1, 2, 3, [...]]</span>
8936 </pre></div>
8937
8938 <p>However, <code>pprint()</code> is more explicit about it by including the unique identity of a self-referencing object:</p>
8939 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">pp</span><span class="p">(</span><span class="n">items</span><span class="p">)</span>
8940 <span class="go">[1, 2, 3, <Recursion on list with id=140635757287688>]</span>
8941 <span class="gp">>>> </span><span class="nb">id</span><span class="p">(</span><span class="n">items</span><span class="p">)</span>
8942 <span class="go">140635757287688</span>
8943 </pre></div>
8944
8945 <p>The last element in the list is the same object as the entire list.</p>
8946 <div class="alert alert-primary" role="alert">
8947 <p><strong>Note:</strong> Recursive or very large data sets can be dealt with using the <code>reprlib</code> module as well:</p>
8948 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">reprlib</span>
8949 <span class="gp">>>> </span><span class="n">reprlib</span><span class="o">.</span><span class="n">repr</span><span class="p">([</span><span class="n">x</span><span class="o">**</span><span class="mi">10</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">)])</span>
8950 <span class="go">'[0, 1, 1024, 59049, 1048576, 9765625, ...]'</span>
8951 </pre></div>
8952
8953 <p>This module supports most of the built-in types and is used by the Python debugger.</p>
8954 </div>
8955 <p><code>pprint()</code> automatically sorts dictionary keys for you before printing, which allows for consistent comparison. When you’re comparing strings, you often don’t care about a particular order of serialized attributes. Anyways, it’s always best to compare actual dictionaries before serialization.</p>
8956 <p>Dictionaries often represent <a href="https://realpython.com/python-json/">JSON data</a>, which is widely used on the Internet. To correctly serialize a dictionary into a valid JSON-formatted string, you can take advantage of the <code>json</code> module. It too has pretty-printing capabilities:</p>
8957 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">json</span>
8958 <span class="gp">>>> </span><span class="n">data</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'username'</span><span class="p">:</span> <span class="s1">'jdoe'</span><span class="p">,</span> <span class="s1">'password'</span><span class="p">:</span> <span class="s1">'s3cret'</span><span class="p">}</span>
8959 <span class="gp">>>> </span><span class="n">ugly</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
8960 <span class="gp">>>> </span><span class="n">pretty</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">indent</span><span class="o">=</span><span class="mi">4</span><span class="p">,</span> <span class="n">sort_keys</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
8961 <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">ugly</span><span class="p">)</span>
8962 <span class="go">{"username": "jdoe", "password": "s3cret"}</span>
8963 <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">pretty</span><span class="p">)</span>
8964 <span class="go">{</span>
8965 <span class="go"> "password": "s3cret",</span>
8966 <span class="go"> "username": "jdoe"</span>
8967 <span class="go">}</span>
8968 </pre></div>
8969
8970 <p>Notice, however, that you need to handle printing yourself, because it’s not something you’d typically want to do. Similarly, the <code>pprint</code> module has an additional <code>pformat()</code> function that returns a string, in case you had to do something other than printing it.</p>
8971 <p>Surprisingly, the signature of <code>pprint()</code> is nothing like the <code>print()</code> function’s one. You can’t even pass more than one positional argument, which shows how much it focuses on printing data structures.</p>
8972 <h3 id="adding-colors-with-ansi-escape-sequences">Adding Colors With ANSI Escape Sequences</h3>
8973 <p>As personal computers got more sophisticated, they had better graphics and could display more colors. However, different vendors had their own idea about the API design for controlling it. That changed a few decades ago when people at the American National Standards Institute decided to unify it by defining <a href="https://en.wikipedia.org/wiki/ANSI_escape_code">ANSI escape codes</a>.</p>
8974 <p>Most of today’s terminal emulators support this standard to some degree. Until recently, the Windows operating system was a notable exception. Therefore, if you want the best portability, use the <a href="https://pypi.org/project/colorama/"><code>colorama</code></a> library in Python. It translates ANSI codes to their appropriate counterparts in Windows while keeping them intact in other operating systems.</p>
8975 <p>To check if your terminal understands a subset of the ANSI escape sequences, for example, related to colors, you can try using the following command:</p>
8976 <div class="highlight sh"><pre><span></span><span class="gp">$</span> tput colors
8977 </pre></div>
8978
8979 <p>My default terminal on Linux says it can display 256 distinct colors, while xterm gives me only 8. The command would return a negative number if colors were unsupported.</p>
8980 <p>ANSI escape sequences are like a markup language for the terminal. In HTML you work with tags, such as <code><b></code> or <code><i></code>, to change how elements look in the document. These tags are mixed with your content, but they’re not visible themselves. Similarly, escape codes won’t show up in the terminal as long as it recognizes them. Otherwise, they’ll appear in the literal form as if you were viewing the source of a website.</p>
8981 <p>As its name implies, a sequence must begin with the non-printable <span class="keys"><kbd class="key-escape">Esc</kbd></span> character, whose ASCII value is 27, sometimes denoted as <code>0x1b</code> in hexadecimal or <code>033</code> in octal. You may use Python number literals to quickly verify it’s indeed the same number:</p>
8982 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="mi">27</span> <span class="o">==</span> <span class="mh">0x1b</span> <span class="o">==</span> <span class="mo">0o33</span>
8983 <span class="go">True</span>
8984 </pre></div>
8985
8986 <p>Additionally, you can obtain it with the <code>\e</code> escape sequence in the shell:</p>
8987 <div class="highlight sh"><pre><span></span><span class="gp">$</span> <span class="nb">echo</span> -e <span class="s2">"\e"</span>
8988 </pre></div>
8989
8990 <p>The most common ANSI escape sequences take the following form:</p>
8991 <div class="table-responsive">
8992 <table class="table table-hover">
8993 <thead>
8994 <tr>
8995 <th>Element</th>
8996 <th>Description</th>
8997 <th>Example</th>
8998 </tr>
8999 </thead>
9000 <tbody>
9001 <tr>
9002 <td><span class="keys"><kbd class="key-escape">Esc</kbd></span></td>
9003 <td>non-printable escape character</td>
9004 <td><code>\033</code></td>
9005 </tr>
9006 <tr>
9007 <td><code>[</code></td>
9008 <td>opening square bracket</td>
9009 <td><code>[</code></td>
9010 </tr>
9011 <tr>
9012 <td>numeric code</td>
9013 <td>one or more numbers separated with <code>;</code></td>
9014 <td><code>0</code></td>
9015 </tr>
9016 <tr>
9017 <td>character code</td>
9018 <td>uppercase or lowercase letter</td>
9019 <td><code>m</code></td>
9020 </tr>
9021 </tbody>
9022 </table>
9023 </div>
9024 <p>The <strong>numeric code</strong> can be one or more numbers separated with a semicolon, while the <strong>character code</strong> is just one letter. Their specific meaning is defined by the ANSI standard. For example, to reset all formatting, you would type one of the following commands, which use the code zero and the letter <code>m</code>:</p>
9025 <div class="highlight sh"><pre><span></span><span class="gp">$</span> <span class="nb">echo</span> -e <span class="s2">"\e[0m"</span>
9026 <span class="gp">$</span> <span class="nb">echo</span> -e <span class="s2">"\x1b[0m"</span>
9027 <span class="gp">$</span> <span class="nb">echo</span> -e <span class="s2">"\033[0m"</span>
9028 </pre></div>
9029
9030 <p>At the other end of the spectrum, you have compound code values. To set foreground and background with RGB channels, given that your terminal supports 24-bit depth, you could provide multiple numbers:</p>
9031 <div class="highlight sh"><pre><span></span><span class="gp">$</span> <span class="nb">echo</span> -e <span class="s2">"\e[38;2;0;0;0m\e[48;2;255;255;255mBlack on white\e[0m"</span>
9032 </pre></div>
9033
9034 <p>It’s not just text color that you can set with the ANSI escape codes. You can, for example, clear and scroll the terminal window, change its background, move the cursor around, make the text blink or decorate it with an underline.</p>
9035 <p>In Python, you’d probably write a helper function to allow for wrapping arbitrary codes into a sequence:</p>
9036 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">def</span> <span class="nf">esc</span><span class="p">(</span><span class="n">code</span><span class="p">):</span>
9037 <span class="gp">... </span> <span class="k">return</span> <span class="n">f</span><span class="s1">'</span><span class="se">\033</span><span class="s1">[</span><span class="si">{code}</span><span class="s1">m'</span>
9038 <span class="gp">...</span>
9039 <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">esc</span><span class="p">(</span><span class="s1">'31;1;4'</span><span class="p">)</span> <span class="o">+</span> <span class="s1">'really'</span> <span class="o">+</span> <span class="n">esc</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">+</span> <span class="s1">' important'</span><span class="p">)</span>
9040 </pre></div>
9041
9042 <p>This would make the word <code>really</code> appear in red, bold, and underlined font:</p>
9043 <p><a href="https://files.realpython.com/media/ansi.21ed85878eb9.png" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/ansi.21ed85878eb9.png" width="576" height="236" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/ansi.21ed85878eb9.png&w=144&sig=1b3e337786114c9e099a662d5e096ce1d41ba6a5 144w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/ansi.21ed85878eb9.png&w=288&sig=cb8720ce7fb4db092d674380ef3bf3543503546a 288w, https://files.realpython.com/media/ansi.21ed85878eb9.png 576w" sizes="75vw" alt="Text formatted with ANSI escape codes"/></a></p>
9044 <p>However, there are higher-level abstractions over ANSI escape codes, such as the mentioned <code>colorama</code> library, as well as tools for building user interfaces in the console.</p>
9045 <h3 id="building-console-user-interfaces">Building Console User Interfaces</h3>
9046 <p>While playing with ANSI escape codes is undeniably a ton of fun, in the real world you’d rather have more abstract building blocks to put together a user interface. There are a few libraries that provide such a high level of control over the terminal, but <a href="https://docs.python.org/3/howto/curses.html"><code>curses</code></a> seems to be the most popular choice.</p>
9047 <div class="alert alert-primary" role="alert">
9048 <p><strong>Note:</strong> To use the <code>curses</code> library in Windows, you need to install a third-party package:</p>
9049 <div class="highlight sh"><pre><span></span><span class="go">C:\> pip install windows-curses</span>
9050 </pre></div>
9051
9052 <p>That’s because <code>curses</code> isn’t available in the standard library of the Python distribution for Windows.</p>
9053 </div>
9054 <p>Primarily, it allows you to think in terms of independent graphical widgets instead of a blob of text. Besides, you get a lot of freedom in expressing your inner artist, because it’s really like painting a blank canvas. The library hides the complexities of having to deal with different terminals. Other than that, it has great support for keyboard events, which might be useful for writing video games.</p>
9055 <p>How about making a retro snake game? Let’s create a Python snake simulator:</p>
9056 <p><a href="https://files.realpython.com/media/snake.a9589582b58a.gif" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/snake.a9589582b58a.gif" width="576" height="392" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/snake.a9589582b58a.gif&w=144&sig=a02bac6a4123b5c1a10112973045e6deec2895c3 144w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/snake.a9589582b58a.gif&w=288&sig=0019bb69f00b55d6201acf7661b870f865758f61 288w, https://files.realpython.com/media/snake.a9589582b58a.gif 576w" sizes="75vw" alt="The retro snake game built with curses library"/></a></p>
9057 <p>First, you need to import the <code>curses</code> module. Since it modifies the state of a running terminal, it’s important to handle errors and gracefully restore the previous state. You can do this manually, but the library comes with a convenient wrapper for your main function:</p>
9058 <div class="highlight python"><pre><span></span><span class="kn">import</span> <span class="nn">curses</span>
9059
9060 <span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="n">screen</span><span class="p">):</span>
9061 <span class="k">pass</span>
9062
9063 <span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
9064 <span class="n">curses</span><span class="o">.</span><span class="n">wrapper</span><span class="p">(</span><span class="n">main</span><span class="p">)</span>
9065 </pre></div>
9066
9067 <p>Note, the function must accept a reference to the screen object, also known as <code>stdscr</code>, that you’ll use later for additional setup.</p>
9068 <p>If you run this program now, you won’t see any effects, because it terminates immediately. However, you can add a small delay to have a sneak peek:</p>
9069 <div class="highlight python"><pre><span></span><span class="kn">import</span> <span class="nn">time</span><span class="o">,</span> <span class="nn">curses</span>
9070
9071 <span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="n">screen</span><span class="p">):</span>
9072 <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
9073
9074 <span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
9075 <span class="n">curses</span><span class="o">.</span><span class="n">wrapper</span><span class="p">(</span><span class="n">main</span><span class="p">)</span>
9076 </pre></div>
9077
9078 <p>This time the screen went completely blank for a second, but the cursor was still blinking. To hide it, just call one of the configuration functions defined in the module:</p>
9079 <div class="highlight python"><pre><span></span><span class="kn">import</span> <span class="nn">time</span><span class="o">,</span> <span class="nn">curses</span>
9080
9081 <span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="n">screen</span><span class="p">):</span>
9082 <span class="hll"> <span class="n">curses</span><span class="o">.</span><span class="n">curs_set</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="c1"># Hide the cursor</span>
9083 </span> <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
9084
9085 <span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
9086 <span class="n">curses</span><span class="o">.</span><span class="n">wrapper</span><span class="p">(</span><span class="n">main</span><span class="p">)</span>
9087 </pre></div>
9088
9089 <p>Let’s define the snake as a list of points in screen coordinates:</p>
9090 <div class="highlight python"><pre><span></span><span class="n">snake</span> <span class="o">=</span> <span class="p">[(</span><span class="mi">0</span><span class="p">,</span> <span class="n">i</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">reversed</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="mi">20</span><span class="p">))]</span>
9091 </pre></div>
9092
9093 <p>The head of the snake is always the first element in the list, whereas the tail is the last one. The initial shape of the snake is horizontal, starting from the top-left corner of the screen and facing to the right. While its y-coordinate stays at zero, its x-coordinate decreases from head to tail.</p>
9094 <p>To draw the snake, you’ll start with the head and then follow with the remaining segments. Each segment carries <code>(y, x)</code> coordinates, so you can unpack them:</p>
9095 <div class="highlight python"><pre><span></span><span class="c1"># Draw the snake</span>
9096 <span class="n">screen</span><span class="o">.</span><span class="n">addstr</span><span class="p">(</span><span class="o">*</span><span class="n">snake</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="s1">'@'</span><span class="p">)</span>
9097 <span class="k">for</span> <span class="n">segment</span> <span class="ow">in</span> <span class="n">snake</span><span class="p">[</span><span class="mi">1</span><span class="p">:]:</span>
9098 <span class="n">screen</span><span class="o">.</span><span class="n">addstr</span><span class="p">(</span><span class="o">*</span><span class="n">segment</span><span class="p">,</span> <span class="s1">'*'</span><span class="p">)</span>
9099 </pre></div>
9100
9101 <p>Again, if you run this code now, it won’t display anything, because you must explicitly refresh the screen afterward:</p>
9102 <div class="highlight python"><pre><span></span><span class="kn">import</span> <span class="nn">time</span><span class="o">,</span> <span class="nn">curses</span>
9103
9104 <span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="n">screen</span><span class="p">):</span>
9105 <span class="n">curses</span><span class="o">.</span><span class="n">curs_set</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="c1"># Hide the cursor</span>
9106
9107 <span class="n">snake</span> <span class="o">=</span> <span class="p">[(</span><span class="mi">0</span><span class="p">,</span> <span class="n">i</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">reversed</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="mi">20</span><span class="p">))]</span>
9108
9109 <span class="c1"># Draw the snake</span>
9110 <span class="n">screen</span><span class="o">.</span><span class="n">addstr</span><span class="p">(</span><span class="o">*</span><span class="n">snake</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="s1">'@'</span><span class="p">)</span>
9111 <span class="k">for</span> <span class="n">segment</span> <span class="ow">in</span> <span class="n">snake</span><span class="p">[</span><span class="mi">1</span><span class="p">:]:</span>
9112 <span class="n">screen</span><span class="o">.</span><span class="n">addstr</span><span class="p">(</span><span class="o">*</span><span class="n">segment</span><span class="p">,</span> <span class="s1">'*'</span><span class="p">)</span>
9113
9114 <span class="hll"> <span class="n">screen</span><span class="o">.</span><span class="n">refresh</span><span class="p">()</span>
9115 </span> <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
9116
9117 <span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
9118 <span class="n">curses</span><span class="o">.</span><span class="n">wrapper</span><span class="p">(</span><span class="n">main</span><span class="p">)</span>
9119 </pre></div>
9120
9121 <p>You want to move the snake in one of four directions, which can be defined as vectors. Eventually, the direction will change in response to an arrow keystroke, so you may hook it up to the library’s key codes:</p>
9122 <div class="highlight python"><pre><span></span><span class="n">directions</span> <span class="o">=</span> <span class="p">{</span>
9123 <span class="n">curses</span><span class="o">.</span><span class="n">KEY_UP</span><span class="p">:</span> <span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">),</span>
9124 <span class="n">curses</span><span class="o">.</span><span class="n">KEY_DOWN</span><span class="p">:</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">),</span>
9125 <span class="n">curses</span><span class="o">.</span><span class="n">KEY_LEFT</span><span class="p">:</span> <span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">),</span>
9126 <span class="n">curses</span><span class="o">.</span><span class="n">KEY_RIGHT</span><span class="p">:</span> <span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span>
9127 <span class="p">}</span>
9128
9129 <span class="n">direction</span> <span class="o">=</span> <span class="n">directions</span><span class="p">[</span><span class="n">curses</span><span class="o">.</span><span class="n">KEY_RIGHT</span><span class="p">]</span>
9130 </pre></div>
9131
9132 <p>How does a snake move? It turns out that only its head really moves to a new location, while all other segments shift towards it. In each step, almost all segments remain the same, except for the head and the tail. Assuming the snake isn’t growing, you can remove the tail and insert a new head at the beginning of the list:</p>
9133 <div class="highlight python"><pre><span></span><span class="c1"># Move the snake</span>
9134 <span class="n">snake</span><span class="o">.</span><span class="n">pop</span><span class="p">()</span>
9135 <span class="n">snake</span><span class="o">.</span><span class="n">insert</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="nb">tuple</span><span class="p">(</span><span class="nb">map</span><span class="p">(</span><span class="nb">sum</span><span class="p">,</span> <span class="nb">zip</span><span class="p">(</span><span class="n">snake</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">direction</span><span class="p">))))</span>
9136 </pre></div>
9137
9138 <p>To get the new coordinates of the head, you need to add the direction vector to it. However, adding tuples in Python results in a bigger tuple instead of the algebraic sum of the corresponding vector components. One way to fix this is by using the built-in <code>zip()</code>, <code>sum()</code>, and <code>map()</code> functions.</p>
9139 <p>The direction will change on a keystroke, so you need to call <code>.getch()</code> to obtain the pressed key code. However, if the pressed key doesn’t correspond to the arrow keys defined earlier as dictionary keys, the direction won’t change:</p>
9140 <div class="highlight python"><pre><span></span><span class="c1"># Change direction on arrow keystroke</span>
9141 <span class="n">direction</span> <span class="o">=</span> <span class="n">directions</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">screen</span><span class="o">.</span><span class="n">getch</span><span class="p">(),</span> <span class="n">direction</span><span class="p">)</span>
9142 </pre></div>
9143
9144 <p>By default, however, <code>.getch()</code> is a blocking call that would prevent the snake from moving unless there was a keystroke. Therefore, you need to make the call non-blocking by adding yet another configuration:</p>
9145 <div class="highlight python"><pre><span></span><span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="n">screen</span><span class="p">):</span>
9146 <span class="n">curses</span><span class="o">.</span><span class="n">curs_set</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="c1"># Hide the cursor</span>
9147 <span class="hll"> <span class="n">screen</span><span class="o">.</span><span class="n">nodelay</span><span class="p">(</span><span class="kc">True</span><span class="p">)</span> <span class="c1"># Don't block I/O calls</span>
9148 </span></pre></div>
9149
9150 <p>You’re almost done, but there’s just one last thing left. If you now loop this code, the snake will appear to be growing instead of moving. That’s because you have to erase the screen explicitly before each iteration.</p>
9151 <p>Finally, this is all you need to play the snake game in Python:</p>
9152 <div class="highlight python"><pre><span></span><span class="kn">import</span> <span class="nn">time</span><span class="o">,</span> <span class="nn">curses</span>
9153
9154 <span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="n">screen</span><span class="p">):</span>
9155 <span class="n">curses</span><span class="o">.</span><span class="n">curs_set</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="c1"># Hide the cursor</span>
9156 <span class="n">screen</span><span class="o">.</span><span class="n">nodelay</span><span class="p">(</span><span class="kc">True</span><span class="p">)</span> <span class="c1"># Don't block I/O calls</span>
9157
9158 <span class="n">directions</span> <span class="o">=</span> <span class="p">{</span>
9159 <span class="n">curses</span><span class="o">.</span><span class="n">KEY_UP</span><span class="p">:</span> <span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">),</span>
9160 <span class="n">curses</span><span class="o">.</span><span class="n">KEY_DOWN</span><span class="p">:</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">),</span>
9161 <span class="n">curses</span><span class="o">.</span><span class="n">KEY_LEFT</span><span class="p">:</span> <span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">),</span>
9162 <span class="n">curses</span><span class="o">.</span><span class="n">KEY_RIGHT</span><span class="p">:</span> <span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span>
9163 <span class="p">}</span>
9164
9165 <span class="n">direction</span> <span class="o">=</span> <span class="n">directions</span><span class="p">[</span><span class="n">curses</span><span class="o">.</span><span class="n">KEY_RIGHT</span><span class="p">]</span>
9166 <span class="n">snake</span> <span class="o">=</span> <span class="p">[(</span><span class="mi">0</span><span class="p">,</span> <span class="n">i</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">reversed</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="mi">20</span><span class="p">))]</span>
9167
9168 <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
9169 <span class="hll"> <span class="n">screen</span><span class="o">.</span><span class="n">erase</span><span class="p">()</span>
9170 </span>
9171 <span class="c1"># Draw the snake</span>
9172 <span class="n">screen</span><span class="o">.</span><span class="n">addstr</span><span class="p">(</span><span class="o">*</span><span class="n">snake</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="s1">'@'</span><span class="p">)</span>
9173 <span class="k">for</span> <span class="n">segment</span> <span class="ow">in</span> <span class="n">snake</span><span class="p">[</span><span class="mi">1</span><span class="p">:]:</span>
9174 <span class="n">screen</span><span class="o">.</span><span class="n">addstr</span><span class="p">(</span><span class="o">*</span><span class="n">segment</span><span class="p">,</span> <span class="s1">'*'</span><span class="p">)</span>
9175
9176 <span class="c1"># Move the snake</span>
9177 <span class="n">snake</span><span class="o">.</span><span class="n">pop</span><span class="p">()</span>
9178 <span class="n">snake</span><span class="o">.</span><span class="n">insert</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="nb">tuple</span><span class="p">(</span><span class="nb">map</span><span class="p">(</span><span class="nb">sum</span><span class="p">,</span> <span class="nb">zip</span><span class="p">(</span><span class="n">snake</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">direction</span><span class="p">))))</span>
9179
9180 <span class="c1"># Change direction on arrow keystroke</span>
9181 <span class="n">direction</span> <span class="o">=</span> <span class="n">directions</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">screen</span><span class="o">.</span><span class="n">getch</span><span class="p">(),</span> <span class="n">direction</span><span class="p">)</span>
9182
9183 <span class="n">screen</span><span class="o">.</span><span class="n">refresh</span><span class="p">()</span>
9184 <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.1</span><span class="p">)</span>
9185
9186 <span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
9187 <span class="n">curses</span><span class="o">.</span><span class="n">wrapper</span><span class="p">(</span><span class="n">main</span><span class="p">)</span>
9188 </pre></div>
9189
9190 <p>This is merely scratching the surface of the possibilities that the <code>curses</code> module opens up. You may use it for game development like this or more business-oriented applications.</p>
9191 <h3 id="living-it-up-with-cool-animations">Living It Up With Cool Animations</h3>
9192 <p>Not only can animations make the user interface more appealing to the eye, but they also improve the overall user experience. When you provide early feedback to the user, for example, they’ll know if your program’s still working or if it’s time to kill it.</p>
9193 <p>To animate text in the terminal, you have to be able to freely move the cursor around. You can do this with one of the tools mentioned previously, that is ANSI escape codes or the <code>curses</code> library. However, I’d like to show you an even simpler way.</p>
9194 <p>If the animation can be constrained to a single line of text, then you might be interested in two special escape character sequences:</p>
9195 <ul>
9196 <li><strong>Carriage return:</strong> <code>\r</code></li>
9197 <li><strong>Backspace:</strong> <code>\b</code></li>
9198 </ul>
9199 <p>The first one moves the cursor to the beginning of the line, whereas the second one moves it only one character to the left. They both work in a non-destructive way without overwriting text that’s already been written.</p>
9200 <p>Let’s take a look at a few examples.</p>
9201 <p>You’ll often want to display some kind of a <strong>spinning wheel</strong> to indicate a work in progress without knowing exactly how much time’s left to finish:</p>
9202 <p><a href="https://files.realpython.com/media/spinning_wheel.c595af6f83ea.gif" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/spinning_wheel.c595af6f83ea.gif" width="576" height="236" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/spinning_wheel.c595af6f83ea.gif&w=144&sig=6668f7484a9d15711f5a9f149647b38baf8683cb 144w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/spinning_wheel.c595af6f83ea.gif&w=288&sig=267bb31016ff764a987006dcecafcf80233af3ad 288w, https://files.realpython.com/media/spinning_wheel.c595af6f83ea.gif 576w" sizes="75vw" alt="Indefinite animation in the terminal"/></a></p>
9203 <p>Many command line tools use this trick while downloading data over the network. You can make a really simple stop motion animation from a sequence of characters that will cycle in a round-robin fashion:</p>
9204 <div class="highlight python"><pre><span></span><span class="kn">from</span> <span class="nn">itertools</span> <span class="k">import</span> <span class="n">cycle</span>
9205 <span class="kn">from</span> <span class="nn">time</span> <span class="k">import</span> <span class="n">sleep</span>
9206
9207 <span class="k">for</span> <span class="n">frame</span> <span class="ow">in</span> <span class="n">cycle</span><span class="p">(</span><span class="sa">r</span><span class="s1">'-\|/-\|/'</span><span class="p">):</span>
9208 <span class="nb">print</span><span class="p">(</span><span class="s1">'</span><span class="se">\r</span><span class="s1">'</span><span class="p">,</span> <span class="n">frame</span><span class="p">,</span> <span class="n">sep</span><span class="o">=</span><span class="s1">''</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s1">''</span><span class="p">,</span> <span class="n">flush</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
9209 <span class="n">sleep</span><span class="p">(</span><span class="mf">0.2</span><span class="p">)</span>
9210 </pre></div>
9211
9212 <p>The loop gets the next character to print, then moves the cursor to the beginning of the line, and overwrites whatever there was before without adding a newline. You don’t want extra space between positional arguments, so separator argument must be blank. Also, notice the use of Python’s raw strings due to backslash characters present in the literal.</p>
9213 <p>When you know the remaining time or task completion percentage, then you’re able to show an animated progress bar:</p>
9214 <p><a href="https://files.realpython.com/media/progress.6bd055d8dcc4.gif" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/progress.6bd055d8dcc4.gif" width="576" height="236" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/progress.6bd055d8dcc4.gif&w=144&sig=b77d52a79d28329d6ec8d5e43fbffd8bc112896e 144w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/progress.6bd055d8dcc4.gif&w=288&sig=57b015e2b6a7534b9a9d027d81a71b52967c20e0 288w, https://files.realpython.com/media/progress.6bd055d8dcc4.gif 576w" sizes="75vw" alt="Progress bar animation in the terminal"/></a></p>
9215 <p>First, you need to calculate how many hashtags to display and how many blank spaces to insert. Next, you erase the line and build the bar from scratch:</p>
9216 <div class="highlight python"><pre><span></span><span class="kn">from</span> <span class="nn">time</span> <span class="k">import</span> <span class="n">sleep</span>
9217
9218 <span class="k">def</span> <span class="nf">progress</span><span class="p">(</span><span class="n">percent</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="n">width</span><span class="o">=</span><span class="mi">30</span><span class="p">):</span>
9219 <span class="n">left</span> <span class="o">=</span> <span class="n">width</span> <span class="o">*</span> <span class="n">percent</span> <span class="o">//</span> <span class="mi">100</span>
9220 <span class="n">right</span> <span class="o">=</span> <span class="n">width</span> <span class="o">-</span> <span class="n">left</span>
9221 <span class="nb">print</span><span class="p">(</span><span class="s1">'</span><span class="se">\r</span><span class="s1">['</span><span class="p">,</span> <span class="s1">'#'</span> <span class="o">*</span> <span class="n">left</span><span class="p">,</span> <span class="s1">' '</span> <span class="o">*</span> <span class="n">right</span><span class="p">,</span> <span class="s1">']'</span><span class="p">,</span>
9222 <span class="n">f</span><span class="s1">' </span><span class="si">{percent:.0f}</span><span class="s1">%'</span><span class="p">,</span>
9223 <span class="n">sep</span><span class="o">=</span><span class="s1">''</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s1">''</span><span class="p">,</span> <span class="n">flush</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
9224
9225 <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">101</span><span class="p">):</span>
9226 <span class="n">progress</span><span class="p">(</span><span class="n">i</span><span class="p">)</span>
9227 <span class="n">sleep</span><span class="p">(</span><span class="mf">0.1</span><span class="p">)</span>
9228 </pre></div>
9229
9230 <p>As before, each request for update repaints the entire line.</p>
9231 <div class="alert alert-primary" role="alert">
9232 <p><strong>Note:</strong> There’s a feature-rich <a href="https://pypi.org/project/progressbar2/"><code>progressbar2</code></a> library, along with a few other similar tools, that can show progress in a much more comprehensive way.</p>
9233 </div>
9234 <h3 id="making-sounds-with-print">Making Sounds With Print</h3>
9235 <p>If you’re old enough to remember computers with a PC speaker, then you must also remember their distinctive <em>beep</em> sound, often used to indicate hardware problems. They could barely make any more noises than that, yet video games seemed so much better with it.</p>
9236 <p>Today you can still take advantage of this small loudspeaker, but chances are your laptop didn’t come with one. In such a case, you can enable <strong>terminal bell</strong> emulation in your shell, so that a system warning sound is played instead.</p>
9237 <p>Go ahead and type this command to see if your terminal can play a sound:</p>
9238 <div class="highlight sh"><pre><span></span><span class="gp">$</span> <span class="nb">echo</span> -e <span class="s2">"\a"</span>
9239 </pre></div>
9240
9241 <p>This would normally print text, but the <code>-e</code> flag enables the interpretation of backslash escapes. As you can see, there’s a dedicated escape sequence <code>\a</code>, which stands for “alert”, that outputs a special <a href="https://en.wikipedia.org/wiki/Bell_character">bell character</a>. Some terminals make a sound whenever they see it.</p>
9242 <p>Similarly, you can print this character in Python. Perhaps in a loop to form some kind of melody. While it’s only a single note, you can still vary the length of pauses between consecutive instances. That seems like a perfect toy for Morse code playback!</p>
9243 <p>The rules are the following:</p>
9244 <ul>
9245 <li>Letters are encoded with a sequence of <strong>dot</strong> (·) and <strong>dash</strong> (–) symbols.</li>
9246 <li>A <strong>dot</strong> is one unit of time.</li>
9247 <li>A <strong>dash</strong> is three units of time.</li>
9248 <li>Individual <strong>symbols</strong> in a letter are spaced one unit of time apart.</li>
9249 <li>Symbols of two adjacent <strong>letters</strong> are spaced three units of time apart.</li>
9250 <li>Symbols of two adjacent <strong>words</strong> are spaced seven units of time apart.</li>
9251 </ul>
9252 <p>According to those rules, you could be “printing” an SOS signal indefinitely in the following way:</p>
9253 <div class="highlight python"><pre><span></span><span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
9254 <span class="n">dot</span><span class="p">()</span>
9255 <span class="n">symbol_space</span><span class="p">()</span>
9256 <span class="n">dot</span><span class="p">()</span>
9257 <span class="n">symbol_space</span><span class="p">()</span>
9258 <span class="n">dot</span><span class="p">()</span>
9259 <span class="n">letter_space</span><span class="p">()</span>
9260 <span class="n">dash</span><span class="p">()</span>
9261 <span class="n">symbol_space</span><span class="p">()</span>
9262 <span class="n">dash</span><span class="p">()</span>
9263 <span class="n">symbol_space</span><span class="p">()</span>
9264 <span class="n">dash</span><span class="p">()</span>
9265 <span class="n">letter_space</span><span class="p">()</span>
9266 <span class="n">dot</span><span class="p">()</span>
9267 <span class="n">symbol_space</span><span class="p">()</span>
9268 <span class="n">dot</span><span class="p">()</span>
9269 <span class="n">symbol_space</span><span class="p">()</span>
9270 <span class="n">dot</span><span class="p">()</span>
9271 <span class="n">word_space</span><span class="p">()</span>
9272 </pre></div>
9273
9274 <p>In Python, you can implement it in merely ten lines of code:</p>
9275 <div class="highlight python"><pre><span></span><span class="kn">from</span> <span class="nn">time</span> <span class="k">import</span> <span class="n">sleep</span>
9276
9277 <span class="n">speed</span> <span class="o">=</span> <span class="mf">0.1</span>
9278
9279 <span class="k">def</span> <span class="nf">signal</span><span class="p">(</span><span class="n">duration</span><span class="p">,</span> <span class="n">symbol</span><span class="p">):</span>
9280 <span class="n">sleep</span><span class="p">(</span><span class="n">duration</span><span class="p">)</span>
9281 <span class="nb">print</span><span class="p">(</span><span class="n">symbol</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s1">''</span><span class="p">,</span> <span class="n">flush</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
9282
9283 <span class="n">dot</span> <span class="o">=</span> <span class="k">lambda</span><span class="p">:</span> <span class="n">signal</span><span class="p">(</span><span class="n">speed</span><span class="p">,</span> <span class="s1">'Β·</span><span class="se">\a</span><span class="s1">'</span><span class="p">)</span>
9284 <span class="n">dash</span> <span class="o">=</span> <span class="k">lambda</span><span class="p">:</span> <span class="n">signal</span><span class="p">(</span><span class="mi">3</span><span class="o">*</span><span class="n">speed</span><span class="p">,</span> <span class="s1">'β</span><span class="se">\a</span><span class="s1">'</span><span class="p">)</span>
9285 <span class="n">symbol_space</span> <span class="o">=</span> <span class="k">lambda</span><span class="p">:</span> <span class="n">signal</span><span class="p">(</span><span class="n">speed</span><span class="p">,</span> <span class="s1">''</span><span class="p">)</span>
9286 <span class="n">letter_space</span> <span class="o">=</span> <span class="k">lambda</span><span class="p">:</span> <span class="n">signal</span><span class="p">(</span><span class="mi">3</span><span class="o">*</span><span class="n">speed</span><span class="p">,</span> <span class="s1">''</span><span class="p">)</span>
9287 <span class="n">word_space</span> <span class="o">=</span> <span class="k">lambda</span><span class="p">:</span> <span class="n">signal</span><span class="p">(</span><span class="mi">7</span><span class="o">*</span><span class="n">speed</span><span class="p">,</span> <span class="s1">' '</span><span class="p">)</span>
9288 </pre></div>
9289
9290 <p>Maybe you could even take it one step further and make a command line tool for translating text into Morse code? Either way, I hope you’re having fun with this!</p>
9291 <h2 id="mocking-python-print-in-unit-tests">Mocking Python Print in Unit Tests</h2>
9292 <p>Nowadays, it’s expected that you ship code that meets high quality standards. If you aspire to become a professional, you must learn <a href="https://realpython.com/python-testing/">how to test</a> your code.</p>
9293 <p>Software testing is especially important in dynamically typed languages, such as Python, which don’t have a compiler to warn you about obvious mistakes. Defects can make their way to the production environment and remain dormant for a long time, until that one day when a branch of code finally gets executed.</p>
9294 <p>Sure, you have <a href="https://realpython.com/python-code-quality/#linters">linters</a>, <a href="https://realpython.com/python-type-checking/">type checkers</a>, and other tools for static code analysis to assist you. But they won’t tell you whether your program does what it’s supposed to do on the business level.</p>
9295 <p>So, should you be testing <code>print()</code>? No. After all, it’s a built-in function that must have already gone through a comprehensive suite of tests. What you want to test, though, is whether your code is calling <code>print()</code> at the right time with the expected parameters. That’s known as a <strong>behavior</strong>.</p>
9296 <p>You can test behaviors by <a href="https://realpython.com/python-mock-library/">mocking</a> real objects or functions. In this case, you want to mock <code>print()</code> to record and verify its invocations.</p>
9297 <div class="alert alert-primary" role="alert">
9298 <p><strong>Note:</strong> You might have heard the terms: <strong>dummy</strong>, <strong>fake</strong>, <strong>stub</strong>, <strong>spy</strong>, or <strong>mock</strong> used interchangeably. Some people make a distinction between them, while others don’t.</p>
9299 <p>Martin Fowler explains their differences in a <a href="https://martinfowler.com/bliki/TestDouble.html">short glossary</a> and collectively calls them <strong>test doubles</strong>.</p>
9300 </div>
9301 <p>Mocking in Python can be done twofold. First, you can take the traditional path of statically-typed languages by employing dependency injection. This may sometimes require you to change the code under test, which isn’t always possible if the code is defined in an external library:</p>
9302 <div class="highlight python"><pre><span></span><span class="k">def</span> <span class="nf">download</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">log</span><span class="o">=</span><span class="nb">print</span><span class="p">):</span>
9303 <span class="n">log</span><span class="p">(</span><span class="n">f</span><span class="s1">'Downloading </span><span class="si">{url}</span><span class="s1">'</span><span class="p">)</span>
9304 <span class="c1"># ...</span>
9305 </pre></div>
9306
9307 <p>This is the same example I used in an earlier section to talk about function composition. It basically allows for substituting <code>print()</code> with a custom function of the same interface. To check if it prints the right message, you have to intercept it by injecting a mocked function:</p>
9308 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">def</span> <span class="nf">mock_print</span><span class="p">(</span><span class="n">message</span><span class="p">):</span>
9309 <span class="gp">... </span> <span class="n">mock_print</span><span class="o">.</span><span class="n">last_message</span> <span class="o">=</span> <span class="n">message</span>
9310 <span class="gp">...</span>
9311 <span class="gp">>>> </span><span class="n">download</span><span class="p">(</span><span class="s1">'resource'</span><span class="p">,</span> <span class="n">mock_print</span><span class="p">)</span>
9312 <span class="gp">>>> </span><span class="k">assert</span> <span class="s1">'Downloading resource'</span> <span class="o">==</span> <span class="n">mock_print</span><span class="o">.</span><span class="n">last_message</span>
9313 </pre></div>
9314
9315 <p>Calling this mock makes it save the last message in an attribute, which you can inspect later, for example in an <code>assert</code> statement.</p>
9316 <p>In a slightly alternative solution, instead of replacing the entire <code>print()</code> function with a custom wrapper, you could redirect the standard output to an in-memory file-like stream of characters:</p>
9317 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">def</span> <span class="nf">download</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">stream</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
9318 <span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Downloading </span><span class="si">{url}</span><span class="s1">'</span><span class="p">,</span> <span class="n">file</span><span class="o">=</span><span class="n">stream</span><span class="p">)</span>
9319 <span class="gp">... </span> <span class="c1"># ...</span>
9320 <span class="gp">...</span>
9321 <span class="gp">>>> </span><span class="kn">import</span> <span class="nn">io</span>
9322 <span class="gp">>>> </span><span class="n">memory_buffer</span> <span class="o">=</span> <span class="n">io</span><span class="o">.</span><span class="n">StringIO</span><span class="p">()</span>
9323 <span class="gp">>>> </span><span class="n">download</span><span class="p">(</span><span class="s1">'app.js'</span><span class="p">,</span> <span class="n">memory_buffer</span><span class="p">)</span>
9324 <span class="gp">>>> </span><span class="n">download</span><span class="p">(</span><span class="s1">'style.css'</span><span class="p">,</span> <span class="n">memory_buffer</span><span class="p">)</span>
9325 <span class="gp">>>> </span><span class="n">memory_buffer</span><span class="o">.</span><span class="n">getvalue</span><span class="p">()</span>
9326 <span class="go">'Downloading app.js\nDownloading style.css\n'</span>
9327 </pre></div>
9328
9329 <p>This time the function explicitly calls <code>print()</code>, but it exposes its <code>file</code> parameter to the outside world.</p>
9330 <p>However, a more Pythonic way of mocking objects takes advantage of the built-in <code>mock</code> module, which uses a technique called <a href="https://en.wikipedia.org/wiki/Monkey_patch">monkey patching</a>. This derogatory name stems from it being a “dirty hack” that you can easily shoot yourself in the foot with. It’s less elegant than dependency injection but definitely quick and convenient.</p>
9331 <div class="alert alert-primary" role="alert">
9332 <p><strong>Note:</strong> The <code>mock</code> module got absorbed by the standard library in Python 3, but before that, it was a third-party package. You had to install it separately:</p>
9333 <div class="highlight sh"><pre><span></span><span class="gp">$</span> pip2 install mock
9334 </pre></div>
9335
9336 <p>Other than that, you referred to it as <code>mock</code>, whereas in Python 3 it’s part of the unit testing module, so you must import from <code>unittest.mock</code>.</p>
9337 </div>
9338 <p>What monkey patching does is alter implementation dynamically at runtime. Such a change is visible globally, so it may have unwanted consequences. In practice, however, patching only affects the code for the duration of test execution.</p>
9339 <p>To mock <code>print()</code> in a test case, you’ll typically use the <code>@patch</code> <a href="https://realpython.com/primer-on-python-decorators/">decorator</a> and specify a target for patching by referring to it with a fully qualified name, that is including the module name:</p>
9340 <div class="highlight python"><pre><span></span><span class="kn">from</span> <span class="nn">unittest.mock</span> <span class="k">import</span> <span class="n">patch</span>
9341
9342 <span class="nd">@patch</span><span class="p">(</span><span class="s1">'builtins.print'</span><span class="p">)</span>
9343 <span class="k">def</span> <span class="nf">test_print</span><span class="p">(</span><span class="n">mock_print</span><span class="p">):</span>
9344 <span class="nb">print</span><span class="p">(</span><span class="s1">'not a real print'</span><span class="p">)</span>
9345 <span class="n">mock_print</span><span class="o">.</span><span class="n">assert_called_with</span><span class="p">(</span><span class="s1">'not a real print'</span><span class="p">)</span>
9346 </pre></div>
9347
9348 <p>This will automatically create the mock for you and inject it to the test function. However, you need to declare that your test function accepts a mock now. The underlying mock object has lots of useful methods and attributes for verifying behavior.</p>
9349 <p>Did you notice anything peculiar about that code snippet?</p>
9350 <p>Despite injecting a mock to the function, you’re not calling it directly, although you could. That injected mock is only used to make assertions afterward and maybe to prepare the context before running the test.</p>
9351 <p>In real life, mocking helps to isolate the code under test by removing dependencies such as a database connection. You rarely call mocks in a test, because that doesn’t make much sense. Rather, it’s other pieces of code that call your mock indirectly without knowing it.</p>
9352 <p>Here’s what that means:</p>
9353 <div class="highlight python"><pre><span></span><span class="kn">from</span> <span class="nn">unittest.mock</span> <span class="k">import</span> <span class="n">patch</span>
9354
9355 <span class="k">def</span> <span class="nf">greet</span><span class="p">(</span><span class="n">name</span><span class="p">):</span>
9356 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Hello, </span><span class="si">{name}</span><span class="s1">!'</span><span class="p">)</span>
9357
9358 <span class="nd">@patch</span><span class="p">(</span><span class="s1">'builtins.print'</span><span class="p">)</span>
9359 <span class="k">def</span> <span class="nf">test_greet</span><span class="p">(</span><span class="n">mock_print</span><span class="p">):</span>
9360 <span class="n">greet</span><span class="p">(</span><span class="s1">'John'</span><span class="p">)</span>
9361 <span class="n">mock_print</span><span class="o">.</span><span class="n">assert_called_with</span><span class="p">(</span><span class="s1">'Hello, John!'</span><span class="p">)</span>
9362 </pre></div>
9363
9364 <p>The code under test is a function that prints a greeting. Even though it’s a fairly simple function, you can’t test it easily because it doesn’t return a value. It has a side-effect.</p>
9365 <p>To eliminate that side-effect, you need to mock the dependency out. Patching lets you avoid making changes to the original function, which can remain agnostic about <code>print()</code>. It thinks it’s calling <code>print()</code>, but in reality, it’s calling a mock you’re in total control of.</p>
9366 <p>There are many reasons for testing software. One of them is looking for bugs. When you write tests, you often want to get rid of the <code>print()</code> function, for example, by mocking it away. Paradoxically, however, that same function can help you find bugs during a related process of debugging you’ll read about in the next section.</p>
9367 <div class="card mb-3" id="collapse_card841486">
9368 <div class="card-header border-0"><p class="m-0"><button class="btn" data-toggle="collapse" data-target="#collapse841486" aria-expanded="false" aria-controls="collapse841486">Syntax in Python 2</button> <button class="btn btn-link float-right" data-toggle="collapse" data-target="#collapse841486" aria-expanded="false" aria-controls="collapse841486">Show/Hide</button></p></div>
9369 <div id="collapse841486" class="collapse" data-parent="#collapse_card841486"><div class="card-body" markdown="1">
9370
9371 <p>You can’t monkey patch the <code>print</code> statement in Python 2, nor can you inject it as a dependency. However, you have a few other options:</p>
9372 <ul>
9373 <li>Use stream redirection.</li>
9374 <li>Patch the standard output defined in the <code>sys</code> module.</li>
9375 <li>Import <code>print()</code> from the <code>__future__</code> module.</li>
9376 </ul>
9377 <p>Let’s examine them one by one.</p>
9378 <p>Stream redirection is almost identical to the example you saw earlier:</p>
9379 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">def</span> <span class="nf">download</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">stream</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
9380 <span class="gp">... </span> <span class="k">print</span> <span class="o">>></span> <span class="n">stream</span><span class="p">,</span> <span class="s1">'Downloading </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="n">url</span>
9381 <span class="gp">... </span> <span class="c1"># ...</span>
9382 <span class="gp">...</span>
9383 <span class="gp">>>> </span><span class="kn">from</span> <span class="nn">StringIO</span> <span class="kn">import</span> <span class="n">StringIO</span>
9384 <span class="gp">>>> </span><span class="n">memory_buffer</span> <span class="o">=</span> <span class="n">StringIO</span><span class="p">()</span>
9385 <span class="gp">>>> </span><span class="n">download</span><span class="p">(</span><span class="s1">'app.js'</span><span class="p">,</span> <span class="n">memory_buffer</span><span class="p">)</span>
9386 <span class="gp">>>> </span><span class="n">download</span><span class="p">(</span><span class="s1">'style.css'</span><span class="p">,</span> <span class="n">memory_buffer</span><span class="p">)</span>
9387 <span class="gp">>>> </span><span class="n">memory_buffer</span><span class="o">.</span><span class="n">getvalue</span><span class="p">()</span>
9388 <span class="go">'Downloading app.js\nDownloading style.css\n'</span>
9389 </pre></div>
9390
9391 <p>There are only two differences. First, the syntax for stream redirection uses chevron (<code>>></code>) instead of the <code>file</code> argument. The other difference is where <code>StringIO</code> is defined. You can import it from a similarly named <code>StringIO</code> module, or <code>cStringIO</code> for a faster implementation.</p>
9392 <p>Patching the standard output from the <code>sys</code> module is exactly what it sounds like, but you need to be aware of a few gotchas:</p>
9393 <div class="highlight python"><pre><span></span><span class="kn">from</span> <span class="nn">mock</span> <span class="kn">import</span> <span class="n">patch</span><span class="p">,</span> <span class="n">call</span>
9394
9395 <span class="k">def</span> <span class="nf">greet</span><span class="p">(</span><span class="n">name</span><span class="p">):</span>
9396 <span class="k">print</span> <span class="s1">'Hello, </span><span class="si">%s</span><span class="s1">!'</span> <span class="o">%</span> <span class="n">name</span>
9397
9398 <span class="nd">@patch</span><span class="p">(</span><span class="s1">'sys.stdout'</span><span class="p">)</span>
9399 <span class="k">def</span> <span class="nf">test_greet</span><span class="p">(</span><span class="n">mock_stdout</span><span class="p">):</span>
9400 <span class="n">greet</span><span class="p">(</span><span class="s1">'John'</span><span class="p">)</span>
9401 <span class="n">mock_stdout</span><span class="o">.</span><span class="n">write</span><span class="o">.</span><span class="n">assert_has_calls</span><span class="p">([</span>
9402 <span class="n">call</span><span class="p">(</span><span class="s1">'Hello, John!'</span><span class="p">),</span>
9403 <span class="n">call</span><span class="p">(</span><span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">)</span>
9404 <span class="p">])</span>
9405 </pre></div>
9406
9407 <p>First of all, remember to install the <code>mock</code> module as it wasn’t available in the standard library in Python 2.</p>
9408 <p>Secondly, the <code>print</code> statement calls the underlying <code>.write()</code> method on the mocked object instead of calling the object itself. That’s why you’ll run assertions against <code>mock_stdout.write</code>. </p>
9409 <p>Finally, a single <code>print</code> statement doesn’t always correspond to a single call to <code>sys.stdout.write()</code>. In fact, you’ll see the newline character written separately.</p>
9410 <p>The last option you have is importing <code>print()</code> from <code>future</code> and patching it:</p>
9411 <div class="highlight python"><pre><span></span><span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">print_function</span>
9412 <span class="kn">from</span> <span class="nn">mock</span> <span class="kn">import</span> <span class="n">patch</span>
9413
9414 <span class="k">def</span> <span class="nf">greet</span><span class="p">(</span><span class="n">name</span><span class="p">):</span>
9415 <span class="k">print</span><span class="p">(</span><span class="s1">'Hello, </span><span class="si">%s</span><span class="s1">!'</span> <span class="o">%</span> <span class="n">name</span><span class="p">)</span>
9416
9417 <span class="nd">@patch</span><span class="p">(</span><span class="s1">'__builtin__.print'</span><span class="p">)</span>
9418 <span class="k">def</span> <span class="nf">test_greet</span><span class="p">(</span><span class="n">mock_print</span><span class="p">):</span>
9419 <span class="n">greet</span><span class="p">(</span><span class="s1">'John'</span><span class="p">)</span>
9420 <span class="n">mock_print</span><span class="o">.</span><span class="n">assert_called_with</span><span class="p">(</span><span class="s1">'Hello, John!'</span><span class="p">)</span>
9421 </pre></div>
9422
9423 <p>Again, it’s nearly identical to Python 3, but the <code>print()</code> function is defined in the <code>__builtin__</code> module rather than <code>builtins</code>.</p>
9424 </div></div>
9425
9426 </div>
9427 <h2 id="print-debugging">Print Debugging</h2>
9428 <p>In this section, you’ll take a look at the available tools for debugging in Python, starting from a humble <code>print()</code> function, through the <code>logging</code> module, to a fully fledged debugger. After reading it, you’ll be able to make an educated decision about which of them is the most suitable in a given situation.</p>
9429 <div class="alert alert-primary" role="alert">
9430 <p><strong>Note:</strong> Debugging is the process of looking for the root causes of <strong>bugs</strong> or defects in software after they’ve been discovered, as well as taking steps to fix them.</p>
9431 <p>The term <strong>bug</strong> has an <a href="https://en.wikipedia.org/wiki/Debugging#Origin_of_the_term">amusing story</a> about the origin of its name.</p>
9432 </div>
9433 <h3 id="tracing">Tracing</h3>
9434 <p>Also known as <strong>print debugging</strong> or <strong>caveman debugging</strong>, it’s the most basic form of debugging. While a little bit old-fashioned, it’s still powerful and has its uses.</p>
9435 <p>The idea is to follow the path of program execution until it stops abruptly, or gives incorrect results, to identify the exact instruction with a problem. You do that by inserting print statements with words that stand out in carefully chosen places.</p>
9436 <p>Take a look at this example, which manifests a rounding error:</p>
9437 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">def</span> <span class="nf">average</span><span class="p">(</span><span class="n">numbers</span><span class="p">):</span>
9438 <span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="s1">'debug1:'</span><span class="p">,</span> <span class="n">numbers</span><span class="p">)</span>
9439 <span class="gp">... </span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">numbers</span><span class="p">)</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span>
9440 <span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="s1">'debug2:'</span><span class="p">,</span> <span class="nb">sum</span><span class="p">(</span><span class="n">numbers</span><span class="p">))</span>
9441 <span class="gp">... </span> <span class="k">return</span> <span class="nb">sum</span><span class="p">(</span><span class="n">numbers</span><span class="p">)</span> <span class="o">/</span> <span class="nb">len</span><span class="p">(</span><span class="n">numbers</span><span class="p">)</span>
9442 <span class="gp">...</span>
9443 <span class="gp">>>> </span><span class="mf">0.1</span> <span class="o">==</span> <span class="n">average</span><span class="p">(</span><span class="mi">3</span><span class="o">*</span><span class="p">[</span><span class="mf">0.1</span><span class="p">])</span>
9444 <span class="go">debug1: [0.1, 0.1, 0.1]</span>
9445 <span class="go">debug2: 0.30000000000000004</span>
9446 <span class="go">False</span>
9447 </pre></div>
9448
9449 <p>As you can see, the function doesn’t return the expected value of <code>0.1</code>, but now you know it’s because the sum is a little off. Tracing the state of variables at different steps of the algorithm can give you a hint where the issue is.</p>
9450 <div class="card mb-3" id="collapse_cardab3ddc">
9451 <div class="card-header border-0"><p class="m-0"><button class="btn" data-toggle="collapse" data-target="#collapseab3ddc" aria-expanded="false" aria-controls="collapseab3ddc">Rounding Error</button> <button class="btn btn-link float-right" data-toggle="collapse" data-target="#collapseab3ddc" aria-expanded="false" aria-controls="collapseab3ddc">Show/Hide</button></p></div>
9452 <div id="collapseab3ddc" class="collapse" data-parent="#collapse_cardab3ddc"><div class="card-body" markdown="1">
9453
9454 <p>In this case, the problem lies in how <strong>floating point</strong> numbers are represented in computer memory. Remember that numbers are stored in binary form. Decimal value of <code>0.1</code> turns out to have an infinite binary representation, which gets rounded.</p>
9455 <p>For more information on rounding numbers in Python, you can check out <a href="https://realpython.com/python-rounding/">How to Round Numbers in Python</a>.</p>
9456 </div></div>
9457
9458 </div>
9459 <p>This method is simple and intuitive and will work in pretty much every programming language out there. Not to mention, it’s a great exercise in the learning process.</p>
9460 <p>On the other hand, once you master more advanced techniques, it’s hard to go back, because they allow you to find bugs much quicker. Tracing is a laborious manual process, which can let even more errors slip through. The build and deploy cycle takes time. Afterward, you need to remember to meticulously remove all the <code>print()</code> calls you made without accidentally touching the genuine ones.</p>
9461 <p>Besides, it requires you to make changes in the code, which isn’t always possible. Maybe you’re debugging an application running in a remote web server or want to diagnose a problem in a <strong>post-mortem</strong> fashion. Sometimes you simply don’t have access to the standard output.</p>
9462 <p>That’s precisely where <a href="https://realpython.com/courses/logging-python/">logging</a> shines.</p>
9463 <h3 id="logging">Logging</h3>
9464 <p>Let’s pretend for a minute that you’re running an e-commerce website. One day, an angry customer makes a phone call complaining about a failed transaction and saying he lost his money. He claims to have tried purchasing a few items, but in the end, there was some cryptic error that prevented him from finishing that order. Yet, when he checked his bank account, the money was gone.</p>
9465 <p>You apologize sincerely and make a refund, but also don’t want this to happen again in the future. How do you debug that? If only you had some trace of what happened, ideally in the form of a chronological list of events with their context.</p>
9466 <p>Whenever you find yourself doing print debugging, consider turning it into permanent log messages. This may help in situations like this, when you need to analyze a problem after it happened, in an environment that you don’t have access to.</p>
9467 <p>There are sophisticated tools for log aggregation and searching, but at the most basic level, you can think of logs as text files. Each line conveys detailed information about an event in your system. Usually, it won’t contain personally identifying information, though, in some cases, it may be mandated by law.</p>
9468 <p>Here’s a breakdown of a typical log record:</p>
9469 <div class="highlight text"><pre><span></span>[2019-06-14 15:18:34,517][DEBUG][root][MainThread] Customer(id=123) logged out
9470 </pre></div>
9471
9472 <p>As you can see, it has a structured form. Apart from a descriptive message, there are a few customizable fields, which provide the context of an event. Here, you have the exact date and time, the log level, the logger name, and the thread name.</p>
9473 <p>Log levels allow you to filter messages quickly to reduce noise. If you’re looking for an error, you don’t want to see all the warnings or debug messages, for example. It’s trivial to disable or enable messages at certain log levels through the configuration, without even touching the code.</p>
9474 <p>With logging, you can keep your debug messages separate from the standard output. All the log messages go to the standard error stream by default, which can conveniently show up in different colors. However, you can redirect log messages to separate files, even for individual modules!</p>
9475 <p>Quite commonly, misconfigured logging can lead to running out of space on the server’s disk. To prevent that, you may set up <strong>log rotation</strong>, which will keep the log files for a specified duration, such as one week, or once they hit a certain size. Nevertheless, it’s always a good practice to archive older logs. Some regulations enforce that customer data be kept for as long as five years!</p>
9476 <p>Compared to other programming languages, <a href="https://realpython.com/python-logging/">logging in Python</a> is simpler, because the <code>logging</code> module is bundled with the standard library. You just import and configure it in as little as two lines of code:</p>
9477 <div class="highlight python"><pre><span></span><span class="kn">import</span> <span class="nn">logging</span>
9478 <span class="n">logging</span><span class="o">.</span><span class="n">basicConfig</span><span class="p">(</span><span class="n">level</span><span class="o">=</span><span class="n">logging</span><span class="o">.</span><span class="n">DEBUG</span><span class="p">)</span>
9479 </pre></div>
9480
9481 <p>You can call functions defined at the module level, which are hooked to the <strong>root logger</strong>, but more the common practice is to obtain a dedicated logger for each of your source files:</p>
9482 <div class="highlight python"><pre><span></span><span class="n">logging</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s1">'hello'</span><span class="p">)</span> <span class="c1"># Module-level function</span>
9483
9484 <span class="n">logger</span> <span class="o">=</span> <span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">(</span><span class="vm">__name__</span><span class="p">)</span>
9485 <span class="n">logger</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s1">'hello'</span><span class="p">)</span> <span class="c1"># Logger's method</span>
9486 </pre></div>
9487
9488 <p>The advantage of using custom loggers is more fine-grain control. They’re usually named after the module they were defined in through the <code>__name__</code> variable.</p>
9489 <div class="alert alert-primary" role="alert">
9490 <p><strong>Note:</strong> There’s a somewhat related <code>warnings</code> module in Python, which can also log messages to the standard error stream. However, it has a narrower spectrum of applications, mostly in library code, whereas client applications should use the <code>logging</code> module.</p>
9491 <p>That said, you can make them work together by calling <code>logging.captureWarnings(True)</code>.</p>
9492 </div>
9493 <p>One last reason to switch from the <code>print()</code> function to logging is thread safety. In the upcoming section, you’ll see that the former doesn’t play well with multiple threads of execution.</p>
9494 <h3 id="debugging">Debugging</h3>
9495 <p>The truth is that neither tracing nor logging can be considered real debugging. To do actual debugging, you need a debugger tool, which allows you to do the following:</p>
9496 <ul>
9497 <li>Step through the code interactively.</li>
9498 <li>Set breakpoints, including conditional breakpoints.</li>
9499 <li>Introspect variables in memory.</li>
9500 <li>Evaluate custom expressions at runtime.</li>
9501 </ul>
9502 <p>A crude debugger that runs in the terminal, unsurprisingly named <strong><code>pdb</code></strong> for “The Python Debugger,” is distributed as part of the standard library. This makes it always available, so it may be your only choice for performing remote debugging. Perhaps that’s a good reason to get familiar with it.</p>
9503 <p>However, it doesn’t come with a graphical interface, so <a href="https://realpython.com/python-debugging-pdb/">using <code>pdb</code></a> may be a bit tricky. If you can’t edit the code, you have to run it as a module and pass your script’s location:</p>
9504 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python -m pdb my_script.py
9505 </pre></div>
9506
9507 <p>Otherwise, you can set up a breakpoint directly in the code, which will pause the execution of your script and drop you into the debugger. The old way of doing this required two steps:</p>
9508 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">pdb</span>
9509 <span class="gp">>>> </span><span class="n">pdb</span><span class="o">.</span><span class="n">set_trace</span><span class="p">()</span>
9510 <span class="go">--Return--</span>
9511 <span class="go">> <stdin>(1)<module>()->None</span>
9512 <span class="go">(Pdb)</span>
9513 </pre></div>
9514
9515 <p>This shows up an interactive prompt, which might look intimidating at first. However, you can still type native Python at this point to examine or modify the state of local variables. Apart from that, there’s really only a handful of debugger-specific commands that you want to use for stepping through the code.</p>
9516 <div class="alert alert-primary" role="alert">
9517 <p><strong>Note:</strong> It’s customary to put the two instructions for spinning up a debugger on a single line. This requires the use of a semicolon, which is rarely found in Python programs:</p>
9518 <div class="highlight python"><pre><span></span><span class="kn">import</span> <span class="nn">pdb</span><span class="p">;</span> <span class="n">pdb</span><span class="o">.</span><span class="n">set_trace</span><span class="p">()</span>
9519 </pre></div>
9520
9521 <p>While certainly not Pythonic, it stands out as a reminder to remove it after you’re done with debugging.</p>
9522 </div>
9523 <p>Since Python 3.7, you can also call the built-in <code>breakpoint()</code> function, which does the same thing, but in a more compact way and with some additional <a href="https://realpython.com/python37-new-features/#the-breakpoint-built-in">bells and whistles</a>:</p>
9524 <div class="highlight python"><pre><span></span><span class="k">def</span> <span class="nf">average</span><span class="p">(</span><span class="n">numbers</span><span class="p">):</span>
9525 <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">numbers</span><span class="p">)</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span>
9526 <span class="n">breakpoint</span><span class="p">()</span> <span class="c1"># Python 3.7+</span>
9527 <span class="k">return</span> <span class="nb">sum</span><span class="p">(</span><span class="n">numbers</span><span class="p">)</span> <span class="o">/</span> <span class="nb">len</span><span class="p">(</span><span class="n">numbers</span><span class="p">)</span>
9528 </pre></div>
9529
9530 <p>You’re probably going to use a visual debugger integrated with a code editor for the most part. <a href="https://www.jetbrains.com/pycharm/">PyCharm</a> has an excellent debugger, which boasts high performance, but you’ll find <a href="https://realpython.com/python-ides-code-editors-guide/">plenty of alternative IDEs</a> with debuggers, both paid and free of charge.</p>
9531 <p>Debugging isn’t the proverbial silver bullet. Sometimes logging or tracing will be a better solution. For example, defects that are hard to reproduce, such as <a href="https://en.wikipedia.org/wiki/Race_condition">race conditions</a>, often result from temporal coupling. When you stop at a breakpoint, that little pause in program execution may mask the problem. It’s kind of like the <a href="https://en.wikipedia.org/wiki/Uncertainty_principle">Heisenberg principle</a>: you can’t measure and observe a bug at the same time.</p>
9532 <p>These methods aren’t mutually exclusive. They complement each other.</p>
9533 <h2 id="thread-safe-printing">Thread-Safe Printing</h2>
9534 <p>I briefly touched upon the thread safety issue before, recommending <code>logging</code> over the <code>print()</code> function. If you’re still reading this, then you must be comfortable with <a href="https://realpython.com/intro-to-python-threading/">the concept of threads</a>.</p>
9535 <p>Thread safety means that a piece of code can be safely shared between multiple threads of execution. The simplest strategy for ensuring thread-safety is by sharing <strong>immutable</strong> objects only. If threads can’t modify an object’s state, then there’s no risk of breaking its consistency.</p>
9536 <p>Another method takes advantage of <strong>local memory</strong>, which makes each thread receive its own copy of the same object. That way, other threads can’t see the changes made to it in the current thread.</p>
9537 <p>But that doesn’t solve the problem, does it? You often want your threads to cooperate by being able to mutate a shared resource. The most common way of synchronizing concurrent access to such a resource is by <strong>locking</strong> it. This gives exclusive write access to one or sometimes a few threads at a time.</p>
9538 <p>However, locking is expensive and reduces concurrent throughput, so other means for controlling access have been invented, such as <strong>atomic variables</strong> or the <strong>compare-and-swap</strong> algorithm.</p>
9539 <p>Printing isn’t thread-safe in Python. The <code>print()</code> function holds a reference to the standard output, which is a shared global variable. In theory, because there’s no locking, a context switch could happen during a call to <code>sys.stdout.write()</code>, intertwining bits of text from multiple <code>print()</code> calls.</p>
9540 <div class="alert alert-primary" role="alert">
9541 <p><strong>Note:</strong> A context switch means that one thread halts its execution, either voluntarily or not, so that another one can take over. This might happen at any moment, even in the middle of a function call.</p>
9542 </div>
9543 <p>In practice, however, that doesn’t happen. No matter how hard you try, writing to the standard output seems to be atomic. The only problem that you may sometimes observe is with messed up line breaks:</p>
9544 <div class="highlight text"><pre><span></span>[Thread-3 A][Thread-2 A][Thread-1 A]
9545
9546 [Thread-3 B][Thread-1 B]
9547
9548
9549 [Thread-1 C][Thread-3 C]
9550
9551 [Thread-2 B]
9552 [Thread-2 C]
9553 </pre></div>
9554
9555 <p>To simulate this, you can increase the likelihood of a context switch by making the underlying <code>.write()</code> method go to sleep for a random amount of time. How? By mocking it, which you already know about from an earlier section:</p>
9556 <div class="highlight python"><pre><span></span><span class="kn">import</span> <span class="nn">sys</span>
9557
9558 <span class="kn">from</span> <span class="nn">time</span> <span class="k">import</span> <span class="n">sleep</span>
9559 <span class="kn">from</span> <span class="nn">random</span> <span class="k">import</span> <span class="n">random</span>
9560 <span class="kn">from</span> <span class="nn">threading</span> <span class="k">import</span> <span class="n">current_thread</span><span class="p">,</span> <span class="n">Thread</span>
9561 <span class="kn">from</span> <span class="nn">unittest.mock</span> <span class="k">import</span> <span class="n">patch</span>
9562
9563 <span class="n">write</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">write</span>
9564
9565 <span class="k">def</span> <span class="nf">slow_write</span><span class="p">(</span><span class="n">text</span><span class="p">):</span>
9566 <span class="n">sleep</span><span class="p">(</span><span class="n">random</span><span class="p">())</span>
9567 <span class="n">write</span><span class="p">(</span><span class="n">text</span><span class="p">)</span>
9568
9569 <span class="k">def</span> <span class="nf">task</span><span class="p">():</span>
9570 <span class="n">thread_name</span> <span class="o">=</span> <span class="n">current_thread</span><span class="p">()</span><span class="o">.</span><span class="n">name</span>
9571 <span class="k">for</span> <span class="n">letter</span> <span class="ow">in</span> <span class="s1">'ABC'</span><span class="p">:</span>
9572 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'[</span><span class="si">{thread_name}</span><span class="s1"> </span><span class="si">{letter}</span><span class="s1">]'</span><span class="p">)</span>
9573
9574 <span class="k">with</span> <span class="n">patch</span><span class="p">(</span><span class="s1">'sys.stdout'</span><span class="p">)</span> <span class="k">as</span> <span class="n">mock_stdout</span><span class="p">:</span>
9575 <span class="n">mock_stdout</span><span class="o">.</span><span class="n">write</span> <span class="o">=</span> <span class="n">slow_write</span>
9576 <span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">3</span><span class="p">):</span>
9577 <span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">task</span><span class="p">)</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
9578 </pre></div>
9579
9580 <p>First, you need to store the original <code>.write()</code> method in a variable, which you’ll delegate to later. Then you provide your fake implementation, which will take up to one second to execute. Each thread will make a few <code>print()</code> calls with its name and a letter: A, B, and C.</p>
9581 <p>If you read the mocking section before, then you may already have an idea of why printing misbehaves like that. Nonetheless, to make it crystal clear, you can capture values fed into your <code>slow_write()</code> function. You’ll notice that you get a slightly different sequence each time:</p>
9582 <div class="highlight python"><pre><span></span><span class="p">[</span>
9583 <span class="s1">'[Thread-3 A]'</span><span class="p">,</span>
9584 <span class="s1">'[Thread-2 A]'</span><span class="p">,</span>
9585 <span class="s1">'[Thread-1 A]'</span><span class="p">,</span>
9586 <span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">,</span>
9587 <span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">,</span>
9588 <span class="s1">'[Thread-3 B]'</span><span class="p">,</span>
9589 <span class="p">(</span><span class="o">...</span><span class="p">)</span>
9590 <span class="p">]</span>
9591 </pre></div>
9592
9593 <p>Even though <code>sys.stdout.write()</code> itself is an atomic operation, a single call to the <code>print()</code> function can yield more than one write. For example, line breaks are written separately from the rest of the text, and context switching takes place between those writes.</p>
9594 <div class="alert alert-primary" role="alert">
9595 <p><strong>Note:</strong> The atomic nature of the standard output in Python is a byproduct of the <a href="https://realpython.com/python-gil/">Global Interpreter Lock</a>, which applies locking around bytecode instructions. Be aware, however, that many interpreter flavors don’t have the GIL, where multi-threaded printing requires explicit locking.</p>
9596 </div>
9597 <p>You can make the newline character become an integral part of the message by handling it manually:</p>
9598 <div class="highlight python"><pre><span></span><span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'[</span><span class="si">{thread_name}</span><span class="s1"> </span><span class="si">{letter}</span><span class="s1">]</span><span class="se">\n</span><span class="s1">'</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s1">''</span><span class="p">)</span>
9599 </pre></div>
9600
9601 <p>This will fix the output:</p>
9602 <div class="highlight text"><pre><span></span>[Thread-2 A]
9603 [Thread-1 A]
9604 [Thread-3 A]
9605 [Thread-1 B]
9606 [Thread-3 B]
9607 [Thread-2 B]
9608 [Thread-1 C]
9609 [Thread-2 C]
9610 [Thread-3 C]
9611 </pre></div>
9612
9613 <p>Notice, however, that the <code>print()</code> function still keeps making a separate call for the empty suffix, which translates to useless <code>sys.stdout.write('')</code> instruction:</p>
9614 <div class="highlight python"><pre><span></span><span class="p">[</span>
9615 <span class="s1">'[Thread-2 A]</span><span class="se">\n</span><span class="s1">'</span><span class="p">,</span>
9616 <span class="s1">'[Thread-1 A]</span><span class="se">\n</span><span class="s1">'</span><span class="p">,</span>
9617 <span class="s1">'[Thread-3 A]</span><span class="se">\n</span><span class="s1">'</span><span class="p">,</span>
9618 <span class="s1">''</span><span class="p">,</span>
9619 <span class="s1">''</span><span class="p">,</span>
9620 <span class="s1">''</span><span class="p">,</span>
9621 <span class="s1">'[Thread-1 B]</span><span class="se">\n</span><span class="s1">'</span><span class="p">,</span>
9622 <span class="p">(</span><span class="o">...</span><span class="p">)</span>
9623 <span class="p">]</span>
9624 </pre></div>
9625
9626 <p>A truly thread-safe version of the <code>print()</code> function could look like this:</p>
9627 <div class="highlight python"><pre><span></span><span class="kn">import</span> <span class="nn">threading</span>
9628
9629 <span class="n">lock</span> <span class="o">=</span> <span class="n">threading</span><span class="o">.</span><span class="n">Lock</span><span class="p">()</span>
9630
9631 <span class="k">def</span> <span class="nf">thread_safe_print</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
9632 <span class="k">with</span> <span class="n">lock</span><span class="p">:</span>
9633 <span class="nb">print</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
9634 </pre></div>
9635
9636 <p>You can put that function in a module and import it elsewhere:</p>
9637 <div class="highlight python"><pre><span></span><span class="kn">from</span> <span class="nn">thread_safe_print</span> <span class="k">import</span> <span class="n">thread_safe_print</span>
9638
9639 <span class="k">def</span> <span class="nf">task</span><span class="p">():</span>
9640 <span class="n">thread_name</span> <span class="o">=</span> <span class="n">current_thread</span><span class="p">()</span><span class="o">.</span><span class="n">name</span>
9641 <span class="k">for</span> <span class="n">letter</span> <span class="ow">in</span> <span class="s1">'ABC'</span><span class="p">:</span>
9642 <span class="n">thread_safe_print</span><span class="p">(</span><span class="n">f</span><span class="s1">'[</span><span class="si">{thread_name}</span><span class="s1"> </span><span class="si">{letter}</span><span class="s1">]'</span><span class="p">)</span>
9643 </pre></div>
9644
9645 <p>Now, despite making two writes per each <code>print()</code> request, only one thread is allowed to interact with the stream, while the rest must wait:</p>
9646 <div class="highlight python"><pre><span></span><span class="p">[</span>
9647 <span class="c1"># Lock acquired by Thread-3 </span>
9648 <span class="s1">'[Thread-3 A]'</span><span class="p">,</span>
9649 <span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">,</span>
9650 <span class="c1"># Lock released by Thread-3</span>
9651 <span class="c1"># Lock acquired by Thread-1</span>
9652 <span class="s1">'[Thread-1 B]'</span><span class="p">,</span>
9653 <span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">,</span>
9654 <span class="c1"># Lock released by Thread-1</span>
9655 <span class="p">(</span><span class="o">...</span><span class="p">)</span>
9656 <span class="p">]</span>
9657 </pre></div>
9658
9659 <p>I added comments to indicate how the lock is limiting access to the shared resource.</p>
9660 <div class="alert alert-primary" role="alert">
9661 <p><strong>Note:</strong> Even in single-threaded code, you might get caught up in a similar situation. Specifically, when you’re printing to the standard output and the standard error streams at the same time. Unless you redirect one or both of them to separate files, they’ll both share a single terminal window.</p>
9662 </div>
9663 <p>Conversely, the <code>logging</code> module is thread-safe by design, which is reflected by its ability to display thread names in the formatted message:</p>
9664 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">logging</span>
9665 <span class="gp">>>> </span><span class="n">logging</span><span class="o">.</span><span class="n">basicConfig</span><span class="p">(</span><span class="nb">format</span><span class="o">=</span><span class="s1">'</span><span class="si">%(threadName)s</span><span class="s1"> </span><span class="si">%(message)s</span><span class="s1">'</span><span class="p">)</span>
9666 <span class="gp">>>> </span><span class="n">logging</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s1">'hello'</span><span class="p">)</span>
9667 <span class="go">MainThread hello</span>
9668 </pre></div>
9669
9670 <p>It’s another reason why you might not want to use the <code>print()</code> function all the time.</p>
9671 <h2 id="python-print-counterparts">Python Print Counterparts</h2>
9672 <p>By now, you know a lot of what there is to know about <code>print()</code>! The subject, however, wouldn’t be complete without talking about its counterparts a little bit. While <code>print()</code> is about the output, there are functions and libraries for the input.</p>
9673 <h3 id="built-in">Built-In</h3>
9674 <p>Python comes with a built-in function for accepting input from the user, predictably called <code>input()</code>. It accepts data from the standard input stream, which is usually the keyboard:</p>
9675 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">name</span> <span class="o">=</span> <span class="nb">input</span><span class="p">(</span><span class="s1">'Enter your name: '</span><span class="p">)</span>
9676 <span class="go">Enter your name: jdoe</span>
9677 <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
9678 <span class="go">jdoe</span>
9679 </pre></div>
9680
9681 <p>The function always returns a string, so you might need to parse it accordingly:</p>
9682 <div class="highlight python"><pre><span></span><span class="k">try</span><span class="p">:</span>
9683 <span class="n">age</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="nb">input</span><span class="p">(</span><span class="s1">'How old are you? '</span><span class="p">))</span>
9684 <span class="k">except</span> <span class="ne">ValueError</span><span class="p">:</span>
9685 <span class="k">pass</span>
9686 </pre></div>
9687
9688 <p>The prompt parameter is completely optional, so nothing will show if you skip it, but the function will still work:</p>
9689 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">x</span> <span class="o">=</span> <span class="nb">input</span><span class="p">()</span>
9690 <span class="go">hello world</span>
9691 <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
9692 <span class="go">hello world</span>
9693 </pre></div>
9694
9695 <p>Nevertheless, throwing in a descriptive call to action makes the user experience so much better.</p>
9696 <div class="alert alert-primary" role="alert">
9697 <p><strong>Note:</strong> To read from the standard input in Python 2, you have to call <code>raw_input()</code> instead, which is yet another built-in. Unfortunately, there’s also a misleadingly named <code>input()</code> function, which does a slightly different thing.</p>
9698 <p>In fact, it also takes the input from the standard stream, but then it tries to evaluate it as if it was Python code. Because that’s a potential <strong>security vulnerability</strong>, this function was completely removed from Python 3, while <code>raw_input()</code> got renamed to <code>input()</code>.</p>
9699 <p>Here’s a quick comparison of the available functions and what they do:</p>
9700 <div class="table-responsive">
9701 <table class="table table-hover">
9702 <thead>
9703 <tr>
9704 <th>Python 2</th>
9705 <th>Python 3</th>
9706 </tr>
9707 </thead>
9708 <tbody>
9709 <tr>
9710 <td><code>raw_input()</code></td>
9711 <td><code>input()</code></td>
9712 </tr>
9713 <tr>
9714 <td><code>input()</code></td>
9715 <td><code>eval(input())</code></td>
9716 </tr>
9717 </tbody>
9718 </table>
9719 </div>
9720 <p>As you can tell, it’s still possible to simulate the old behavior in Python 3.</p>
9721 </div>
9722 <p>Asking the user for a password with <code>input()</code> is a bad idea because it’ll show up in plaintext as they’re typing it. In this case, you should be using the <code>getpass()</code> function instead, which masks typed characters. This function is defined in a module under the same name, which is also available in the standard library:</p>
9723 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">getpass</span> <span class="k">import</span> <span class="n">getpass</span>
9724 <span class="gp">>>> </span><span class="n">password</span> <span class="o">=</span> <span class="n">getpass</span><span class="p">()</span>
9725 <span class="go">Password: </span>
9726 <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">password</span><span class="p">)</span>
9727 <span class="go">s3cret</span>
9728 </pre></div>
9729
9730 <p>The <code>getpass</code> module has another function for getting the user’s name from an environment variable:</p>
9731 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">getpass</span> <span class="k">import</span> <span class="n">getuser</span>
9732 <span class="gp">>>> </span><span class="n">getuser</span><span class="p">()</span>
9733 <span class="go">'jdoe'</span>
9734 </pre></div>
9735
9736 <p>Python’s built-in functions for handling the standard input are quite limited. At the same time, there are plenty of third-party packages, which offer much more sophisticated tools.</p>
9737 <h3 id="third-party">Third-Party</h3>
9738 <p>There are external Python packages out there that allow for building complex graphical interfaces specifically to collect data from the user. Some of their features include:</p>
9739 <ul>
9740 <li>Advanced formatting and styling</li>
9741 <li>Automated parsing, validation, and sanitization of user data</li>
9742 <li>A declarative style of defining layouts</li>
9743 <li>Interactive autocompletion</li>
9744 <li>Mouse support</li>
9745 <li>Predefined widgets such as checklists or menus</li>
9746 <li>Searchable history of typed commands</li>
9747 <li>Syntax highlighting</li>
9748 </ul>
9749 <p>Demonstrating such tools is outside of the scope of this article, but you may want to try them out. I personally got to know about some of those through the <a href="https://pythonbytes.fm/">Python Bytes Podcast</a>. Here they are:</p>
9750 <ul>
9751 <li><a href="https://github.com/Mckinsey666/bullet"><code>bullet</code></a></li>
9752 <li><a href="https://pypi.org/project/cooked-input/"><code>cooked-input</code></a></li>
9753 <li><a href="https://pypi.org/project/prompt_toolkit/"><code>prompt_toolkit</code></a></li>
9754 <li><a href="https://github.com/kylebebak/questionnaire"><code>questionnaire</code></a></li>
9755 </ul>
9756 <p>Nonetheless, it’s worth mentioning a command line tool called <code>rlwrap</code> that adds powerful line editing capabilities to your Python scripts for free. You don’t have to do anything for it to work!</p>
9757 <p>Let’s assume you wrote a command-line interface that understands three instructions, including one for adding numbers:</p>
9758 <div class="highlight python"><pre><span></span><span class="nb">print</span><span class="p">(</span><span class="s1">'Type "help", "exit", "add a [b [c ...]]"'</span><span class="p">)</span>
9759 <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
9760 <span class="n">command</span><span class="p">,</span> <span class="o">*</span><span class="n">arguments</span> <span class="o">=</span> <span class="nb">input</span><span class="p">(</span><span class="s1">'~ '</span><span class="p">)</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">' '</span><span class="p">)</span>
9761 <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">command</span><span class="p">)</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span>
9762 <span class="k">if</span> <span class="n">command</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">==</span> <span class="s1">'exit'</span><span class="p">:</span>
9763 <span class="k">break</span>
9764 <span class="k">elif</span> <span class="n">command</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">==</span> <span class="s1">'help'</span><span class="p">:</span>
9765 <span class="nb">print</span><span class="p">(</span><span class="s1">'This is help.'</span><span class="p">)</span>
9766 <span class="k">elif</span> <span class="n">command</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">==</span> <span class="s1">'add'</span><span class="p">:</span>
9767 <span class="nb">print</span><span class="p">(</span><span class="nb">sum</span><span class="p">(</span><span class="nb">map</span><span class="p">(</span><span class="nb">int</span><span class="p">,</span> <span class="n">arguments</span><span class="p">)))</span>
9768 <span class="k">else</span><span class="p">:</span>
9769 <span class="nb">print</span><span class="p">(</span><span class="s1">'Unknown command'</span><span class="p">)</span>
9770 </pre></div>
9771
9772 <p>At first glance, it seems like a typical prompt when you run it:</p>
9773 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python calculator.py
9774 <span class="go">Type "help", "exit", "add a [b [c ...]]"</span>
9775 <span class="go">~ add 1 2 3 4</span>
9776 <span class="go">10</span>
9777 <span class="go">~ aad 2 3</span>
9778 <span class="go">Unknown command</span>
9779 <span class="go">~ exit</span>
9780 <span class="gp">$</span>
9781 </pre></div>
9782
9783 <p>But as soon as you make a mistake and want to fix it, you’ll see that none of the function keys work as expected. Hitting the <span class="keys"><kbd class="key-arrow-left">Left</kbd></span> arrow, for example, results in this instead of moving the cursor back:</p>
9784 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python calculator.py
9785 <span class="go">Type "help", "exit", "add a [b [c ...]]"</span>
9786 <span class="go">~ aad^[[D</span>
9787 </pre></div>
9788
9789 <p>Now, you can wrap the same script with the <code>rlwrap</code> command. Not only will you get the arrow keys working, but you’ll also be able to search through the persistent history of your custom commands, use autocompletion, and edit the line with shortcuts:</p>
9790 <div class="highlight sh"><pre><span></span><span class="gp">$</span> rlwrap python calculator.py
9791 <span class="go">Type "help", "exit", "add a [b [c ...]]"</span>
9792 <span class="go">(reverse-i-search)`a': add 1 2 3 4</span>
9793 </pre></div>
9794
9795 <p>Isn’t that great?</p>
9796 <h2 id="conclusion">Conclusion</h2>
9797 <p>You’re now armed with a body of knowledge about the <code>print()</code> function in Python, as well as many surrounding topics. You have a deep understanding of what it is and how it works, involving all of its key elements. Numerous examples gave you insight into its evolution from Python 2.</p>
9798 <p>Apart from that, you learned how to:</p>
9799 <ul>
9800 <li>Avoid common mistakes with <code>print()</code> in Python</li>
9801 <li>Deal with newlines, character encodings and buffering</li>
9802 <li>Write text to files</li>
9803 <li>Mock the <code>print()</code> function in unit tests</li>
9804 <li>Build advanced user interfaces in the terminal</li>
9805 </ul>
9806 <p>Now that you know all this, you can make interactive programs that communicate with users or produce data in popular file formats. You’re able to quickly diagnose problems in your code and protect yourself from them. Last but not least, you know how to implement the classic snake game.</p>
9807 <p>If you’re still thirsty for more information, have questions, or simply would like to share your thoughts, then feel free to reach out in the comments section below.</p>
9808 <hr />
9809 <p><em>[ Improve Your Python With π Python Tricks π β Get a short & sweet Python Trick delivered to your inbox every couple of days. <a href="https://realpython.com/python-tricks/?utm_source=realpython&utm_medium=rss&utm_campaign=footer">>> Click here to learn more and see examples</a> ]</em></p>#
9810 datePublished: #Mon Aug 12 14:00:00 2019#
9811 dateUpdated: #Mon Aug 12 14:00:00 2019#
9812 # Person begin ####################
9813 name: #Real Python#
9814 # Person end ######################
9815 # Item end ########################
9816 # Item begin ######################
9817 id: #https://realpython.com/inheritance-composition-python/#
9818 title: #Inheritance and Composition: A Python OOP Guide#
9819 link: #https://realpython.com/inheritance-composition-python/#
9820 description: #In this step-by-step tutorial, you'll learn about inheritance and composition in Python. You'll improve your object-oriented programming (OOP) skills by understanding how to use inheritance and composition and how to leverage them in their design.#
9821 content: #<p>In this article, you’ll explore <strong>inheritance</strong> and <strong>composition</strong> in Python. <a href="https://en.wikipedia.org/wiki/Inheritance_(object-oriented_programming)">Inheritance</a> and <a href="https://en.wikipedia.org/wiki/Object_composition">composition</a> are two important concepts in object oriented programming that model the relationship between two classes. They are the building blocks of <a href="https://realpython.com/python3-object-oriented-programming/">object oriented design</a>, and they help programmers to write reusable code.</p>
9822 <p><strong>By the end of this article, you’ll know how to</strong>:</p>
9823 <ul>
9824 <li>Use inheritance in Python</li>
9825 <li>Model class hierarchies using inheritance</li>
9826 <li>Use multiple inheritance in Python and understand its drawbacks</li>
9827 <li>Use composition to create complex objects</li>
9828 <li>Reuse existing code by applying composition</li>
9829 <li>Change application behavior at run-time through composition</li>
9830 </ul>
9831 <div class="alert alert-warning" role="alert"><p><strong>Free Bonus:</strong> <a href="#" class="alert-link" data-toggle="modal" data-target="#modal-python-oop" data-focus="false">Click here to get access to a free Python OOP Cheat Sheet</a> that points you to the best tutorials, videos, and books to learn more about Object-Oriented Programming with Python.</p></div>
9832
9833 <h2 id="what-are-inheritance-and-composition">What Are Inheritance and Composition?</h2>
9834 <p><strong>Inheritance</strong> and <strong>composition</strong> are two major concepts in object oriented programming that model the relationship between two classes. They drive the design of an application and determine how the application should evolve as new features are added or requirements change.</p>
9835 <p>Both of them enable code reuse, but they do it in different ways.</p>
9836 <h3 id="whats-inheritance">What’s Inheritance?</h3>
9837 <p><strong>Inheritance</strong> models what is called an <strong>is a</strong> relationship. This means that when you have a <code>Derived</code> class that inherits from a <code>Base</code> class, you created a relationship where <code>Derived</code> <strong>is a</strong> specialized version of <code>Base</code>.</p>
9838 <p>Inheritance is represented using the <a href="https://www.uml.org/">Unified Modeling Language</a> or UML in the following way:</p>
9839 <p><a href="https://files.realpython.com/media/ic-basic-inheritance.f8dc9ffee4d7.jpg" target="_blank"><img class="img-fluid mx-auto d-block w-33" src="https://files.realpython.com/media/ic-basic-inheritance.f8dc9ffee4d7.jpg" width="242" height="343" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/ic-basic-inheritance.f8dc9ffee4d7.jpg&w=60&sig=5b8ced824343b1ab983944639b080f1fb53c0ba4 60w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/ic-basic-inheritance.f8dc9ffee4d7.jpg&w=121&sig=e32f6166c127193b0856fcb61f9d2677ff7aa293 121w, https://files.realpython.com/media/ic-basic-inheritance.f8dc9ffee4d7.jpg 242w" sizes="75vw" alt="Basic inheritance between Base and Derived classes"/></a></p>
9840 <p>Classes are represented as boxes with the class name on top. The inheritance relationship is represented by an arrow from the derived class pointing to the base class. The word <strong>extends</strong> is usually added to the arrow.</p>
9841 <div class="alert alert-primary" role="alert">
9842 <p><strong>Note:</strong> In an inheritance relationship:</p>
9843 <ul>
9844 <li>Classes that inherit from another are called derived classes, subclasses, or subtypes. </li>
9845 <li>Classes from which other classes are derived are called base classes or super classes. </li>
9846 <li>A derived class is said to derive, inherit, or extend a base class.</li>
9847 </ul>
9848 </div>
9849 <p>Let’s say you have a base class <code>Animal</code> and you derive from it to create a <code>Horse</code> class. The inheritance relationship states that a <code>Horse</code> <strong>is an</strong> <code>Animal</code>. This means that <code>Horse</code> inherits the interface and implementation of <code>Animal</code>, and <code>Horse</code> objects can be used to replace <code>Animal</code> objects in the application.</p>
9850 <p>This is known as the <a href="https://en.wikipedia.org/wiki/Liskov_substitution_principle">Liskov substitution principle</a>. The principle states that “in a computer program, if <code>S</code> is a subtype of <code>T</code>, then objects of type <code>T</code> may be replaced with objects of type <code>S</code> without altering any of the desired properties of the program”.</p>
9851 <p>You’ll see in this article why you should always follow the Liskov substitution principle when creating your class hierarchies, and the problems you’ll run into if you don’t.</p>
9852 <h3 id="whats-composition">What’s Composition?</h3>
9853 <p><strong>Composition</strong> is a concept that models a <strong>has a</strong> relationship. It enables creating complex types by combining objects of other types. This means that a class <code>Composite</code> can contain an object of another class <code>Component</code>. This relationship means that a <code>Composite</code> <strong>has a</strong> <code>Component</code>.</p>
9854 <p>UML represents composition as follows:</p>
9855 <p><a href="https://files.realpython.com/media/ic-basic-composition.8a15876f7db2.jpg" target="_blank"><img class="img-fluid mx-auto d-block w-33" src="https://files.realpython.com/media/ic-basic-composition.8a15876f7db2.jpg" width="249" height="348" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/ic-basic-composition.8a15876f7db2.jpg&w=62&sig=fe9cee54da335ac7d82f147d947f595d9063864f 62w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/ic-basic-composition.8a15876f7db2.jpg&w=124&sig=28fd4da7064494aa88f126b3b073aaab8a500b61 124w, https://files.realpython.com/media/ic-basic-composition.8a15876f7db2.jpg 249w" sizes="75vw" alt="Basic composition between Composite and Component classes"/></a></p>
9856 <p>Composition is represented through a line with a diamond at the composite class pointing to the component class. The composite side can express the cardinality of the relationship. The cardinality indicates the number or valid range of <code>Component</code> instances the <code>Composite</code> class will contain.</p>
9857 <p>In the diagram above, the <code>1</code> represents that the <code>Composite</code> class contains one object of type <code>Component</code>. Cardinality can be expressed in the following ways:</p>
9858 <ul>
9859 <li><strong>A number</strong> indicates the number of <code>Component</code> instances that are contained in the <code>Composite</code>.</li>
9860 <li><strong>The * symbol</strong> indicates that the <code>Composite</code> class can contain a variable number of <code>Component</code> instances.</li>
9861 <li><strong>A range 1..4</strong> indicates that the <code>Composite</code> class can contain a range of <code>Component</code> instances. The range is indicated with the minimum and maximum number of instances, or minimum and many instances like in <strong>1..*</strong>.</li>
9862 </ul>
9863 <div class="alert alert-primary" role="alert">
9864 <p><strong>Note:</strong> Classes that contain objects of other classes are usually referred to as composites, where classes that are used to create more complex types are referred to as components.</p>
9865 </div>
9866 <p>For example, your <code>Horse</code> class can be composed by another object of type <code>Tail</code>. Composition allows you to express that relationship by saying a <code>Horse</code> <strong>has a</strong> <code>Tail</code>.</p>
9867 <p>Composition enables you to reuse code by adding objects to other objects, as opposed to inheriting the interface and implementation of other classes. Both <code>Horse</code> and <code>Dog</code> classes can leverage the functionality of <code>Tail</code> through composition without deriving one class from the other.</p>
9868 <h2 id="an-overview-of-inheritance-in-python">An Overview of Inheritance in Python</h2>
9869 <p>Everything in Python is an object. Modules are objects, class definitions and functions are objects, and of course, objects created from classes are objects too.</p>
9870 <p>Inheritance is a required feature of every object oriented programming language. This means that Python supports inheritance, and as you’ll see later, it’s one of the few languages that supports multiple inheritance.</p>
9871 <p>When you write Python code using classes, you are using inheritance even if you don’t know you’re using it. Let’s take a look at what that means.</p>
9872 <h3 id="the-object-super-class">The Object Super Class</h3>
9873 <p>The easiest way to see inheritance in Python is to jump into the <a href="https://realpython.com/interacting-with-python/#using-the-python-interpreter-interactively">Python interactive shell</a> and write a little bit of code. You’ll start by writing the simplest class possible:</p>
9874 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">class</span> <span class="nc">MyClass</span><span class="p">:</span>
9875 <span class="gp">... </span> <span class="k">pass</span>
9876 <span class="gp">...</span>
9877 </pre></div>
9878
9879 <p>You declared a class <code>MyClass</code> that doesn’t do much, but it will illustrate the most basic inheritance concepts. Now that you have the class declared, you can use the <code>dir()</code> function to list its members:</p>
9880 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">c</span> <span class="o">=</span> <span class="n">MyClass</span><span class="p">()</span>
9881 <span class="gp">>>> </span><span class="nb">dir</span><span class="p">(</span><span class="n">c</span><span class="p">)</span>
9882 <span class="go">['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__',</span>
9883 <span class="go">'__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__',</span>
9884 <span class="go">'__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__',</span>
9885 <span class="go">'__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__',</span>
9886 <span class="go">'__str__', '__subclasshook__', '__weakref__']</span>
9887 </pre></div>
9888
9889 <p><a href="https://docs.python.org/3/library/functions.html#dir"><code>dir()</code></a> returns a list of all the members in the specified object. You have not declared any members in <code>MyClass</code>, so where is the list coming from? You can find out using the interactive interpreter:</p>
9890 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="n">o</span> <span class="o">=</span> <span class="nb">object</span><span class="p">()</span>
9891 <span class="gp">>>> </span><span class="nb">dir</span><span class="p">(</span><span class="n">o</span><span class="p">)</span>
9892 <span class="go">['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__',</span>
9893 <span class="go">'__ge__', '__getattribute__', '__gt__', '__hash__', '__init__',</span>
9894 <span class="go">'__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__',</span>
9895 <span class="go">'__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__',</span>
9896 <span class="go">'__subclasshook__']</span>
9897 </pre></div>
9898
9899 <p>As you can see, the two lists are nearly identical. There are some additional members in <code>MyClass</code> like <code>__dict__</code> and <code>__weakref__</code>, but every single member of the <code>object</code> class is also present in <code>MyClass</code>.</p>
9900 <p>This is because every class you create in Python implicitly derives from <code>object</code>. You could be more explicit and write <code>class MyClass(object):</code>, but it’s redundant and unnecessary.</p>
9901 <div class="alert alert-primary" role="alert">
9902 <p><strong>Note:</strong> In Python 2, you have to explicitly derive from <code>object</code> for reasons beyond the scope of this article, but you can read about it in the <a href="https://docs.python.org/2/reference/datamodel.html#new-style-and-classic-classes">New-style and classic classes</a> section of the Python 2 documentation.</p>
9903 </div>
9904 <h3 id="exceptions-are-an-exception">Exceptions Are an Exception</h3>
9905 <p>Every class that you create in Python will implicitly derive from <code>object</code>. The exception to this rule are classes used to indicate errors by raising an <a href="https://realpython.com/courses/python-exceptions-101/">exception</a>.</p>
9906 <p>You can see the problem using the Python interactive interpreter:</p>
9907 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">class</span> <span class="nc">MyError</span><span class="p">:</span>
9908 <span class="gp">... </span> <span class="k">pass</span>
9909 <span class="gp">...</span>
9910 <span class="gp">>>> </span><span class="k">raise</span> <span class="n">MyError</span><span class="p">()</span>
9911
9912 <span class="gt">Traceback (most recent call last):</span>
9913 File <span class="nb">"<stdin>"</span>, line <span class="m">1</span>, in <span class="n"><module></span>
9914 <span class="gr">TypeError</span>: <span class="n">exceptions must derive from BaseException</span>
9915 </pre></div>
9916
9917 <p>You created a new class to indicate a type of error. Then you tried to use it to raise an exception. An exception is raised but the output states that the exception is of type <code>TypeError</code> not <code>MyError</code> and that all <code>exceptions must derive from BaseException</code>.</p>
9918 <p><code>BaseException</code> is a base class provided for all error types. To create a new error type, you must derive your class from <code>BaseException</code> or one of its derived classes. The convention in Python is to derive your custom error types from <code>Exception</code>, which in turn derives from <code>BaseException</code>.</p>
9919 <p>The correct way to define your error type is the following:</p>
9920 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="k">class</span> <span class="nc">MyError</span><span class="p">(</span><span class="ne">Exception</span><span class="p">):</span>
9921 <span class="gp">... </span> <span class="k">pass</span>
9922 <span class="gp">...</span>
9923 <span class="gp">>>> </span><span class="k">raise</span> <span class="n">MyError</span><span class="p">()</span>
9924
9925 <span class="gt">Traceback (most recent call last):</span>
9926 File <span class="nb">"<stdin>"</span>, line <span class="m">1</span>, in <span class="n"><module></span>
9927 <span class="err">__main__.MyError</span>
9928 </pre></div>
9929
9930 <p>As you can see, when you raise <code>MyError</code>, the output correctly states the type of error raised.</p>
9931 <h3 id="creating-class-hierarchies">Creating Class Hierarchies</h3>
9932 <p>Inheritance is the mechanism you’ll use to create hierarchies of related classes. These related classes will share a common interface that will be defined in the base classes. Derived classes can specialize the interface by providing a particular implementation where applies.</p>
9933 <p>In this section, you’ll start modeling an HR system. The example will demonstrate the use of inheritance and how derived classes can provide a concrete implementation of the base class interface.</p>
9934 <p>The HR system needs to process payroll for the company’s employees, but there are different types of employees depending on how their payroll is calculated.</p>
9935 <p>You start by implementing a <code>PayrollSystem</code> class that processes payroll:</p>
9936 <div class="highlight python"><pre><span></span><span class="c1"># In hr.py</span>
9937
9938 <span class="k">class</span> <span class="nc">PayrollSystem</span><span class="p">:</span>
9939 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">employees</span><span class="p">):</span>
9940 <span class="nb">print</span><span class="p">(</span><span class="s1">'Calculating Payroll'</span><span class="p">)</span>
9941 <span class="nb">print</span><span class="p">(</span><span class="s1">'==================='</span><span class="p">)</span>
9942 <span class="k">for</span> <span class="n">employee</span> <span class="ow">in</span> <span class="n">employees</span><span class="p">:</span>
9943 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Payroll for: </span><span class="si">{employee.id}</span><span class="s1"> - </span><span class="si">{employee.name}</span><span class="s1">'</span><span class="p">)</span>
9944 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'- Check amount: {employee.calculate_payroll()}'</span><span class="p">)</span>
9945 <span class="nb">print</span><span class="p">(</span><span class="s1">''</span><span class="p">)</span>
9946 </pre></div>
9947
9948 <p>The <code>PayrollSystem</code> implements a <code>.calculate_payroll()</code> method that takes a collection of employees and prints their <code>id</code>, <code>name</code>, and check amount using the <code>.calculate_payroll()</code> method exposed on each employee object.</p>
9949 <p>Now, you implement a base class <code>Employee</code> that handles the common interface for every employee type:</p>
9950 <div class="highlight python"><pre><span></span><span class="c1"># In hr.py</span>
9951
9952 <span class="k">class</span> <span class="nc">Employee</span><span class="p">:</span>
9953 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
9954 <span class="bp">self</span><span class="o">.</span><span class="n">id</span> <span class="o">=</span> <span class="nb">id</span>
9955 <span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">name</span>
9956 </pre></div>
9957
9958 <p><code>Employee</code> is the base class for all employee types. It is constructed with an <code>id</code> and a <code>name</code>. What you are saying is that every <code>Employee</code> must have an <code>id</code> assigned as well as a name.</p>
9959 <p>The HR system requires that every <code>Employee</code> processed must provide a <code>.calculate_payroll()</code> interface that returns the weekly salary for the employee. The implementation of that interface differs depending on the type of <code>Employee</code>.</p>
9960 <p>For example, administrative workers have a fixed salary, so every week they get paid the same amount:</p>
9961 <div class="highlight python"><pre><span></span><span class="c1"># In hr.py</span>
9962
9963 <span class="k">class</span> <span class="nc">SalaryEmployee</span><span class="p">(</span><span class="n">Employee</span><span class="p">):</span>
9964 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">weekly_salary</span><span class="p">):</span>
9965 <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span>
9966 <span class="bp">self</span><span class="o">.</span><span class="n">weekly_salary</span> <span class="o">=</span> <span class="n">weekly_salary</span>
9967
9968 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
9969 <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">weekly_salary</span>
9970 </pre></div>
9971
9972 <p>You create a derived class <code>SalaryEmployee</code> that inherits <code>Employee</code>. The class is initialized with the <code>id</code> and <code>name</code> required by the base class, and you use <code>super()</code> to initialize the members of the base class. You can read all about <code>super()</code> in <a href="https://realpython.com/python-super/">Supercharge Your Classes With Python super()</a>.</p>
9973 <p><code>SalaryEmployee</code> also requires a <code>weekly_salary</code> initialization parameter that represents the amount the employee makes per week.</p>
9974 <p>The class provides the required <code>.calculate_payroll()</code> method used by the HR system. The implementation just returns the amount stored in <code>weekly_salary</code>.</p>
9975 <p>The company also employs manufacturing workers that are paid by the hour, so you add an <code>HourlyEmployee</code> to the HR system:</p>
9976 <div class="highlight python"><pre><span></span><span class="c1"># In hr.py</span>
9977
9978 <span class="k">class</span> <span class="nc">HourlyEmployee</span><span class="p">(</span><span class="n">Employee</span><span class="p">):</span>
9979 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">hours_worked</span><span class="p">,</span> <span class="n">hour_rate</span><span class="p">):</span>
9980 <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span>
9981 <span class="bp">self</span><span class="o">.</span><span class="n">hours_worked</span> <span class="o">=</span> <span class="n">hours_worked</span>
9982 <span class="bp">self</span><span class="o">.</span><span class="n">hour_rate</span> <span class="o">=</span> <span class="n">hour_rate</span>
9983
9984 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
9985 <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">hours_worked</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">hour_rate</span>
9986 </pre></div>
9987
9988 <p>The <code>HourlyEmployee</code> class is initialized with <code>id</code> and <code>name</code>, like the base class, plus the <code>hours_worked</code> and the <code>hour_rate</code> required to calculate the payroll. The <code>.calculate_payroll()</code> method is implemented by returning the hours worked times the hour rate.</p>
9989 <p>Finally, the company employs sales associates that are paid through a fixed salary plus a commission based on their sales, so you create a <code>CommissionEmployee</code> class:</p>
9990 <div class="highlight python"><pre><span></span><span class="c1"># In hr.py</span>
9991
9992 <span class="k">class</span> <span class="nc">CommissionEmployee</span><span class="p">(</span><span class="n">SalaryEmployee</span><span class="p">):</span>
9993 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">weekly_salary</span><span class="p">,</span> <span class="n">commission</span><span class="p">):</span>
9994 <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">weekly_salary</span><span class="p">)</span>
9995 <span class="bp">self</span><span class="o">.</span><span class="n">commission</span> <span class="o">=</span> <span class="n">commission</span>
9996
9997 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
9998 <span class="n">fixed</span> <span class="o">=</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">calculate_payroll</span><span class="p">()</span>
9999 <span class="k">return</span> <span class="n">fixed</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">commission</span>
10000 </pre></div>
10001
10002 <p>You derive <code>CommissionEmployee</code> from <code>SalaryEmployee</code> because both classes have a <code>weekly_salary</code> to consider. At the same time, <code>CommissionEmployee</code> is initialized with a <code>commission</code> value that is based on the sales for the employee.</p>
10003 <p><code>.calculate_payroll()</code> leverages the implementation of the base class to retrieve the <code>fixed</code> salary and adds the commission value.</p>
10004 <p>Since <code>CommissionEmployee</code> derives from <code>SalaryEmployee</code>, you have access to the <code>weekly_salary</code> property directly, and you could’ve implemented <code>.calculate_payroll()</code> using the value of that property.</p>
10005 <p>The problem with accessing the property directly is that if the implementation of <code>SalaryEmployee.calculate_payroll()</code> changes, then you’ll have to also change the implementation of <code>CommissionEmployee.calculate_payroll()</code>. It’s better to rely on the already implemented method in the base class and extend the functionality as needed.</p>
10006 <p>You created your first class hierarchy for the system. The UML diagram of the classes looks like this:</p>
10007 <p><a href="https://files.realpython.com/media/ic-initial-employee-inheritance.b5f1e65cb8d1.jpg" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/ic-initial-employee-inheritance.b5f1e65cb8d1.jpg" width="785" height="744" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/ic-initial-employee-inheritance.b5f1e65cb8d1.jpg&w=196&sig=58f0045964de008f5860e073922ac03b99fcaf9c 196w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/ic-initial-employee-inheritance.b5f1e65cb8d1.jpg&w=392&sig=fce4157604c9d5ad7f8670d79b4a386bae166b02 392w, https://files.realpython.com/media/ic-initial-employee-inheritance.b5f1e65cb8d1.jpg 785w" sizes="75vw" alt="Inheritance example with multiple Employee derived classes"/></a></p>
10008 <p>The diagram shows the inheritance hierarchy of the classes. The derived classes implement the <code>IPayrollCalculator</code> interface, which is required by the <code>PayrollSystem</code>. The <code>PayrollSystem.calculate_payroll()</code> implementation requires that the <code>employee</code> objects passed contain an <code>id</code>, <code>name</code>, and <code>calculate_payroll()</code> implementation. </p>
10009 <p>Interfaces are represented similarly to classes with the word <strong>interface</strong> above the interface name. Interface names are usually prefixed with a capital <code>I</code>.</p>
10010 <p>The application creates its employees and passes them to the payroll system to process payroll:</p>
10011 <div class="highlight python"><pre><span></span><span class="c1"># In program.py</span>
10012
10013 <span class="kn">import</span> <span class="nn">hr</span>
10014
10015 <span class="n">salary_employee</span> <span class="o">=</span> <span class="n">hr</span><span class="o">.</span><span class="n">SalaryEmployee</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="s1">'John Smith'</span><span class="p">,</span> <span class="mi">1500</span><span class="p">)</span>
10016 <span class="n">hourly_employee</span> <span class="o">=</span> <span class="n">hr</span><span class="o">.</span><span class="n">HourlyEmployee</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="s1">'Jane Doe'</span><span class="p">,</span> <span class="mi">40</span><span class="p">,</span> <span class="mi">15</span><span class="p">)</span>
10017 <span class="n">commission_employee</span> <span class="o">=</span> <span class="n">hr</span><span class="o">.</span><span class="n">CommissionEmployee</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="s1">'Kevin Bacon'</span><span class="p">,</span> <span class="mi">1000</span><span class="p">,</span> <span class="mi">250</span><span class="p">)</span>
10018 <span class="n">payroll_system</span> <span class="o">=</span> <span class="n">hr</span><span class="o">.</span><span class="n">PayrollSystem</span><span class="p">()</span>
10019 <span class="n">payroll_system</span><span class="o">.</span><span class="n">calculate_payroll</span><span class="p">([</span>
10020 <span class="n">salary_employee</span><span class="p">,</span>
10021 <span class="n">hourly_employee</span><span class="p">,</span>
10022 <span class="n">commission_employee</span>
10023 <span class="p">])</span>
10024 </pre></div>
10025
10026 <p>You can run the program in the command line and see the results:</p>
10027 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python program.py
10028
10029 <span class="go">Calculating Payroll</span>
10030 <span class="go">===================</span>
10031 <span class="go">Payroll for: 1 - John Smith</span>
10032 <span class="go">- Check amount: 1500</span>
10033
10034 <span class="go">Payroll for: 2 - Jane Doe</span>
10035 <span class="go">- Check amount: 600</span>
10036
10037 <span class="go">Payroll for: 3 - Kevin Bacon</span>
10038 <span class="go">- Check amount: 1250</span>
10039 </pre></div>
10040
10041 <p>The program creates three employee objects, one for each of the derived classes. Then, it creates the payroll system and passes a list of the employees to its <code>.calculate_payroll()</code> method, which calculates the payroll for each employee and prints the results.</p>
10042 <p>Notice how the <code>Employee</code> base class doesn’t define a <code>.calculate_payroll()</code> method. This means that if you were to create a plain <code>Employee</code> object and pass it to the <code>PayrollSystem</code>, then you’d get an error. You can try it in the Python interactive interpreter:</p>
10043 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">hr</span>
10044 <span class="gp">>>> </span><span class="n">employee</span> <span class="o">=</span> <span class="n">hr</span><span class="o">.</span><span class="n">Employee</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="s1">'Invalid'</span><span class="p">)</span>
10045 <span class="gp">>>> </span><span class="n">payroll_system</span> <span class="o">=</span> <span class="n">hr</span><span class="o">.</span><span class="n">PayrollSystem</span><span class="p">()</span>
10046 <span class="gp">>>> </span><span class="n">payroll_system</span><span class="o">.</span><span class="n">calculate_payroll</span><span class="p">([</span><span class="n">employee</span><span class="p">])</span>
10047
10048 <span class="go">Payroll for: 1 - Invalid</span>
10049 <span class="gt">Traceback (most recent call last):</span>
10050 File <span class="nb">"<stdin>"</span>, line <span class="m">1</span>, in <span class="n"><module></span>
10051 File <span class="nb">"/hr.py"</span>, line <span class="m">39</span>, in <span class="n">calculate_payroll</span>
10052 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'- Check amount: {employee.calculate_payroll()}'</span><span class="p">)</span>
10053 <span class="gr">AttributeError</span>: <span class="n">'Employee' object has no attribute 'calculate_payroll'</span>
10054 </pre></div>
10055
10056 <p>While you can instantiate an <code>Employee</code> object, the object can’t be used by the <code>PayrollSystem</code>. Why? Because it can’t <code>.calculate_payroll()</code> for an <code>Employee</code>. To meet the requirements of <code>PayrollSystem</code>, you’ll want to convert the <code>Employee</code> class, which is currently a concrete class, to an abstract class. That way, no employee is ever just an <code>Employee</code>, but one that implements <code>.calculate_payroll()</code>.</p>
10057 <h3 id="abstract-base-classes-in-python">Abstract Base Classes in Python</h3>
10058 <p>The <code>Employee</code> class in the example above is what is called an abstract base class. Abstract base classes exist to be inherited, but never instantiated. Python provides the <code>abc</code> module to define abstract base classes.</p>
10059 <p>You can use <a href="https://dbader.org/blog/meaning-of-underscores-in-python">leading underscores</a> in your class name to communicate that objects of that class should not be created. Underscores provide a friendly way to prevent misuse of your code, but they don’t prevent eager users from creating instances of that class. </p>
10060 <p>The <a href="https://docs.python.org/3/library/abc.html#module-abc"><code>abc</code> module</a> in the Python standard library provides functionality to prevent creating objects from abstract base classes.</p>
10061 <p>You can modify the implementation of the <code>Employee</code> class to ensure that it can’t be instantiated:</p>
10062 <div class="highlight python"><pre><span></span><span class="c1"># In hr.py</span>
10063
10064 <span class="kn">from</span> <span class="nn">abc</span> <span class="k">import</span> <span class="n">ABC</span><span class="p">,</span> <span class="n">abstractmethod</span>
10065
10066 <span class="k">class</span> <span class="nc">Employee</span><span class="p">(</span><span class="n">ABC</span><span class="p">):</span>
10067 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
10068 <span class="bp">self</span><span class="o">.</span><span class="n">id</span> <span class="o">=</span> <span class="nb">id</span>
10069 <span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">name</span>
10070
10071 <span class="nd">@abstractmethod</span>
10072 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
10073 <span class="k">pass</span>
10074 </pre></div>
10075
10076 <p>You derive <code>Employee</code> from <code>ABC</code>, making it an abstract base class. Then, you decorate the <code>.calculate_payroll()</code> method with the <code>@abstractmethod</code> <a href="https://realpython.com/primer-on-python-decorators/">decorator</a>.</p>
10077 <p>This change has two nice side-effects:</p>
10078 <ol>
10079 <li>You’re telling users of the module that objects of type <code>Employee</code> can’t be created. </li>
10080 <li>You’re telling other developers working on the <code>hr</code> module that if they derive from <code>Employee</code>, then they must override the <code>.calculate_payroll()</code> abstract method.</li>
10081 </ol>
10082 <p>You can see that objects of type <code>Employee</code> can’t be created using the interactive interpreter:</p>
10083 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">hr</span>
10084 <span class="gp">>>> </span><span class="n">employee</span> <span class="o">=</span> <span class="n">hr</span><span class="o">.</span><span class="n">Employee</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="s1">'abstract'</span><span class="p">)</span>
10085
10086 <span class="gt">Traceback (most recent call last):</span>
10087 File <span class="nb">"<stdin>"</span>, line <span class="m">1</span>, in <span class="n"><module></span>
10088 <span class="gr">TypeError</span>: <span class="n">Can't instantiate abstract class Employee with abstract methods </span>
10089 <span class="go">calculate_payroll</span>
10090 </pre></div>
10091
10092 <p>The output shows that the class cannot be instantiated because it contains an abstract method <code>calculate_payroll()</code>. Derived classes must override the method to allow creating objects of their type.</p>
10093 <h3 id="implementation-inheritance-vs-interface-inheritance">Implementation Inheritance vs Interface Inheritance</h3>
10094 <p>When you derive one class from another, the derived class inherits both:</p>
10095 <ol>
10096 <li>
10097 <p><strong>The base class interface:</strong> The derived class inherits all the methods, properties, and attributes of the base class.</p>
10098 </li>
10099 <li>
10100 <p><strong>The base class implementation:</strong> The derived class inherits the code that implements the class interface.</p>
10101 </li>
10102 </ol>
10103 <p>Most of the time, you’ll want to inherit the implementation of a class, but you will want to implement multiple interfaces, so your objects can be used in different situations.</p>
10104 <p>Modern programming languages are designed with this basic concept in mind. They allow you to inherit from a single class, but you can implement multiple interfaces.</p>
10105 <p>In Python, you don’t have to explicitly declare an interface. Any object that implements the desired interface can be used in place of another object. This is known as <a href="https://realpython.com/python-type-checking/#duck-typing"><strong>duck typing</strong></a>. Duck typing is usually explained as “if it behaves like a duck, then it’s a duck.”</p>
10106 <p>To illustrate this, you will now add a <code>DisgruntledEmployee</code> class to the example above which doesn’t derive from <code>Employee</code>:</p>
10107 <div class="highlight python"><pre><span></span><span class="c1"># In disgruntled.py</span>
10108
10109 <span class="k">class</span> <span class="nc">DisgruntledEmployee</span><span class="p">:</span>
10110 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
10111 <span class="bp">self</span><span class="o">.</span><span class="n">id</span> <span class="o">=</span> <span class="nb">id</span>
10112 <span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">name</span>
10113
10114 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
10115 <span class="k">return</span> <span class="mi">1000000</span>
10116 </pre></div>
10117
10118 <p>The <code>DisgruntledEmployee</code> class doesn’t derive from <code>Employee</code>, but it exposes the same interface required by the <code>PayrollSystem</code>. The <code>PayrollSystem.calculate_payroll()</code> requires a list of objects that implement the following interface:</p>
10119 <ul>
10120 <li>An <strong><code>id</code></strong> property or attribute that returns the employee’s id</li>
10121 <li>A <strong><code>name</code></strong> property or attribute that represents the employee’s name</li>
10122 <li>A <strong><code>.calculate_payroll()</code></strong> method that doesn’t take any parameters and returns the payroll amount to process</li>
10123 </ul>
10124 <p>All these requirements are met by the <code>DisgruntledEmployee</code> class, so the <code>PayrollSystem</code> can still calculate its payroll.</p>
10125 <p>You can modify the program to use the <code>DisgruntledEmployee</code> class:</p>
10126 <div class="highlight python"><pre><span></span><span class="c1"># In program.py</span>
10127
10128 <span class="kn">import</span> <span class="nn">hr</span>
10129 <span class="kn">import</span> <span class="nn">disgruntled</span>
10130
10131 <span class="n">salary_employee</span> <span class="o">=</span> <span class="n">hr</span><span class="o">.</span><span class="n">SalaryEmployee</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="s1">'John Smith'</span><span class="p">,</span> <span class="mi">1500</span><span class="p">)</span>
10132 <span class="n">hourly_employee</span> <span class="o">=</span> <span class="n">hr</span><span class="o">.</span><span class="n">HourlyEmployee</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="s1">'Jane Doe'</span><span class="p">,</span> <span class="mi">40</span><span class="p">,</span> <span class="mi">15</span><span class="p">)</span>
10133 <span class="n">commission_employee</span> <span class="o">=</span> <span class="n">hr</span><span class="o">.</span><span class="n">CommissionEmployee</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="s1">'Kevin Bacon'</span><span class="p">,</span> <span class="mi">1000</span><span class="p">,</span> <span class="mi">250</span><span class="p">)</span>
10134 <span class="n">disgruntled_employee</span> <span class="o">=</span> <span class="n">disgruntled</span><span class="o">.</span><span class="n">DisgruntledEmployee</span><span class="p">(</span><span class="mi">20000</span><span class="p">,</span> <span class="s1">'Anonymous'</span><span class="p">)</span>
10135 <span class="n">payroll_system</span> <span class="o">=</span> <span class="n">hr</span><span class="o">.</span><span class="n">PayrollSystem</span><span class="p">()</span>
10136 <span class="n">payroll_system</span><span class="o">.</span><span class="n">calculate_payroll</span><span class="p">([</span>
10137 <span class="n">salary_employee</span><span class="p">,</span>
10138 <span class="n">hourly_employee</span><span class="p">,</span>
10139 <span class="n">commission_employee</span><span class="p">,</span>
10140 <span class="n">disgruntled_employee</span>
10141 <span class="p">])</span>
10142 </pre></div>
10143
10144 <p>The program creates a <code>DisgruntledEmployee</code> object and adds it to the list processed by the <code>PayrollSystem</code>. You can now run the program and see its output:</p>
10145 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python program.py
10146
10147 <span class="go">Calculating Payroll</span>
10148 <span class="go">===================</span>
10149 <span class="go">Payroll for: 1 - John Smith</span>
10150 <span class="go">- Check amount: 1500</span>
10151
10152 <span class="go">Payroll for: 2 - Jane Doe</span>
10153 <span class="go">- Check amount: 600</span>
10154
10155 <span class="go">Payroll for: 3 - Kevin Bacon</span>
10156 <span class="go">- Check amount: 1250</span>
10157
10158 <span class="go">Payroll for: 20000 - Anonymous</span>
10159 <span class="go">- Check amount: 1000000</span>
10160 </pre></div>
10161
10162 <p>As you can see, the <code>PayrollSystem</code> can still process the new object because it meets the desired interface.</p>
10163 <p>Since you don’t have to derive from a specific class for your objects to be reusable by the program, you may be asking why you should use inheritance instead of just implementing the desired interface. The following rules may help you:</p>
10164 <ul>
10165 <li>
10166 <p><strong>Use inheritance to reuse an implementation:</strong> Your derived classes should leverage most of their base class implementation. They must also model an <strong>is a</strong> relationship. A <code>Customer</code> class might also have an <code>id</code> and a <code>name</code>, but a <code>Customer</code> is not an <code>Employee</code>, so you should not use inheritance.</p>
10167 </li>
10168 <li>
10169 <p><strong>Implement an interface to be reused:</strong> When you want your class to be reused by a specific part of your application, you implement the required interface in your class, but you don’t need to provide a base class, or inherit from another class.</p>
10170 </li>
10171 </ul>
10172 <p>You can now clean up the example above to move onto the next topic. You can delete the <code>disgruntled.py</code> file and then modify the <code>hr</code> module to its original state:</p>
10173 <div class="highlight python"><pre><span></span><span class="c1"># In hr.py</span>
10174
10175 <span class="k">class</span> <span class="nc">PayrollSystem</span><span class="p">:</span>
10176 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">employees</span><span class="p">):</span>
10177 <span class="nb">print</span><span class="p">(</span><span class="s1">'Calculating Payroll'</span><span class="p">)</span>
10178 <span class="nb">print</span><span class="p">(</span><span class="s1">'==================='</span><span class="p">)</span>
10179 <span class="k">for</span> <span class="n">employee</span> <span class="ow">in</span> <span class="n">employees</span><span class="p">:</span>
10180 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Payroll for: </span><span class="si">{employee.id}</span><span class="s1"> - </span><span class="si">{employee.name}</span><span class="s1">'</span><span class="p">)</span>
10181 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'- Check amount: {employee.calculate_payroll()}'</span><span class="p">)</span>
10182 <span class="nb">print</span><span class="p">(</span><span class="s1">''</span><span class="p">)</span>
10183
10184 <span class="k">class</span> <span class="nc">Employee</span><span class="p">:</span>
10185 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
10186 <span class="bp">self</span><span class="o">.</span><span class="n">id</span> <span class="o">=</span> <span class="nb">id</span>
10187 <span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">name</span>
10188
10189 <span class="k">class</span> <span class="nc">SalaryEmployee</span><span class="p">(</span><span class="n">Employee</span><span class="p">):</span>
10190 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">weekly_salary</span><span class="p">):</span>
10191 <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span>
10192 <span class="bp">self</span><span class="o">.</span><span class="n">weekly_salary</span> <span class="o">=</span> <span class="n">weekly_salary</span>
10193
10194 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
10195 <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">weekly_salary</span>
10196
10197 <span class="k">class</span> <span class="nc">HourlyEmployee</span><span class="p">(</span><span class="n">Employee</span><span class="p">):</span>
10198 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">hours_worked</span><span class="p">,</span> <span class="n">hour_rate</span><span class="p">):</span>
10199 <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span>
10200 <span class="bp">self</span><span class="o">.</span><span class="n">hours_worked</span> <span class="o">=</span> <span class="n">hours_worked</span>
10201 <span class="bp">self</span><span class="o">.</span><span class="n">hour_rate</span> <span class="o">=</span> <span class="n">hour_rate</span>
10202
10203 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
10204 <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">hours_worked</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">hour_rate</span>
10205
10206 <span class="k">class</span> <span class="nc">CommissionEmployee</span><span class="p">(</span><span class="n">SalaryEmployee</span><span class="p">):</span>
10207 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">weekly_salary</span><span class="p">,</span> <span class="n">commission</span><span class="p">):</span>
10208 <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">weekly_salary</span><span class="p">)</span>
10209 <span class="bp">self</span><span class="o">.</span><span class="n">commission</span> <span class="o">=</span> <span class="n">commission</span>
10210
10211 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
10212 <span class="n">fixed</span> <span class="o">=</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">calculate_payroll</span><span class="p">()</span>
10213 <span class="k">return</span> <span class="n">fixed</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">commission</span>
10214 </pre></div>
10215
10216 <p>You removed the import of the <code>abc</code> module since the <code>Employee</code> class doesn’t need to be abstract. You also removed the abstract <code>calculate_payroll()</code> method from it since it doesn’t provide any implementation.</p>
10217 <p>Basically, you are inheriting the implementation of the <code>id</code> and <code>name</code> attributes of the <code>Employee</code> class in your derived classes. Since <code>.calculate_payroll()</code> is just an interface to the <code>PayrollSystem.calculate_payroll()</code> method, you don’t need to implement it in the <code>Employee</code> base class.</p>
10218 <p>Notice how the <code>CommissionEmployee</code> class derives from <code>SalaryEmployee</code>. This means that <code>CommissionEmployee</code> inherits the implementation and interface of <code>SalaryEmployee</code>. You can see how the <code>CommissionEmployee.calculate_payroll()</code> method leverages the base class implementation because it relies on the result from <code>super().calculate_payroll()</code> to implement its own version.</p>
10219 <h3 id="the-class-explosion-problem">The Class Explosion Problem</h3>
10220 <p>If you are not careful, inheritance can lead you to a huge hierarchical structure of classes that is hard to understand and maintain. This is known as the <strong>class explosion problem</strong>.</p>
10221 <p>You started building a class hierarchy of <code>Employee</code> types used by the <code>PayrollSystem</code> to calculate payroll. Now, you need to add some functionality to those classes, so they can be used with the new <code>ProductivitySystem</code>. </p>
10222 <p>The <code>ProductivitySystem</code> tracks productivity based on employee roles. There are different employee roles:</p>
10223 <ul>
10224 <li><strong>Managers:</strong> They walk around yelling at people telling them what to do. They are salaried employees and make more money.</li>
10225 <li><strong>Secretaries:</strong> They do all the paper work for managers and ensure that everything gets billed and payed on time. They are also salaried employees but make less money.</li>
10226 <li><strong>Sales employees:</strong> They make a lot of phone calls to sell products. They have a salary, but they also get commissions for sales.</li>
10227 <li><strong>Factory workers:</strong> They manufacture the products for the company. They are paid by the hour.</li>
10228 </ul>
10229 <p>With those requirements, you start to see that <code>Employee</code> and its derived classes might belong somewhere other than the <code>hr</code> module because now they’re also used by the <code>ProductivitySystem</code>.</p>
10230 <p>You create an <code>employees</code> module and move the classes there:</p>
10231 <div class="highlight python"><pre><span></span><span class="c1"># In employees.py</span>
10232
10233 <span class="k">class</span> <span class="nc">Employee</span><span class="p">:</span>
10234 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
10235 <span class="bp">self</span><span class="o">.</span><span class="n">id</span> <span class="o">=</span> <span class="nb">id</span>
10236 <span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">name</span>
10237
10238 <span class="k">class</span> <span class="nc">SalaryEmployee</span><span class="p">(</span><span class="n">Employee</span><span class="p">):</span>
10239 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">weekly_salary</span><span class="p">):</span>
10240 <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span>
10241 <span class="bp">self</span><span class="o">.</span><span class="n">weekly_salary</span> <span class="o">=</span> <span class="n">weekly_salary</span>
10242
10243 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
10244 <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">weekly_salary</span>
10245
10246 <span class="k">class</span> <span class="nc">HourlyEmployee</span><span class="p">(</span><span class="n">Employee</span><span class="p">):</span>
10247 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">hours_worked</span><span class="p">,</span> <span class="n">hour_rate</span><span class="p">):</span>
10248 <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span>
10249 <span class="bp">self</span><span class="o">.</span><span class="n">hours_worked</span> <span class="o">=</span> <span class="n">hours_worked</span>
10250 <span class="bp">self</span><span class="o">.</span><span class="n">hour_rate</span> <span class="o">=</span> <span class="n">hour_rate</span>
10251
10252 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
10253 <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">hours_worked</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">hour_rate</span>
10254
10255 <span class="k">class</span> <span class="nc">CommissionEmployee</span><span class="p">(</span><span class="n">SalaryEmployee</span><span class="p">):</span>
10256 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">weekly_salary</span><span class="p">,</span> <span class="n">commission</span><span class="p">):</span>
10257 <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">weekly_salary</span><span class="p">)</span>
10258 <span class="bp">self</span><span class="o">.</span><span class="n">commission</span> <span class="o">=</span> <span class="n">commission</span>
10259
10260 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
10261 <span class="n">fixed</span> <span class="o">=</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">calculate_payroll</span><span class="p">()</span>
10262 <span class="k">return</span> <span class="n">fixed</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">commission</span>
10263 </pre></div>
10264
10265 <p>The implementation remains the same, but you move the classes to the <code>employee</code> module. Now, you change your program to support the change:</p>
10266 <div class="highlight python"><pre><span></span><span class="c1"># In program.py</span>
10267
10268 <span class="kn">import</span> <span class="nn">hr</span>
10269 <span class="kn">import</span> <span class="nn">employees</span>
10270
10271 <span class="n">salary_employee</span> <span class="o">=</span> <span class="n">employees</span><span class="o">.</span><span class="n">SalaryEmployee</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="s1">'John Smith'</span><span class="p">,</span> <span class="mi">1500</span><span class="p">)</span>
10272 <span class="n">hourly_employee</span> <span class="o">=</span> <span class="n">employees</span><span class="o">.</span><span class="n">HourlyEmployee</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="s1">'Jane Doe'</span><span class="p">,</span> <span class="mi">40</span><span class="p">,</span> <span class="mi">15</span><span class="p">)</span>
10273 <span class="n">commission_employee</span> <span class="o">=</span> <span class="n">employees</span><span class="o">.</span><span class="n">CommissionEmployee</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="s1">'Kevin Bacon'</span><span class="p">,</span> <span class="mi">1000</span><span class="p">,</span> <span class="mi">250</span><span class="p">)</span>
10274 <span class="n">payroll_system</span> <span class="o">=</span> <span class="n">hr</span><span class="o">.</span><span class="n">PayrollSystem</span><span class="p">()</span>
10275 <span class="n">payroll_system</span><span class="o">.</span><span class="n">calculate_payroll</span><span class="p">([</span>
10276 <span class="n">salary_employee</span><span class="p">,</span>
10277 <span class="n">hourly_employee</span><span class="p">,</span>
10278 <span class="n">commission_employee</span>
10279 <span class="p">])</span>
10280 </pre></div>
10281
10282 <p>You run the program and verify that it still works:</p>
10283 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python program.py
10284
10285 <span class="go">Calculating Payroll</span>
10286 <span class="go">===================</span>
10287 <span class="go">Payroll for: 1 - John Smith</span>
10288 <span class="go">- Check amount: 1500</span>
10289
10290 <span class="go">Payroll for: 2 - Jane Doe</span>
10291 <span class="go">- Check amount: 600</span>
10292
10293 <span class="go">Payroll for: 3 - Kevin Bacon</span>
10294 <span class="go">- Check amount: 1250</span>
10295 </pre></div>
10296
10297 <p>With everything in place, you start adding the new classes:</p>
10298 <div class="highlight python"><pre><span></span><span class="c1"># In employees.py</span>
10299
10300 <span class="k">class</span> <span class="nc">Manager</span><span class="p">(</span><span class="n">SalaryEmployee</span><span class="p">):</span>
10301 <span class="k">def</span> <span class="nf">work</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hours</span><span class="p">):</span>
10302 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'</span><span class="si">{self.name}</span><span class="s1"> screams and yells for </span><span class="si">{hours}</span><span class="s1"> hours.'</span><span class="p">)</span>
10303
10304 <span class="k">class</span> <span class="nc">Secretary</span><span class="p">(</span><span class="n">SalaryEmployee</span><span class="p">):</span>
10305 <span class="k">def</span> <span class="nf">work</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hours</span><span class="p">):</span>
10306 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'</span><span class="si">{self.name}</span><span class="s1"> expends </span><span class="si">{hours}</span><span class="s1"> hours doing office paperwork.'</span><span class="p">)</span>
10307
10308 <span class="k">class</span> <span class="nc">SalesPerson</span><span class="p">(</span><span class="n">CommissionEmployee</span><span class="p">):</span>
10309 <span class="k">def</span> <span class="nf">work</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hours</span><span class="p">):</span>
10310 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'</span><span class="si">{self.name}</span><span class="s1"> expends </span><span class="si">{hours}</span><span class="s1"> hours on the phone.'</span><span class="p">)</span>
10311
10312 <span class="k">class</span> <span class="nc">FactoryWorker</span><span class="p">(</span><span class="n">HourlyEmployee</span><span class="p">):</span>
10313 <span class="k">def</span> <span class="nf">work</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hours</span><span class="p">):</span>
10314 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'</span><span class="si">{self.name}</span><span class="s1"> manufactures gadgets for </span><span class="si">{hours}</span><span class="s1"> hours.'</span><span class="p">)</span>
10315 </pre></div>
10316
10317 <p>First, you add a <code>Manager</code> class that derives from <code>SalaryEmployee</code>. The class exposes a method <code>work()</code> that will be used by the productivity system. The method takes the <code>hours</code> the employee worked.</p>
10318 <p>Then you add <code>Secretary</code>, <code>SalesPerson</code>, and <code>FactoryWorker</code> and then implement the <code>work()</code> interface, so they can be used by the productivity system.</p>
10319 <p>Now, you can add the <code>ProductivitySytem</code> class:</p>
10320 <div class="highlight python"><pre><span></span><span class="c1"># In productivity.py</span>
10321
10322 <span class="k">class</span> <span class="nc">ProductivitySystem</span><span class="p">:</span>
10323 <span class="k">def</span> <span class="nf">track</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">employees</span><span class="p">,</span> <span class="n">hours</span><span class="p">):</span>
10324 <span class="nb">print</span><span class="p">(</span><span class="s1">'Tracking Employee Productivity'</span><span class="p">)</span>
10325 <span class="nb">print</span><span class="p">(</span><span class="s1">'=============================='</span><span class="p">)</span>
10326 <span class="k">for</span> <span class="n">employee</span> <span class="ow">in</span> <span class="n">employees</span><span class="p">:</span>
10327 <span class="n">employee</span><span class="o">.</span><span class="n">work</span><span class="p">(</span><span class="n">hours</span><span class="p">)</span>
10328 <span class="nb">print</span><span class="p">(</span><span class="s1">''</span><span class="p">)</span>
10329 </pre></div>
10330
10331 <p>The class tracks employees in the <code>track()</code> method that takes a list of employees and the number of hours to track. You can now add the productivity system to your program:</p>
10332 <div class="highlight python"><pre><span></span><span class="c1"># In program.py</span>
10333
10334 <span class="kn">import</span> <span class="nn">hr</span>
10335 <span class="kn">import</span> <span class="nn">employees</span>
10336 <span class="kn">import</span> <span class="nn">productivity</span>
10337
10338 <span class="n">manager</span> <span class="o">=</span> <span class="n">employees</span><span class="o">.</span><span class="n">Manager</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="s1">'Mary Poppins'</span><span class="p">,</span> <span class="mi">3000</span><span class="p">)</span>
10339 <span class="n">secretary</span> <span class="o">=</span> <span class="n">employees</span><span class="o">.</span><span class="n">Secretary</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="s1">'John Smith'</span><span class="p">,</span> <span class="mi">1500</span><span class="p">)</span>
10340 <span class="n">sales_guy</span> <span class="o">=</span> <span class="n">employees</span><span class="o">.</span><span class="n">SalesPerson</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="s1">'Kevin Bacon'</span><span class="p">,</span> <span class="mi">1000</span><span class="p">,</span> <span class="mi">250</span><span class="p">)</span>
10341 <span class="n">factory_worker</span> <span class="o">=</span> <span class="n">employees</span><span class="o">.</span><span class="n">FactoryWorker</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="s1">'Jane Doe'</span><span class="p">,</span> <span class="mi">40</span><span class="p">,</span> <span class="mi">15</span><span class="p">)</span>
10342 <span class="n">employees</span> <span class="o">=</span> <span class="p">[</span>
10343 <span class="n">manager</span><span class="p">,</span>
10344 <span class="n">secretary</span><span class="p">,</span>
10345 <span class="n">sales_guy</span><span class="p">,</span>
10346 <span class="n">factory_worker</span><span class="p">,</span>
10347 <span class="p">]</span>
10348 <span class="n">productivity_system</span> <span class="o">=</span> <span class="n">productivity</span><span class="o">.</span><span class="n">ProductivitySystem</span><span class="p">()</span>
10349 <span class="n">productivity_system</span><span class="o">.</span><span class="n">track</span><span class="p">(</span><span class="n">employees</span><span class="p">,</span> <span class="mi">40</span><span class="p">)</span>
10350 <span class="n">payroll_system</span> <span class="o">=</span> <span class="n">hr</span><span class="o">.</span><span class="n">PayrollSystem</span><span class="p">()</span>
10351 <span class="n">payroll_system</span><span class="o">.</span><span class="n">calculate_payroll</span><span class="p">(</span><span class="n">employees</span><span class="p">)</span>
10352 </pre></div>
10353
10354 <p>The program creates a list of employees of different types. The employee list is sent to the productivity system to track their work for 40 hours. Then the same list of employees is sent to the payroll system to calculate their payroll.</p>
10355 <p>You can run the program to see the output:</p>
10356 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python program.py
10357
10358 <span class="go">Tracking Employee Productivity</span>
10359 <span class="go">==============================</span>
10360 <span class="go">Mary Poppins screams and yells for 40 hours.</span>
10361 <span class="go">John Smith expends 40 hours doing office paperwork.</span>
10362 <span class="go">Kevin Bacon expends 40 hours on the phone.</span>
10363 <span class="go">Jane Doe manufactures gadgets for 40 hours.</span>
10364
10365 <span class="go">Calculating Payroll</span>
10366 <span class="go">===================</span>
10367 <span class="go">Payroll for: 1 - Mary Poppins</span>
10368 <span class="go">- Check amount: 3000</span>
10369
10370 <span class="go">Payroll for: 2 - John Smith</span>
10371 <span class="go">- Check amount: 1500</span>
10372
10373 <span class="go">Payroll for: 3 - Kevin Bacon</span>
10374 <span class="go">- Check amount: 1250</span>
10375
10376 <span class="go">Payroll for: 4 - Jane Doe</span>
10377 <span class="go">- Check amount: 600</span>
10378 </pre></div>
10379
10380 <p>The program shows the employees working for 40 hours through the productivity system. Then it calculates and displays the payroll for each of the employees.</p>
10381 <p>The program works as expected, but you had to add four new classes to support the changes. As new requirements come, your class hierarchy will inevitably grow, leading to the class explosion problem where your hierarchies will become so big that they’ll be hard to understand and maintain.</p>
10382 <p>The following diagram shows the new class hierarchy:</p>
10383 <p><a href="https://files.realpython.com/media/ic-class-explosion.a3d42b8c9b91.jpg" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/ic-class-explosion.a3d42b8c9b91.jpg" width="1184" height="1199" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/ic-class-explosion.a3d42b8c9b91.jpg&w=296&sig=3df3dc7dac2263ed3beba586e8731fc276936b39 296w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/ic-class-explosion.a3d42b8c9b91.jpg&w=592&sig=e2d4c2f1d5145230cd248e731aca8cdfe5e25709 592w, https://files.realpython.com/media/ic-class-explosion.a3d42b8c9b91.jpg 1184w" sizes="75vw" alt="Class design explosion by inheritance"/></a></p>
10384 <p>The diagram shows how the class hierarchy is growing. Additional requirements might have an exponential effect in the number of classes with this design.</p>
10385 <h3 id="inheriting-multiple-classes">Inheriting Multiple Classes</h3>
10386 <p>Python is one of the few modern programming languages that supports multiple inheritance. Multiple inheritance is the ability to derive a class from multiple base classes at the same time.</p>
10387 <p>Multiple inheritance has a bad reputation to the extent that most modern programming languages don’t support it. Instead, modern programming languages support the concept of interfaces. In those languages, you inherit from a single base class and then implement multiple interfaces, so your class can be re-used in different situations.</p>
10388 <p>This approach puts some constraints in your designs. You can only inherit the implementation of one class by directly deriving from it. You can implement multiple interfaces, but you can’t inherit the implementation of multiple classes.</p>
10389 <p>This constraint is good for software design because it forces you to design your classes with fewer dependencies on each other. You will see later in this article that you can leverage multiple implementations through composition, which makes software more flexible. This section, however, is about multiple inheritance, so let’s take a look at how it works.</p>
10390 <p>It turns out that sometimes temporary secretaries are hired when there is too much paperwork to do. The <code>TemporarySecretary</code> class performs the role of a <code>Secretary</code> in the context of the <code>ProductivitySystem</code>, but for payroll purposes, it is an <code>HourlyEmployee</code>.</p>
10391 <p>You look at your class design. It has grown a little bit, but you can still understand how it works. It seems you have two options:</p>
10392 <ol>
10393 <li>
10394 <p><strong>Derive from <code>Secretary</code>:</strong> You can derive from <code>Secretary</code> to inherit the <code>.work()</code> method for the role, and then override the <code>.calculate_payroll()</code> method to implement it as an <code>HourlyEmployee</code>.</p>
10395 </li>
10396 <li>
10397 <p><strong>Derive from <code>HourlyEmployee</code>:</strong> You can derive from <code>HourlyEmployee</code> to inherit the <code>.calculate_payroll()</code> method, and then override the <code>.work()</code> method to implement it as a <code>Secretary</code>.</p>
10398 </li>
10399 </ol>
10400 <p>Then, you remember that Python supports multiple inheritance, so you decide to derive from both <code>Secretary</code> and <code>HourlyEmployee</code>:</p>
10401 <div class="highlight python"><pre><span></span><span class="c1"># In employees.py</span>
10402
10403 <span class="k">class</span> <span class="nc">TemporarySecretary</span><span class="p">(</span><span class="n">Secretary</span><span class="p">,</span> <span class="n">HourlyEmployee</span><span class="p">):</span>
10404 <span class="k">pass</span>
10405 </pre></div>
10406
10407 <p>Python allows you to inherit from two different classes by specifying them between parenthesis in the class declaration.</p>
10408 <p>Now, you modify your program to add the new temporary secretary employee:</p>
10409 <div class="highlight python"><pre><span></span><span class="kn">import</span> <span class="nn">hr</span>
10410 <span class="kn">import</span> <span class="nn">employees</span>
10411 <span class="kn">import</span> <span class="nn">productivity</span>
10412
10413 <span class="n">manager</span> <span class="o">=</span> <span class="n">employees</span><span class="o">.</span><span class="n">Manager</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="s1">'Mary Poppins'</span><span class="p">,</span> <span class="mi">3000</span><span class="p">)</span>
10414 <span class="n">secretary</span> <span class="o">=</span> <span class="n">employees</span><span class="o">.</span><span class="n">Secretary</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="s1">'John Smith'</span><span class="p">,</span> <span class="mi">1500</span><span class="p">)</span>
10415 <span class="n">sales_guy</span> <span class="o">=</span> <span class="n">employees</span><span class="o">.</span><span class="n">SalesPerson</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="s1">'Kevin Bacon'</span><span class="p">,</span> <span class="mi">1000</span><span class="p">,</span> <span class="mi">250</span><span class="p">)</span>
10416 <span class="n">factory_worker</span> <span class="o">=</span> <span class="n">employees</span><span class="o">.</span><span class="n">FactoryWorker</span><span class="p">(</span><span class="mi">4</span><span class="p">,</span> <span class="s1">'Jane Doe'</span><span class="p">,</span> <span class="mi">40</span><span class="p">,</span> <span class="mi">15</span><span class="p">)</span>
10417 <span class="hll"><span class="n">temporary_secretary</span> <span class="o">=</span> <span class="n">employees</span><span class="o">.</span><span class="n">TemporarySecretary</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="s1">'Robin Williams'</span><span class="p">,</span> <span class="mi">40</span><span class="p">,</span> <span class="mi">9</span><span class="p">)</span>
10418 </span><span class="n">company_employees</span> <span class="o">=</span> <span class="p">[</span>
10419 <span class="n">manager</span><span class="p">,</span>
10420 <span class="n">secretary</span><span class="p">,</span>
10421 <span class="n">sales_guy</span><span class="p">,</span>
10422 <span class="n">factory_worker</span><span class="p">,</span>
10423 <span class="hll"> <span class="n">temporary_secretary</span><span class="p">,</span>
10424 </span><span class="p">]</span>
10425 <span class="n">productivity_system</span> <span class="o">=</span> <span class="n">productivity</span><span class="o">.</span><span class="n">ProductivitySystem</span><span class="p">()</span>
10426 <span class="n">productivity_system</span><span class="o">.</span><span class="n">track</span><span class="p">(</span><span class="n">company_employees</span><span class="p">,</span> <span class="mi">40</span><span class="p">)</span>
10427 <span class="n">payroll_system</span> <span class="o">=</span> <span class="n">hr</span><span class="o">.</span><span class="n">PayrollSystem</span><span class="p">()</span>
10428 <span class="n">payroll_system</span><span class="o">.</span><span class="n">calculate_payroll</span><span class="p">(</span><span class="n">company_employees</span><span class="p">)</span>
10429 </pre></div>
10430
10431 <p>You run the program to test it:</p>
10432 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python program.py
10433
10434 <span class="go">Traceback (most recent call last):</span>
10435 <span class="go"> File ".\program.py", line 9, in <module></span>
10436 <span class="go"> temporary_secretary = employee.TemporarySecretary(5, 'Robin Williams', 40, 9)</span>
10437 <span class="go">TypeError: __init__() takes 4 positional arguments but 5 were given</span>
10438 </pre></div>
10439
10440 <p>You get a <a href="https://docs.python.org/3/library/exceptions.html#TypeError"><code>TypeError</code></a> exception saying that <code>4</code> positional arguments where expected, but <code>5</code> were given.</p>
10441 <p>This is because you derived <code>TemporarySecretary</code> first from <code>Secretary</code> and then from <code>HourlyEmployee</code>, so the interpreter is trying to use <code>Secretary.__init__()</code> to initialize the object.</p>
10442 <p>Okay, let’s reverse it:</p>
10443 <div class="highlight python"><pre><span></span><span class="k">class</span> <span class="nc">TemporarySecretary</span><span class="p">(</span><span class="n">HourlyEmployee</span><span class="p">,</span> <span class="n">Secretary</span><span class="p">):</span>
10444 <span class="k">pass</span>
10445 </pre></div>
10446
10447 <p>Now, run the program again and see what happens:</p>
10448 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python program.py
10449
10450 <span class="go">Traceback (most recent call last):</span>
10451 <span class="go"> File ".\program.py", line 9, in <module></span>
10452 <span class="go"> temporary_secretary = employee.TemporarySecretary(5, 'Robin Williams', 40, 9)</span>
10453 <span class="go"> File "employee.py", line 16, in __init__</span>
10454 <span class="go"> super().__init__(id, name)</span>
10455 <span class="go">TypeError: __init__() missing 1 required positional argument: 'weekly_salary'</span>
10456 </pre></div>
10457
10458 <p>Now it seems you are missing a <code>weekly_salary</code> parameter, which is necessary to initialize <code>Secretary</code>, but that parameter doesn’t make sense in the context of a <code>TemporarySecretary</code> because it’s an <code>HourlyEmployee</code>.</p>
10459 <p>Maybe implementing <code>TemporarySecretary.__init__()</code> will help:</p>
10460 <div class="highlight python"><pre><span></span><span class="c1"># In employees.py</span>
10461
10462 <span class="k">class</span> <span class="nc">TemporarySecretary</span><span class="p">(</span><span class="n">HourlyEmployee</span><span class="p">,</span> <span class="n">Secretary</span><span class="p">):</span>
10463 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">hours_worked</span><span class="p">,</span> <span class="n">hour_rate</span><span class="p">):</span>
10464 <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">hours_worked</span><span class="p">,</span> <span class="n">hour_rate</span><span class="p">)</span>
10465 </pre></div>
10466
10467 <p>Try it:</p>
10468 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python program.py
10469
10470 <span class="go">Traceback (most recent call last):</span>
10471 <span class="go"> File ".\program.py", line 9, in <module></span>
10472 <span class="go"> temporary_secretary = employee.TemporarySecretary(5, 'Robin Williams', 40, 9)</span>
10473 <span class="go"> File "employee.py", line 54, in __init__</span>
10474 <span class="go"> super().__init__(id, name, hours_worked, hour_rate)</span>
10475 <span class="go"> File "employee.py", line 16, in __init__</span>
10476 <span class="go"> super().__init__(id, name)</span>
10477 <span class="go">TypeError: __init__() missing 1 required positional argument: 'weekly_salary'</span>
10478 </pre></div>
10479
10480 <p>That didn’t work either. Okay, it’s time for you to dive into Python’s <strong>method resolution order</strong> (MRO) to see what’s going on.</p>
10481 <p>When a method or attribute of a class is accessed, Python uses the class <a href="https://www.python.org/download/releases/2.3/mro/">MRO</a> to find it. The MRO is also used by <code>super()</code> to determine which method or attribute to invoke. You can learn more about <code>super()</code> in <a href="https://realpython.com/python-super/">Supercharge Your Classes With Python super()</a>.</p>
10482 <p>You can evaluate the <code>TemporarySecretary</code> class MRO using the interactive interpreter:</p>
10483 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">employees</span> <span class="k">import</span> <span class="n">TemporarySecretary</span>
10484 <span class="gp">>>> </span><span class="n">TemporarySecretary</span><span class="o">.</span><span class="vm">__mro__</span>
10485
10486 <span class="go">(<class 'employees.TemporarySecretary'>,</span>
10487 <span class="go"> <class 'employees.HourlyEmployee'>,</span>
10488 <span class="go"> <class 'employees.Secretary'>,</span>
10489 <span class="go"> <class 'employees.SalaryEmployee'>,</span>
10490 <span class="go"> <class 'employees.Employee'>,</span>
10491 <span class="go"> <class 'object'></span>
10492 <span class="go">)</span>
10493 </pre></div>
10494
10495 <p>The MRO shows the order in which Python is going to look for a matching attribute or method. In the example, this is what happens when we create the <code>TemporarySecretary</code> object:</p>
10496 <ol>
10497 <li>
10498 <p>The <code>TemporarySecretary.__init__(self, id, name, hours_worked, hour_rate)</code> method is called.</p>
10499 </li>
10500 <li>
10501 <p>The <code>super().__init__(id, name, hours_worked, hour_rate)</code> call matches <code>HourlyEmployee.__init__(self, id, name, hour_worked, hour_rate)</code>.</p>
10502 </li>
10503 <li>
10504 <p><code>HourlyEmployee</code> calls <code>super().__init__(id, name)</code>, which the MRO is going to match to <code>Secretary.__init__()</code>, which is inherited from <code>SalaryEmployee.__init__(self, id, name, weekly_salary)</code>.</p>
10505 </li>
10506 </ol>
10507 <p>Because the parameters don’t match, a <code>TypeError</code> exception is raised.</p>
10508 <p>You can bypass the MRO by reversing the inheritance order and directly calling <code>HourlyEmployee.__init__()</code> as follows:</p>
10509 <div class="highlight python"><pre><span></span><span class="k">class</span> <span class="nc">TemporarySecretary</span><span class="p">(</span><span class="n">Secretary</span><span class="p">,</span> <span class="n">HourlyEmployee</span><span class="p">):</span>
10510 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">hours_worked</span><span class="p">,</span> <span class="n">hour_rate</span><span class="p">):</span>
10511 <span class="n">HourlyEmployee</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">hours_worked</span><span class="p">,</span> <span class="n">hour_rate</span><span class="p">)</span>
10512 </pre></div>
10513
10514 <p>That solves the problem of creating the object, but you will run into a similar problem when trying to calculate payroll. You can run the program to see the problem:</p>
10515 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python program.py
10516
10517 <span class="go">Tracking Employee Productivity</span>
10518 <span class="go">==============================</span>
10519 <span class="go">Mary Poppins screams and yells for 40 hours.</span>
10520 <span class="go">John Smith expends 40 hours doing office paperwork.</span>
10521 <span class="go">Kevin Bacon expends 40 hours on the phone.</span>
10522 <span class="go">Jane Doe manufactures gadgets for 40 hours.</span>
10523 <span class="go">Robin Williams expends 40 hours doing office paperwork.</span>
10524
10525 <span class="go">Calculating Payroll</span>
10526 <span class="go">===================</span>
10527 <span class="go">Payroll for: 1 - Mary Poppins</span>
10528 <span class="go">- Check amount: 3000</span>
10529
10530 <span class="go">Payroll for: 2 - John Smith</span>
10531 <span class="go">- Check amount: 1500</span>
10532
10533 <span class="go">Payroll for: 3 - Kevin Bacon</span>
10534 <span class="go">- Check amount: 1250</span>
10535
10536 <span class="go">Payroll for: 4 - Jane Doe</span>
10537 <span class="go">- Check amount: 600</span>
10538
10539 <span class="go">Payroll for: 5 - Robin Williams</span>
10540 <span class="go">Traceback (most recent call last):</span>
10541 <span class="go"> File ".\program.py", line 20, in <module></span>
10542 <span class="go"> payroll_system.calculate_payroll(employees)</span>
10543 <span class="go"> File "hr.py", line 7, in calculate_payroll</span>
10544 <span class="go"> print(f'- Check amount: {employee.calculate_payroll()}')</span>
10545 <span class="go"> File "employee.py", line 12, in calculate_payroll</span>
10546 <span class="go"> return self.weekly_salary</span>
10547 <span class="go">AttributeError: 'TemporarySecretary' object has no attribute 'weekly_salary'</span>
10548 </pre></div>
10549
10550 <p>The problem now is that because you reversed the inheritance order, the MRO is finding the <code>.calculate_payroll()</code> method of <code>SalariedEmployee</code> before the one in <code>HourlyEmployee</code>. You need to override <code>.calculate_payroll()</code> in <code>TemporarySecretary</code> and invoke the right implementation from it:</p>
10551 <div class="highlight python"><pre><span></span><span class="k">class</span> <span class="nc">TemporarySecretary</span><span class="p">(</span><span class="n">Secretary</span><span class="p">,</span> <span class="n">HourlyEmployee</span><span class="p">):</span>
10552 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">hours_worked</span><span class="p">,</span> <span class="n">hour_rate</span><span class="p">):</span>
10553 <span class="n">HourlyEmployee</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">hours_worked</span><span class="p">,</span> <span class="n">hour_rate</span><span class="p">)</span>
10554
10555 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
10556 <span class="k">return</span> <span class="n">HourlyEmployee</span><span class="o">.</span><span class="n">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span>
10557 </pre></div>
10558
10559 <p>The <code>calculate_payroll()</code> method directly invokes <code>HourlyEmployee.calculate_payroll()</code> to ensure that you get the correct result. You can run the program again to see it working:</p>
10560 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python program.py
10561
10562 <span class="go">Tracking Employee Productivity</span>
10563 <span class="go">==============================</span>
10564 <span class="go">Mary Poppins screams and yells for 40 hours.</span>
10565 <span class="go">John Smith expends 40 hours doing office paperwork.</span>
10566 <span class="go">Kevin Bacon expends 40 hours on the phone.</span>
10567 <span class="go">Jane Doe manufactures gadgets for 40 hours.</span>
10568 <span class="go">Robin Williams expends 40 hours doing office paperwork.</span>
10569
10570 <span class="go">Calculating Payroll</span>
10571 <span class="go">===================</span>
10572 <span class="go">Payroll for: 1 - Mary Poppins</span>
10573 <span class="go">- Check amount: 3000</span>
10574
10575 <span class="go">Payroll for: 2 - John Smith</span>
10576 <span class="go">- Check amount: 1500</span>
10577
10578 <span class="go">Payroll for: 3 - Kevin Bacon</span>
10579 <span class="go">- Check amount: 1250</span>
10580
10581 <span class="go">Payroll for: 4 - Jane Doe</span>
10582 <span class="go">- Check amount: 600</span>
10583
10584 <span class="go">Payroll for: 5 - Robin Williams</span>
10585 <span class="go">- Check amount: 360</span>
10586 </pre></div>
10587
10588 <p>The program now works as expected because you’re forcing the method resolution order by explicitly telling the interpreter which method we want to use.</p>
10589 <p>As you can see, multiple inheritance can be confusing, especially when you run into the <a href="https://en.wikipedia.org/wiki/Multiple_inheritance#The_diamond_problem">diamond problem</a>.</p>
10590 <p>The following diagram shows the diamond problem in your class hierarchy:</p>
10591 <p><a href="https://files.realpython.com/media/ic-diamond-problem.8e685f12d3c2.jpg" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/ic-diamond-problem.8e685f12d3c2.jpg" width="726" height="845" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/ic-diamond-problem.8e685f12d3c2.jpg&w=181&sig=8c590866d2b03d475c750f72405d8fd9e325c350 181w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/ic-diamond-problem.8e685f12d3c2.jpg&w=363&sig=13e5b6884a585b8f452fc384238507c40f9c97b8 363w, https://files.realpython.com/media/ic-diamond-problem.8e685f12d3c2.jpg 726w" sizes="75vw" alt="Diamond problem caused by multiple inheritance"/></a></p>
10592 <p>The diagram shows the diamond problem with the current class design. <code>TemporarySecretary</code> uses multiple inheritance to derive from two classes that ultimately also derive from <code>Employee</code>. This causes two paths to reach the <code>Employee</code> base class, which is something you want to avoid in your designs.</p>
10593 <p>The diamond problem appears when you’re using multiple inheritance and deriving from two classes that have a common base class. This can cause the wrong version of a method to be called. </p>
10594 <p>As you’ve seen, Python provides a way to force the right method to be invoked, and analyzing the MRO can help you understand the problem.</p>
10595 <p>Still, when you run into the diamond problem, it’s better to re-think the design. You will now make some changes to leverage multiple inheritance, avoiding the diamond problem.</p>
10596 <p>The <code>Employee</code> derived classes are used by two different systems:</p>
10597 <ol>
10598 <li>
10599 <p><strong>The productivity system</strong> that tracks employee productivity.</p>
10600 </li>
10601 <li>
10602 <p><strong>The payroll system</strong> that calculates the employee payroll.</p>
10603 </li>
10604 </ol>
10605 <p>This means that everything related to productivity should be together in one module and everything related to payroll should be together in another. You can start making changes to the productivity module:</p>
10606 <div class="highlight python"><pre><span></span><span class="c1"># In productivity.py</span>
10607
10608 <span class="k">class</span> <span class="nc">ProductivitySystem</span><span class="p">:</span>
10609 <span class="k">def</span> <span class="nf">track</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">employees</span><span class="p">,</span> <span class="n">hours</span><span class="p">):</span>
10610 <span class="nb">print</span><span class="p">(</span><span class="s1">'Tracking Employee Productivity'</span><span class="p">)</span>
10611 <span class="nb">print</span><span class="p">(</span><span class="s1">'=============================='</span><span class="p">)</span>
10612 <span class="k">for</span> <span class="n">employee</span> <span class="ow">in</span> <span class="n">employees</span><span class="p">:</span>
10613 <span class="n">result</span> <span class="o">=</span> <span class="n">employee</span><span class="o">.</span><span class="n">work</span><span class="p">(</span><span class="n">hours</span><span class="p">)</span>
10614 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'</span><span class="si">{employee.name}</span><span class="s1">: </span><span class="si">{result}</span><span class="s1">'</span><span class="p">)</span>
10615 <span class="nb">print</span><span class="p">(</span><span class="s1">''</span><span class="p">)</span>
10616
10617 <span class="k">class</span> <span class="nc">ManagerRole</span><span class="p">:</span>
10618 <span class="k">def</span> <span class="nf">work</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hours</span><span class="p">):</span>
10619 <span class="k">return</span> <span class="n">f</span><span class="s1">'screams and yells for </span><span class="si">{hours}</span><span class="s1"> hours.'</span>
10620
10621 <span class="k">class</span> <span class="nc">SecretaryRole</span><span class="p">:</span>
10622 <span class="k">def</span> <span class="nf">work</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hours</span><span class="p">):</span>
10623 <span class="k">return</span> <span class="n">f</span><span class="s1">'expends </span><span class="si">{hours}</span><span class="s1"> hours doing office paperwork.'</span>
10624
10625 <span class="k">class</span> <span class="nc">SalesRole</span><span class="p">:</span>
10626 <span class="k">def</span> <span class="nf">work</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hours</span><span class="p">):</span>
10627 <span class="k">return</span> <span class="n">f</span><span class="s1">'expends </span><span class="si">{hours}</span><span class="s1"> hours on the phone.'</span>
10628
10629 <span class="k">class</span> <span class="nc">FactoryRole</span><span class="p">:</span>
10630 <span class="k">def</span> <span class="nf">work</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hours</span><span class="p">):</span>
10631 <span class="k">return</span> <span class="n">f</span><span class="s1">'manufactures gadgets for </span><span class="si">{hours}</span><span class="s1"> hours.'</span>
10632 </pre></div>
10633
10634 <p>The <code>productivity</code> module implements the <code>ProductivitySystem</code> class, as well as the related roles it supports. The classes implement the <code>work()</code> interface required by the system, but they don’t derived from <code>Employee</code>.</p>
10635 <p>You can do the same with the <code>hr</code> module:</p>
10636 <div class="highlight python"><pre><span></span><span class="c1"># In hr.py</span>
10637
10638 <span class="k">class</span> <span class="nc">PayrollSystem</span><span class="p">:</span>
10639 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">employees</span><span class="p">):</span>
10640 <span class="nb">print</span><span class="p">(</span><span class="s1">'Calculating Payroll'</span><span class="p">)</span>
10641 <span class="nb">print</span><span class="p">(</span><span class="s1">'==================='</span><span class="p">)</span>
10642 <span class="k">for</span> <span class="n">employee</span> <span class="ow">in</span> <span class="n">employees</span><span class="p">:</span>
10643 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Payroll for: </span><span class="si">{employee.id}</span><span class="s1"> - </span><span class="si">{employee.name}</span><span class="s1">'</span><span class="p">)</span>
10644 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'- Check amount: {employee.calculate_payroll()}'</span><span class="p">)</span>
10645 <span class="nb">print</span><span class="p">(</span><span class="s1">''</span><span class="p">)</span>
10646
10647 <span class="k">class</span> <span class="nc">SalaryPolicy</span><span class="p">:</span>
10648 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">weekly_salary</span><span class="p">):</span>
10649 <span class="bp">self</span><span class="o">.</span><span class="n">weekly_salary</span> <span class="o">=</span> <span class="n">weekly_salary</span>
10650
10651 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
10652 <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">weekly_salary</span>
10653
10654 <span class="k">class</span> <span class="nc">HourlyPolicy</span><span class="p">:</span>
10655 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hours_worked</span><span class="p">,</span> <span class="n">hour_rate</span><span class="p">):</span>
10656 <span class="bp">self</span><span class="o">.</span><span class="n">hours_worked</span> <span class="o">=</span> <span class="n">hours_worked</span>
10657 <span class="bp">self</span><span class="o">.</span><span class="n">hour_rate</span> <span class="o">=</span> <span class="n">hour_rate</span>
10658
10659 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
10660 <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">hours_worked</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">hour_rate</span>
10661
10662 <span class="k">class</span> <span class="nc">CommissionPolicy</span><span class="p">(</span><span class="n">SalaryPolicy</span><span class="p">):</span>
10663 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">weekly_salary</span><span class="p">,</span> <span class="n">commission</span><span class="p">):</span>
10664 <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">weekly_salary</span><span class="p">)</span>
10665 <span class="bp">self</span><span class="o">.</span><span class="n">commission</span> <span class="o">=</span> <span class="n">commission</span>
10666
10667 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
10668 <span class="n">fixed</span> <span class="o">=</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">calculate_payroll</span><span class="p">()</span>
10669 <span class="k">return</span> <span class="n">fixed</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">commission</span>
10670 </pre></div>
10671
10672 <p>The <code>hr</code> module implements the <code>PayrollSystem</code>, which calculates payroll for the employees. It also implements the policy classes for payroll. As you can see, the policy classes don’t derive from <code>Employee</code> anymore.</p>
10673 <p>You can now add the necessary classes to the <code>employee</code> module:</p>
10674 <div class="highlight python"><pre><span></span><span class="c1"># In employees.py</span>
10675
10676 <span class="kn">from</span> <span class="nn">hr</span> <span class="k">import</span> <span class="p">(</span>
10677 <span class="n">SalaryPolicy</span><span class="p">,</span>
10678 <span class="n">CommissionPolicy</span><span class="p">,</span>
10679 <span class="n">HourlyPolicy</span>
10680 <span class="p">)</span>
10681 <span class="kn">from</span> <span class="nn">productivity</span> <span class="k">import</span> <span class="p">(</span>
10682 <span class="n">ManagerRole</span><span class="p">,</span>
10683 <span class="n">SecretaryRole</span><span class="p">,</span>
10684 <span class="n">SalesRole</span><span class="p">,</span>
10685 <span class="n">FactoryRole</span>
10686 <span class="p">)</span>
10687
10688 <span class="k">class</span> <span class="nc">Employee</span><span class="p">:</span>
10689 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
10690 <span class="bp">self</span><span class="o">.</span><span class="n">id</span> <span class="o">=</span> <span class="nb">id</span>
10691 <span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">name</span>
10692
10693 <span class="k">class</span> <span class="nc">Manager</span><span class="p">(</span><span class="n">Employee</span><span class="p">,</span> <span class="n">ManagerRole</span><span class="p">,</span> <span class="n">SalaryPolicy</span><span class="p">):</span>
10694 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">weekly_salary</span><span class="p">):</span>
10695 <span class="n">SalaryPolicy</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">weekly_salary</span><span class="p">)</span>
10696 <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span>
10697
10698 <span class="k">class</span> <span class="nc">Secretary</span><span class="p">(</span><span class="n">Employee</span><span class="p">,</span> <span class="n">SecretaryRole</span><span class="p">,</span> <span class="n">SalaryPolicy</span><span class="p">):</span>
10699 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">weekly_salary</span><span class="p">):</span>
10700 <span class="n">SalaryPolicy</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">weekly_salary</span><span class="p">)</span>
10701 <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span>
10702
10703 <span class="k">class</span> <span class="nc">SalesPerson</span><span class="p">(</span><span class="n">Employee</span><span class="p">,</span> <span class="n">SalesRole</span><span class="p">,</span> <span class="n">CommissionPolicy</span><span class="p">):</span>
10704 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">weekly_salary</span><span class="p">,</span> <span class="n">commission</span><span class="p">):</span>
10705 <span class="n">CommissionPolicy</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">weekly_salary</span><span class="p">,</span> <span class="n">commission</span><span class="p">)</span>
10706 <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span>
10707
10708 <span class="k">class</span> <span class="nc">FactoryWorker</span><span class="p">(</span><span class="n">Employee</span><span class="p">,</span> <span class="n">FactoryRole</span><span class="p">,</span> <span class="n">HourlyPolicy</span><span class="p">):</span>
10709 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">hours_worked</span><span class="p">,</span> <span class="n">hour_rate</span><span class="p">):</span>
10710 <span class="n">HourlyPolicy</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hours_worked</span><span class="p">,</span> <span class="n">hour_rate</span><span class="p">)</span>
10711 <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span>
10712
10713 <span class="k">class</span> <span class="nc">TemporarySecretary</span><span class="p">(</span><span class="n">Employee</span><span class="p">,</span> <span class="n">SecretaryRole</span><span class="p">,</span> <span class="n">HourlyPolicy</span><span class="p">):</span>
10714 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">hours_worked</span><span class="p">,</span> <span class="n">hour_rate</span><span class="p">):</span>
10715 <span class="n">HourlyPolicy</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hours_worked</span><span class="p">,</span> <span class="n">hour_rate</span><span class="p">)</span>
10716 <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span>
10717 </pre></div>
10718
10719 <p>The <code>employees</code> module imports policies and roles from the other modules and implements the different <code>Employee</code> types. You are still using multiple inheritance to inherit the implementation of the salary policy classes and the productivity roles, but the implementation of each class only needs to deal with initialization.</p>
10720 <p>Notice that you still need to explicitly initialize the salary policies in the constructors. You probably saw that the initializations of <code>Manager</code> and <code>Secretary</code> are identical. Also, the initializations of <code>FactoryWorker</code> and <code>TemporarySecretary</code> are the same.</p>
10721 <p>You will not want to have this kind of code duplication in more complex designs, so you have to be careful when designing class hierarchies.</p>
10722 <p>Here’s the UML diagram for the new design:</p>
10723 <p><a href="https://files.realpython.com/media/ic-inheritance-policies.0a0de2d42a25.jpg" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/ic-inheritance-policies.0a0de2d42a25.jpg" width="1221" height="1030" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/ic-inheritance-policies.0a0de2d42a25.jpg&w=305&sig=b0ddba71545417031c5f11c94ea9b3bd5870e6fb 305w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/ic-inheritance-policies.0a0de2d42a25.jpg&w=610&sig=eb49f48a49342bc70a46148ca0a780d517d69f39 610w, https://files.realpython.com/media/ic-inheritance-policies.0a0de2d42a25.jpg 1221w" sizes="75vw" alt="Policy based design using multiple inheritance"/></a></p>
10724 <p>The diagram shows the relationships to define the <code>Secretary</code> and <code>TemporarySecretary</code> using multiple inheritance, but avoiding the diamond problem.</p>
10725 <p>You can run the program and see how it works:</p>
10726 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python program.py
10727
10728 <span class="go">Tracking Employee Productivity</span>
10729 <span class="go">==============================</span>
10730 <span class="go">Mary Poppins: screams and yells for 40 hours.</span>
10731 <span class="go">John Smith: expends 40 hours doing office paperwork.</span>
10732 <span class="go">Kevin Bacon: expends 40 hours on the phone.</span>
10733 <span class="go">Jane Doe: manufactures gadgets for 40 hours.</span>
10734 <span class="go">Robin Williams: expends 40 hours doing office paperwork.</span>
10735
10736 <span class="go">Calculating Payroll</span>
10737 <span class="go">===================</span>
10738 <span class="go">Payroll for: 1 - Mary Poppins</span>
10739 <span class="go">- Check amount: 3000</span>
10740
10741 <span class="go">Payroll for: 2 - John Smith</span>
10742 <span class="go">- Check amount: 1500</span>
10743
10744 <span class="go">Payroll for: 3 - Kevin Bacon</span>
10745 <span class="go">- Check amount: 1250</span>
10746
10747 <span class="go">Payroll for: 4 - Jane Doe</span>
10748 <span class="go">- Check amount: 600</span>
10749
10750 <span class="go">Payroll for: 5 - Robin Williams</span>
10751 <span class="go">- Check amount: 360</span>
10752 </pre></div>
10753
10754 <p>You’ve seen how inheritance and multiple inheritance work in Python. You can now explore the topic of composition.</p>
10755 <h2 id="composition-in-python">Composition in Python</h2>
10756 <p><strong>Composition</strong> is an object oriented design concept that models a <strong>has a</strong> relationship. In composition, a class known as <strong>composite</strong> contains an object of another class known to as <strong>component</strong>. In other words, a composite class <strong>has a</strong> component of another class.</p>
10757 <p>Composition allows composite classes to reuse the implementation of the components it contains. The composite class doesn’t inherit the component class interface, but it can leverage its implementation.</p>
10758 <p>The composition relation between two classes is considered loosely coupled. That means that changes to the component class rarely affect the composite class, and changes to the composite class never affect the component class.</p>
10759 <p>This provides better adaptability to change and allows applications to introduce new requirements without affecting existing code.</p>
10760 <p>When looking at two competing software designs, one based on inheritance and another based on composition, the composition solution usually is the most flexible. You can now look at how composition works.</p>
10761 <p>You’ve already used composition in our examples. If you look at the <code>Employee</code> class, you’ll see that it contains two attributes:</p>
10762 <ol>
10763 <li><strong><code>id</code></strong> to identify an employee.</li>
10764 <li><strong><code>name</code></strong> to contain the name of the employee.</li>
10765 </ol>
10766 <p>These two attributes are objects that the <code>Employee</code> class has. Therefore, you can say that an <code>Employee</code> <strong>has an</strong> <code>id</code> and <strong>has a</strong> name.</p>
10767 <p>Another attribute for an <code>Employee</code> might be an <code>Address</code>:</p>
10768 <div class="highlight python"><pre><span></span><span class="c1"># In contacts.py</span>
10769
10770 <span class="k">class</span> <span class="nc">Address</span><span class="p">:</span>
10771 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">street</span><span class="p">,</span> <span class="n">city</span><span class="p">,</span> <span class="n">state</span><span class="p">,</span> <span class="n">zipcode</span><span class="p">,</span> <span class="n">street2</span><span class="o">=</span><span class="s1">''</span><span class="p">):</span>
10772 <span class="bp">self</span><span class="o">.</span><span class="n">street</span> <span class="o">=</span> <span class="n">street</span>
10773 <span class="bp">self</span><span class="o">.</span><span class="n">street2</span> <span class="o">=</span> <span class="n">street2</span>
10774 <span class="bp">self</span><span class="o">.</span><span class="n">city</span> <span class="o">=</span> <span class="n">city</span>
10775 <span class="bp">self</span><span class="o">.</span><span class="n">state</span> <span class="o">=</span> <span class="n">state</span>
10776 <span class="bp">self</span><span class="o">.</span><span class="n">zipcode</span> <span class="o">=</span> <span class="n">zipcode</span>
10777
10778 <span class="k">def</span> <span class="nf">__str__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
10779 <span class="n">lines</span> <span class="o">=</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">street</span><span class="p">]</span>
10780 <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">street2</span><span class="p">:</span>
10781 <span class="n">lines</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">street2</span><span class="p">)</span>
10782 <span class="n">lines</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">f</span><span class="s1">'</span><span class="si">{self.city}</span><span class="s1">, </span><span class="si">{self.state}</span><span class="s1"> </span><span class="si">{self.zipcode}</span><span class="s1">'</span><span class="p">)</span>
10783 <span class="k">return</span> <span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">lines</span><span class="p">)</span>
10784 </pre></div>
10785
10786 <p>You implemented a basic address class that contains the usual components for an address. You made the <code>street2</code> attribute optional because not all addresses will have that component.</p>
10787 <p>You implemented <code>__str__()</code> to provide a pretty representation of an <code>Address</code>. You can see this implementation in the interactive interpreter:</p>
10788 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">contacts</span> <span class="k">import</span> <span class="n">Address</span>
10789 <span class="gp">>>> </span><span class="n">address</span> <span class="o">=</span> <span class="n">Address</span><span class="p">(</span><span class="s1">'55 Main St.'</span><span class="p">,</span> <span class="s1">'Concord'</span><span class="p">,</span> <span class="s1">'NH'</span><span class="p">,</span> <span class="s1">'03301'</span><span class="p">)</span>
10790 <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">address</span><span class="p">)</span>
10791
10792 <span class="go">55 Main St.</span>
10793 <span class="go">Concord, NH 03301</span>
10794 </pre></div>
10795
10796 <p>When you <code>print()</code> the <code>address</code> variable, the special method <code>__str__()</code> is invoked. Since you overloaded the method to return a string formatted as an address, you get a nice, readable representation. <a href="https://realpython.com/operator-function-overloading/">Operator and Function Overloading in Custom Python Classes</a> gives a good overview of the special methods available in classes that can be implemented to customize the behavior of your objects.</p>
10797 <p>You can now add the <code>Address</code> to the <code>Employee</code> class through composition:</p>
10798 <div class="highlight python"><pre><span></span><span class="c1"># In employees.py</span>
10799
10800 <span class="k">class</span> <span class="nc">Employee</span><span class="p">:</span>
10801 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
10802 <span class="bp">self</span><span class="o">.</span><span class="n">id</span> <span class="o">=</span> <span class="nb">id</span>
10803 <span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">name</span>
10804 <span class="bp">self</span><span class="o">.</span><span class="n">address</span> <span class="o">=</span> <span class="kc">None</span>
10805 </pre></div>
10806
10807 <p>You initialize the <code>address</code> attribute to <code>None</code> for now to make it optional, but by doing that, you can now assign an <code>Address</code> to an <code>Employee</code>. Also notice that there is no reference in the <code>employee</code> module to the <code>contacts</code> module.</p>
10808 <p>Composition is a loosely coupled relationship that often doesn’t require the composite class to have knowledge of the component.</p>
10809 <p>The UML diagram representing the relationship between <code>Employee</code> and <code>Address</code> looks like this:</p>
10810 <p><a href="https://files.realpython.com/media/ic-employee-address.240e806b1101.jpg" target="_blank"><img class="img-fluid mx-auto d-block w-33" src="https://files.realpython.com/media/ic-employee-address.240e806b1101.jpg" width="284" height="469" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/ic-employee-address.240e806b1101.jpg&w=71&sig=871e9fce940fb84544980c474624a07c5d52b8e8 71w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/ic-employee-address.240e806b1101.jpg&w=142&sig=658578725c830eb5554ba1e349b27a198c2aa7e9 142w, https://files.realpython.com/media/ic-employee-address.240e806b1101.jpg 284w" sizes="75vw" alt="Composition example with Employee containing Address"/></a></p>
10811 <p>The diagram shows the basic composition relationship between <code>Employee</code> and <code>Address</code>.</p>
10812 <p>You can now modify the <code>PayrollSystem</code> class to leverage the <code>address</code> attribute in <code>Employee</code>:</p>
10813 <div class="highlight python"><pre><span></span><span class="c1"># In hr.py</span>
10814
10815 <span class="k">class</span> <span class="nc">PayrollSystem</span><span class="p">:</span>
10816 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">employees</span><span class="p">):</span>
10817 <span class="nb">print</span><span class="p">(</span><span class="s1">'Calculating Payroll'</span><span class="p">)</span>
10818 <span class="nb">print</span><span class="p">(</span><span class="s1">'==================='</span><span class="p">)</span>
10819 <span class="k">for</span> <span class="n">employee</span> <span class="ow">in</span> <span class="n">employees</span><span class="p">:</span>
10820 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Payroll for: </span><span class="si">{employee.id}</span><span class="s1"> - </span><span class="si">{employee.name}</span><span class="s1">'</span><span class="p">)</span>
10821 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'- Check amount: {employee.calculate_payroll()}'</span><span class="p">)</span>
10822 <span class="hll"> <span class="k">if</span> <span class="n">employee</span><span class="o">.</span><span class="n">address</span><span class="p">:</span>
10823 </span><span class="hll"> <span class="nb">print</span><span class="p">(</span><span class="s1">'- Sent to:'</span><span class="p">)</span>
10824 </span><span class="hll"> <span class="nb">print</span><span class="p">(</span><span class="n">employee</span><span class="o">.</span><span class="n">address</span><span class="p">)</span>
10825 </span> <span class="nb">print</span><span class="p">(</span><span class="s1">''</span><span class="p">)</span>
10826 </pre></div>
10827
10828 <p>You check to see if the <code>employee</code> object has an address, and if it does, you print it. You can now modify the program to assign some addresses to the employees:</p>
10829 <div class="highlight python"><pre><span></span><span class="c1"># In program.py</span>
10830
10831 <span class="kn">import</span> <span class="nn">hr</span>
10832 <span class="kn">import</span> <span class="nn">employees</span>
10833 <span class="kn">import</span> <span class="nn">productivity</span>
10834 <span class="kn">import</span> <span class="nn">contacts</span>
10835
10836 <span class="n">manager</span> <span class="o">=</span> <span class="n">employees</span><span class="o">.</span><span class="n">Manager</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="s1">'Mary Poppins'</span><span class="p">,</span> <span class="mi">3000</span><span class="p">)</span>
10837 <span class="n">manager</span><span class="o">.</span><span class="n">address</span> <span class="o">=</span> <span class="n">contacts</span><span class="o">.</span><span class="n">Address</span><span class="p">(</span>
10838 <span class="s1">'121 Admin Rd'</span><span class="p">,</span>
10839 <span class="s1">'Concord'</span><span class="p">,</span>
10840 <span class="s1">'NH'</span><span class="p">,</span>
10841 <span class="s1">'03301'</span>
10842 <span class="p">)</span>
10843 <span class="n">secretary</span> <span class="o">=</span> <span class="n">employees</span><span class="o">.</span><span class="n">Secretary</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="s1">'John Smith'</span><span class="p">,</span> <span class="mi">1500</span><span class="p">)</span>
10844 <span class="n">secretary</span><span class="o">.</span><span class="n">address</span> <span class="o">=</span> <span class="n">contacts</span><span class="o">.</span><span class="n">Address</span><span class="p">(</span>
10845 <span class="s1">'67 Paperwork Ave.'</span><span class="p">,</span>
10846 <span class="s1">'Manchester'</span><span class="p">,</span>
10847 <span class="s1">'NH'</span><span class="p">,</span>
10848 <span class="s1">'03101'</span>
10849 <span class="p">)</span>
10850 <span class="n">sales_guy</span> <span class="o">=</span> <span class="n">employees</span><span class="o">.</span><span class="n">SalesPerson</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="s1">'Kevin Bacon'</span><span class="p">,</span> <span class="mi">1000</span><span class="p">,</span> <span class="mi">250</span><span class="p">)</span>
10851 <span class="n">factory_worker</span> <span class="o">=</span> <span class="n">employees</span><span class="o">.</span><span class="n">FactoryWorker</span><span class="p">(</span><span class="mi">4</span><span class="p">,</span> <span class="s1">'Jane Doe'</span><span class="p">,</span> <span class="mi">40</span><span class="p">,</span> <span class="mi">15</span><span class="p">)</span>
10852 <span class="n">temporary_secretary</span> <span class="o">=</span> <span class="n">employees</span><span class="o">.</span><span class="n">TemporarySecretary</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="s1">'Robin Williams'</span><span class="p">,</span> <span class="mi">40</span><span class="p">,</span> <span class="mi">9</span><span class="p">)</span>
10853 <span class="n">employees</span> <span class="o">=</span> <span class="p">[</span>
10854 <span class="n">manager</span><span class="p">,</span>
10855 <span class="n">secretary</span><span class="p">,</span>
10856 <span class="n">sales_guy</span><span class="p">,</span>
10857 <span class="n">factory_worker</span><span class="p">,</span>
10858 <span class="n">temporary_secretary</span><span class="p">,</span>
10859 <span class="p">]</span>
10860 <span class="n">productivity_system</span> <span class="o">=</span> <span class="n">productivity</span><span class="o">.</span><span class="n">ProductivitySystem</span><span class="p">()</span>
10861 <span class="n">productivity_system</span><span class="o">.</span><span class="n">track</span><span class="p">(</span><span class="n">employees</span><span class="p">,</span> <span class="mi">40</span><span class="p">)</span>
10862 <span class="n">payroll_system</span> <span class="o">=</span> <span class="n">hr</span><span class="o">.</span><span class="n">PayrollSystem</span><span class="p">()</span>
10863 <span class="n">payroll_system</span><span class="o">.</span><span class="n">calculate_payroll</span><span class="p">(</span><span class="n">employees</span><span class="p">)</span>
10864 </pre></div>
10865
10866 <p>You added a couple of addresses to the <code>manager</code> and <code>secretary</code> objects. When you run the program, you will see the addresses printed:</p>
10867 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python program.py
10868
10869 <span class="go">Tracking Employee Productivity</span>
10870 <span class="go">==============================</span>
10871 <span class="go">Mary Poppins: screams and yells for {hours} hours.</span>
10872 <span class="go">John Smith: expends {hours} hours doing office paperwork.</span>
10873 <span class="go">Kevin Bacon: expends {hours} hours on the phone.</span>
10874 <span class="go">Jane Doe: manufactures gadgets for {hours} hours.</span>
10875 <span class="go">Robin Williams: expends {hours} hours doing office paperwork.</span>
10876
10877 <span class="go">Calculating Payroll</span>
10878 <span class="go">===================</span>
10879 <span class="go">Payroll for: 1 - Mary Poppins</span>
10880 <span class="go">- Check amount: 3000</span>
10881 <span class="go">- Sent to:</span>
10882 <span class="go">121 Admin Rd</span>
10883 <span class="go">Concord, NH 03301</span>
10884
10885 <span class="go">Payroll for: 2 - John Smith</span>
10886 <span class="go">- Check amount: 1500</span>
10887 <span class="go">- Sent to:</span>
10888 <span class="go">67 Paperwork Ave.</span>
10889 <span class="go">Manchester, NH 03101</span>
10890
10891 <span class="go">Payroll for: 3 - Kevin Bacon</span>
10892 <span class="go">- Check amount: 1250</span>
10893
10894 <span class="go">Payroll for: 4 - Jane Doe</span>
10895 <span class="go">- Check amount: 600</span>
10896
10897 <span class="go">Payroll for: 5 - Robin Williams</span>
10898 <span class="go">- Check amount: 360</span>
10899 </pre></div>
10900
10901 <p>Notice how the payroll output for the <code>manager</code> and <code>secretary</code> objects show the addresses where the checks were sent.</p>
10902 <p>The <code>Employee</code> class leverages the implementation of the <code>Address</code> class without any knowledge of what an <code>Address</code> object is or how it’s represented. This type of design is so flexible that you can change the <code>Address</code> class without any impact to the <code>Employee</code> class.</p>
10903 <h3 id="flexible-designs-with-composition">Flexible Designs With Composition</h3>
10904 <p>Composition is more flexible than inheritance because it models a loosely coupled relationship. Changes to a component class have minimal or no effects on the composite class. Designs based on composition are more suitable to change.</p>
10905 <p>You change behavior by providing new components that implement those behaviors instead of adding new classes to your hierarchy.</p>
10906 <p>Take a look at the multiple inheritance example above. Imagine how new payroll policies will affect the design. Try to picture what the class hierarchy will look like if new roles are needed. As you saw before, relying too heavily on inheritance can lead to class explosion.</p>
10907 <p>The biggest problem is not so much the number of classes in your design, but how tightly coupled the relationships between those classes are. Tightly coupled classes affect each other when changes are introduced.</p>
10908 <p>In this section, you are going to use composition to implement a better design that still fits the requirements of the <code>PayrollSystem</code> and the <code>ProductivitySystem</code>.</p>
10909 <p>You can start by implementing the functionality of the <code>ProductivitySystem</code>:</p>
10910 <div class="highlight python"><pre><span></span><span class="c1"># In productivity.py</span>
10911
10912 <span class="k">class</span> <span class="nc">ProductivitySystem</span><span class="p">:</span>
10913 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
10914 <span class="bp">self</span><span class="o">.</span><span class="n">_roles</span> <span class="o">=</span> <span class="p">{</span>
10915 <span class="s1">'manager'</span><span class="p">:</span> <span class="n">ManagerRole</span><span class="p">,</span>
10916 <span class="s1">'secretary'</span><span class="p">:</span> <span class="n">SecretaryRole</span><span class="p">,</span>
10917 <span class="s1">'sales'</span><span class="p">:</span> <span class="n">SalesRole</span><span class="p">,</span>
10918 <span class="s1">'factory'</span><span class="p">:</span> <span class="n">FactoryRole</span><span class="p">,</span>
10919 <span class="p">}</span>
10920
10921 <span class="k">def</span> <span class="nf">get_role</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">role_id</span><span class="p">):</span>
10922 <span class="n">role_type</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_roles</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">role_id</span><span class="p">)</span>
10923 <span class="k">if</span> <span class="ow">not</span> <span class="n">role_type</span><span class="p">:</span>
10924 <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'role_id'</span><span class="p">)</span>
10925 <span class="k">return</span> <span class="n">role_type</span><span class="p">()</span>
10926
10927 <span class="k">def</span> <span class="nf">track</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">employees</span><span class="p">,</span> <span class="n">hours</span><span class="p">):</span>
10928 <span class="nb">print</span><span class="p">(</span><span class="s1">'Tracking Employee Productivity'</span><span class="p">)</span>
10929 <span class="nb">print</span><span class="p">(</span><span class="s1">'=============================='</span><span class="p">)</span>
10930 <span class="k">for</span> <span class="n">employee</span> <span class="ow">in</span> <span class="n">employees</span><span class="p">:</span>
10931 <span class="n">employee</span><span class="o">.</span><span class="n">work</span><span class="p">(</span><span class="n">hours</span><span class="p">)</span>
10932 <span class="nb">print</span><span class="p">(</span><span class="s1">''</span><span class="p">)</span>
10933 </pre></div>
10934
10935 <p>The <code>ProductivitySystem</code> class defines some roles using a string identifier mapped to a role class that implements the role. It exposes a <code>.get_role()</code> method that, given a role identifier, returns the role type object. If the role is not found, then a <code>ValueError</code> exception is raised.</p>
10936 <p>It also exposes the previous functionality in the <code>.track()</code> method, where given a list of employees it tracks the productivity of those employees.</p>
10937 <p>You can now implement the different role classes:</p>
10938 <div class="highlight python"><pre><span></span><span class="c1"># In productivity.py</span>
10939
10940 <span class="k">class</span> <span class="nc">ManagerRole</span><span class="p">:</span>
10941 <span class="k">def</span> <span class="nf">perform_duties</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hours</span><span class="p">):</span>
10942 <span class="k">return</span> <span class="n">f</span><span class="s1">'screams and yells for </span><span class="si">{hours}</span><span class="s1"> hours.'</span>
10943
10944 <span class="k">class</span> <span class="nc">SecretaryRole</span><span class="p">:</span>
10945 <span class="k">def</span> <span class="nf">perform_duties</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hours</span><span class="p">):</span>
10946 <span class="k">return</span> <span class="n">f</span><span class="s1">'does paperwork for </span><span class="si">{hours}</span><span class="s1"> hours.'</span>
10947
10948 <span class="k">class</span> <span class="nc">SalesRole</span><span class="p">:</span>
10949 <span class="k">def</span> <span class="nf">perform_duties</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hours</span><span class="p">):</span>
10950 <span class="k">return</span> <span class="n">f</span><span class="s1">'expends </span><span class="si">{hours}</span><span class="s1"> hours on the phone.'</span>
10951
10952 <span class="k">class</span> <span class="nc">FactoryRole</span><span class="p">:</span>
10953 <span class="k">def</span> <span class="nf">perform_duties</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hours</span><span class="p">):</span>
10954 <span class="k">return</span> <span class="n">f</span><span class="s1">'manufactures gadgets for </span><span class="si">{hours}</span><span class="s1"> hours.'</span>
10955 </pre></div>
10956
10957 <p>Each of the roles you implemented expose a <code>.perform_duties()</code> that takes the number of <code>hours</code> worked. The methods return a string representing the duties.</p>
10958 <p>The role classes are independent of each other, but they expose the same interface, so they are interchangeable. You’ll see later how they are used in the application.</p>
10959 <p>Now, you can implement the <code>PayrollSystem</code> for the application:</p>
10960 <div class="highlight python"><pre><span></span><span class="c1"># In hr.py</span>
10961
10962 <span class="k">class</span> <span class="nc">PayrollSystem</span><span class="p">:</span>
10963 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
10964 <span class="bp">self</span><span class="o">.</span><span class="n">_employee_policies</span> <span class="o">=</span> <span class="p">{</span>
10965 <span class="mi">1</span><span class="p">:</span> <span class="n">SalaryPolicy</span><span class="p">(</span><span class="mi">3000</span><span class="p">),</span>
10966 <span class="mi">2</span><span class="p">:</span> <span class="n">SalaryPolicy</span><span class="p">(</span><span class="mi">1500</span><span class="p">),</span>
10967 <span class="mi">3</span><span class="p">:</span> <span class="n">CommissionPolicy</span><span class="p">(</span><span class="mi">1000</span><span class="p">,</span> <span class="mi">100</span><span class="p">),</span>
10968 <span class="mi">4</span><span class="p">:</span> <span class="n">HourlyPolicy</span><span class="p">(</span><span class="mi">15</span><span class="p">),</span>
10969 <span class="mi">5</span><span class="p">:</span> <span class="n">HourlyPolicy</span><span class="p">(</span><span class="mi">9</span><span class="p">)</span>
10970 <span class="p">}</span>
10971
10972 <span class="k">def</span> <span class="nf">get_policy</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">employee_id</span><span class="p">):</span>
10973 <span class="n">policy</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_employee_policies</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">employee_id</span><span class="p">)</span>
10974 <span class="k">if</span> <span class="ow">not</span> <span class="n">policy</span><span class="p">:</span>
10975 <span class="k">return</span> <span class="ne">ValueError</span><span class="p">(</span><span class="n">employee_id</span><span class="p">)</span>
10976 <span class="k">return</span> <span class="n">policy</span>
10977
10978 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">employees</span><span class="p">):</span>
10979 <span class="nb">print</span><span class="p">(</span><span class="s1">'Calculating Payroll'</span><span class="p">)</span>
10980 <span class="nb">print</span><span class="p">(</span><span class="s1">'==================='</span><span class="p">)</span>
10981 <span class="k">for</span> <span class="n">employee</span> <span class="ow">in</span> <span class="n">employees</span><span class="p">:</span>
10982 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Payroll for: </span><span class="si">{employee.id}</span><span class="s1"> - </span><span class="si">{employee.name}</span><span class="s1">'</span><span class="p">)</span>
10983 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'- Check amount: {employee.calculate_payroll()}'</span><span class="p">)</span>
10984 <span class="k">if</span> <span class="n">employee</span><span class="o">.</span><span class="n">address</span><span class="p">:</span>
10985 <span class="nb">print</span><span class="p">(</span><span class="s1">'- Sent to:'</span><span class="p">)</span>
10986 <span class="nb">print</span><span class="p">(</span><span class="n">employee</span><span class="o">.</span><span class="n">address</span><span class="p">)</span>
10987 <span class="nb">print</span><span class="p">(</span><span class="s1">''</span><span class="p">)</span>
10988 </pre></div>
10989
10990 <p>The <code>PayrollSystem</code> keeps an internal database of payroll policies for each employee. It exposes a <code>.get_policy()</code> that, given an employee <code>id</code>, returns its payroll policy. If a specified <code>id</code> doesn’t exist in the system, then the method raises a <code>ValueError</code> exception.</p>
10991 <p>The implementation of <code>.calculate_payroll()</code> works the same as before. It takes a list of employees, calculates the payroll, and prints the results.</p>
10992 <p>You can now implement the payroll policy classes:</p>
10993 <div class="highlight python"><pre><span></span><span class="c1"># In hr.py</span>
10994
10995 <span class="k">class</span> <span class="nc">PayrollPolicy</span><span class="p">:</span>
10996 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
10997 <span class="bp">self</span><span class="o">.</span><span class="n">hours_worked</span> <span class="o">=</span> <span class="mi">0</span>
10998
10999 <span class="k">def</span> <span class="nf">track_work</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hours</span><span class="p">):</span>
11000 <span class="bp">self</span><span class="o">.</span><span class="n">hours_worked</span> <span class="o">+=</span> <span class="n">hours</span>
11001
11002 <span class="k">class</span> <span class="nc">SalaryPolicy</span><span class="p">(</span><span class="n">PayrollPolicy</span><span class="p">):</span>
11003 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">weekly_salary</span><span class="p">):</span>
11004 <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span>
11005 <span class="bp">self</span><span class="o">.</span><span class="n">weekly_salary</span> <span class="o">=</span> <span class="n">weekly_salary</span>
11006
11007 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
11008 <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">weekly_salary</span>
11009
11010 <span class="k">class</span> <span class="nc">HourlyPolicy</span><span class="p">(</span><span class="n">PayrollPolicy</span><span class="p">):</span>
11011 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hour_rate</span><span class="p">):</span>
11012 <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span>
11013 <span class="bp">self</span><span class="o">.</span><span class="n">hour_rate</span> <span class="o">=</span> <span class="n">hour_rate</span>
11014
11015 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
11016 <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">hours_worked</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">hour_rate</span>
11017
11018 <span class="k">class</span> <span class="nc">CommissionPolicy</span><span class="p">(</span><span class="n">SalaryPolicy</span><span class="p">):</span>
11019 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">weekly_salary</span><span class="p">,</span> <span class="n">commission_per_sale</span><span class="p">):</span>
11020 <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">weekly_salary</span><span class="p">)</span>
11021 <span class="bp">self</span><span class="o">.</span><span class="n">commission_per_sale</span> <span class="o">=</span> <span class="n">commission_per_sale</span>
11022
11023 <span class="nd">@property</span>
11024 <span class="k">def</span> <span class="nf">commission</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
11025 <span class="n">sales</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">hours_worked</span> <span class="o">/</span> <span class="mi">5</span>
11026 <span class="k">return</span> <span class="n">sales</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">commission_per_sale</span>
11027
11028 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
11029 <span class="n">fixed</span> <span class="o">=</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">calculate_payroll</span><span class="p">()</span>
11030 <span class="k">return</span> <span class="n">fixed</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">commission</span>
11031 </pre></div>
11032
11033 <p>You first implement a <code>PayrollPolicy</code> class that serves as a base class for all the payroll policies. This class tracks the <code>hours_worked</code>, which is common to all payroll policies.</p>
11034 <p>The other policy classes derive from <code>PayrollPolicy</code>. We use inheritance here because we want to leverage the implementation of <code>PayrollPolicy</code>. Also, <code>SalaryPolicy</code>, <code>HourlyPolicy</code>, and <code>CommissionPolicy</code> <strong>are a</strong> <code>PayrollPolicy</code>.</p>
11035 <p><code>SalaryPolicy</code> is initialized with a <code>weekly_salary</code> value that is then used in <code>.calculate_payroll()</code>. <code>HourlyPolicy</code> is initialized with the <code>hour_rate</code>, and implements <code>.calculate_payroll()</code> by leveraging the base class <code>hours_worked</code>.</p>
11036 <p>The <code>CommissionPolicy</code> class derives from <code>SalaryPolicy</code> because it wants to inherit its implementation. It is initialized with the <code>weekly_salary</code> parameters, but it also requires a <code>commission_per_sale</code> parameter.</p>
11037 <p>The <code>commission_per_sale</code> is used to calculate the <code>.commission</code>, which is implemented as a property so it gets calculated when requested. In the example, we are assuming that a sale happens every 5 hours worked, and the <code>.commission</code> is the number of sales times the <code>commission_per_sale</code> value.</p>
11038 <p><code>CommissionPolicy</code> implements the <code>.calculate_payroll()</code> method by first leveraging the implementation in <code>SalaryPolicy</code> and then adding the calculated commission.</p>
11039 <p>You can now add an <code>AddressBook</code> class to manage employee addresses:</p>
11040 <div class="highlight python"><pre><span></span><span class="c1"># In contacts.py</span>
11041
11042 <span class="k">class</span> <span class="nc">AddressBook</span><span class="p">:</span>
11043 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
11044 <span class="bp">self</span><span class="o">.</span><span class="n">_employee_addresses</span> <span class="o">=</span> <span class="p">{</span>
11045 <span class="mi">1</span><span class="p">:</span> <span class="n">Address</span><span class="p">(</span><span class="s1">'121 Admin Rd.'</span><span class="p">,</span> <span class="s1">'Concord'</span><span class="p">,</span> <span class="s1">'NH'</span><span class="p">,</span> <span class="s1">'03301'</span><span class="p">),</span>
11046 <span class="mi">2</span><span class="p">:</span> <span class="n">Address</span><span class="p">(</span><span class="s1">'67 Paperwork Ave'</span><span class="p">,</span> <span class="s1">'Manchester'</span><span class="p">,</span> <span class="s1">'NH'</span><span class="p">,</span> <span class="s1">'03101'</span><span class="p">),</span>
11047 <span class="mi">3</span><span class="p">:</span> <span class="n">Address</span><span class="p">(</span><span class="s1">'15 Rose St'</span><span class="p">,</span> <span class="s1">'Concord'</span><span class="p">,</span> <span class="s1">'NH'</span><span class="p">,</span> <span class="s1">'03301'</span><span class="p">,</span> <span class="s1">'Apt. B-1'</span><span class="p">),</span>
11048 <span class="mi">4</span><span class="p">:</span> <span class="n">Address</span><span class="p">(</span><span class="s1">'39 Sole St.'</span><span class="p">,</span> <span class="s1">'Concord'</span><span class="p">,</span> <span class="s1">'NH'</span><span class="p">,</span> <span class="s1">'03301'</span><span class="p">),</span>
11049 <span class="mi">5</span><span class="p">:</span> <span class="n">Address</span><span class="p">(</span><span class="s1">'99 Mountain Rd.'</span><span class="p">,</span> <span class="s1">'Concord'</span><span class="p">,</span> <span class="s1">'NH'</span><span class="p">,</span> <span class="s1">'03301'</span><span class="p">),</span>
11050 <span class="p">}</span>
11051
11052 <span class="k">def</span> <span class="nf">get_employee_address</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">employee_id</span><span class="p">):</span>
11053 <span class="n">address</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_employee_addresses</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">employee_id</span><span class="p">)</span>
11054 <span class="k">if</span> <span class="ow">not</span> <span class="n">address</span><span class="p">:</span>
11055 <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="n">employee_id</span><span class="p">)</span>
11056 <span class="k">return</span> <span class="n">address</span>
11057 </pre></div>
11058
11059 <p>The <code>AddressBook</code> class keeps an internal database of <code>Address</code> objects for each employee. It exposes a <code>get_employee_address()</code> method that returns the address of the specified employee <code>id</code>. If the employee <code>id</code> doesn’t exist, then it raises a <code>ValueError</code>.</p>
11060 <p>The <code>Address</code> class implementation remains the same as before:</p>
11061 <div class="highlight python"><pre><span></span><span class="c1"># In contacts.py</span>
11062
11063 <span class="k">class</span> <span class="nc">Address</span><span class="p">:</span>
11064 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">street</span><span class="p">,</span> <span class="n">city</span><span class="p">,</span> <span class="n">state</span><span class="p">,</span> <span class="n">zipcode</span><span class="p">,</span> <span class="n">street2</span><span class="o">=</span><span class="s1">''</span><span class="p">):</span>
11065 <span class="bp">self</span><span class="o">.</span><span class="n">street</span> <span class="o">=</span> <span class="n">street</span>
11066 <span class="bp">self</span><span class="o">.</span><span class="n">street2</span> <span class="o">=</span> <span class="n">street2</span>
11067 <span class="bp">self</span><span class="o">.</span><span class="n">city</span> <span class="o">=</span> <span class="n">city</span>
11068 <span class="bp">self</span><span class="o">.</span><span class="n">state</span> <span class="o">=</span> <span class="n">state</span>
11069 <span class="bp">self</span><span class="o">.</span><span class="n">zipcode</span> <span class="o">=</span> <span class="n">zipcode</span>
11070
11071 <span class="k">def</span> <span class="nf">__str__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
11072 <span class="n">lines</span> <span class="o">=</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">street</span><span class="p">]</span>
11073 <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">street2</span><span class="p">:</span>
11074 <span class="n">lines</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">street2</span><span class="p">)</span>
11075 <span class="n">lines</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">f</span><span class="s1">'</span><span class="si">{self.city}</span><span class="s1">, </span><span class="si">{self.state}</span><span class="s1"> </span><span class="si">{self.zipcode}</span><span class="s1">'</span><span class="p">)</span>
11076 <span class="k">return</span> <span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">lines</span><span class="p">)</span>
11077 </pre></div>
11078
11079 <p>The class manages the address components and provides a pretty representation of an address.</p>
11080 <p>So far, the new classes have been extended to support more functionality, but there are no significant changes to the previous design. This is going to change with the design of the <code>employees</code> module and its classes.</p>
11081 <p>You can start by implementing an <code>EmployeeDatabase</code> class:</p>
11082 <div class="highlight python"><pre><span></span><span class="c1"># In employees.py</span>
11083
11084 <span class="kn">from</span> <span class="nn">productivity</span> <span class="k">import</span> <span class="n">ProductivitySystem</span>
11085 <span class="kn">from</span> <span class="nn">hr</span> <span class="k">import</span> <span class="n">PayrollSystem</span>
11086 <span class="kn">from</span> <span class="nn">contacts</span> <span class="k">import</span> <span class="n">AddressBook</span>
11087
11088 <span class="k">class</span> <span class="nc">EmployeeDatabase</span><span class="p">:</span>
11089 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
11090 <span class="bp">self</span><span class="o">.</span><span class="n">_employees</span> <span class="o">=</span> <span class="p">[</span>
11091 <span class="p">{</span>
11092 <span class="s1">'id'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
11093 <span class="s1">'name'</span><span class="p">:</span> <span class="s1">'Mary Poppins'</span><span class="p">,</span>
11094 <span class="s1">'role'</span><span class="p">:</span> <span class="s1">'manager'</span>
11095 <span class="p">},</span>
11096 <span class="p">{</span>
11097 <span class="s1">'id'</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span>
11098 <span class="s1">'name'</span><span class="p">:</span> <span class="s1">'John Smith'</span><span class="p">,</span>
11099 <span class="s1">'role'</span><span class="p">:</span> <span class="s1">'secretary'</span>
11100 <span class="p">},</span>
11101 <span class="p">{</span>
11102 <span class="s1">'id'</span><span class="p">:</span> <span class="mi">3</span><span class="p">,</span>
11103 <span class="s1">'name'</span><span class="p">:</span> <span class="s1">'Kevin Bacon'</span><span class="p">,</span>
11104 <span class="s1">'role'</span><span class="p">:</span> <span class="s1">'sales'</span>
11105 <span class="p">},</span>
11106 <span class="p">{</span>
11107 <span class="s1">'id'</span><span class="p">:</span> <span class="mi">4</span><span class="p">,</span>
11108 <span class="s1">'name'</span><span class="p">:</span> <span class="s1">'Jane Doe'</span><span class="p">,</span>
11109 <span class="s1">'role'</span><span class="p">:</span> <span class="s1">'factory'</span>
11110 <span class="p">},</span>
11111 <span class="p">{</span>
11112 <span class="s1">'id'</span><span class="p">:</span> <span class="mi">5</span><span class="p">,</span>
11113 <span class="s1">'name'</span><span class="p">:</span> <span class="s1">'Robin Williams'</span><span class="p">,</span>
11114 <span class="s1">'role'</span><span class="p">:</span> <span class="s1">'secretary'</span>
11115 <span class="p">},</span>
11116 <span class="p">]</span>
11117 <span class="bp">self</span><span class="o">.</span><span class="n">productivity</span> <span class="o">=</span> <span class="n">ProductivitySystem</span><span class="p">()</span>
11118 <span class="bp">self</span><span class="o">.</span><span class="n">payroll</span> <span class="o">=</span> <span class="n">PayrollSystem</span><span class="p">()</span>
11119 <span class="bp">self</span><span class="o">.</span><span class="n">employee_addresses</span> <span class="o">=</span> <span class="n">AddressBook</span><span class="p">()</span>
11120
11121 <span class="nd">@property</span>
11122 <span class="k">def</span> <span class="nf">employees</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
11123 <span class="k">return</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">_create_employee</span><span class="p">(</span><span class="o">**</span><span class="n">data</span><span class="p">)</span> <span class="k">for</span> <span class="n">data</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">_employees</span><span class="p">]</span>
11124
11125 <span class="k">def</span> <span class="nf">_create_employee</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">role</span><span class="p">):</span>
11126 <span class="n">address</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">employee_addresses</span><span class="o">.</span><span class="n">get_employee_address</span><span class="p">(</span><span class="nb">id</span><span class="p">)</span>
11127 <span class="n">employee_role</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">productivity</span><span class="o">.</span><span class="n">get_role</span><span class="p">(</span><span class="n">role</span><span class="p">)</span>
11128 <span class="n">payroll_policy</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">payroll</span><span class="o">.</span><span class="n">get_policy</span><span class="p">(</span><span class="nb">id</span><span class="p">)</span>
11129 <span class="k">return</span> <span class="n">Employee</span><span class="p">(</span><span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">address</span><span class="p">,</span> <span class="n">employee_role</span><span class="p">,</span> <span class="n">payroll_policy</span><span class="p">)</span>
11130 </pre></div>
11131
11132 <p>The <code>EmployeeDatabase</code> keeps track of all the employees in the company. For each employee, it tracks the <code>id</code>, <code>name</code>, and <code>role</code>. It <strong>has an</strong> instance of the <code>ProductivitySystem</code>, the <code>PayrollSystem</code>, and the <code>AddressBook</code>. These instances are used to create employees.</p>
11133 <p>It exposes an <code>.employees</code> property that returns the list of employees. The <code>Employee</code> objects are created in an internal method <code>._create_employee()</code>. Notice that you don’t have different types of <code>Employee</code> classes. You just need to implement a single <code>Employee</code> class:</p>
11134 <div class="highlight python"><pre><span></span><span class="c1"># In employees.py</span>
11135
11136 <span class="k">class</span> <span class="nc">Employee</span><span class="p">:</span>
11137 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">address</span><span class="p">,</span> <span class="n">role</span><span class="p">,</span> <span class="n">payroll</span><span class="p">):</span>
11138 <span class="bp">self</span><span class="o">.</span><span class="n">id</span> <span class="o">=</span> <span class="nb">id</span>
11139 <span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">name</span>
11140 <span class="bp">self</span><span class="o">.</span><span class="n">address</span> <span class="o">=</span> <span class="n">address</span>
11141 <span class="bp">self</span><span class="o">.</span><span class="n">role</span> <span class="o">=</span> <span class="n">role</span>
11142 <span class="bp">self</span><span class="o">.</span><span class="n">payroll</span> <span class="o">=</span> <span class="n">payroll</span>
11143
11144 <span class="k">def</span> <span class="nf">work</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hours</span><span class="p">):</span>
11145 <span class="n">duties</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">role</span><span class="o">.</span><span class="n">perform_duties</span><span class="p">(</span><span class="n">hours</span><span class="p">)</span>
11146 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Employee </span><span class="si">{self.id}</span><span class="s1"> - </span><span class="si">{self.name}</span><span class="s1">:'</span><span class="p">)</span>
11147 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'- </span><span class="si">{duties}</span><span class="s1">'</span><span class="p">)</span>
11148 <span class="nb">print</span><span class="p">(</span><span class="s1">''</span><span class="p">)</span>
11149 <span class="bp">self</span><span class="o">.</span><span class="n">payroll</span><span class="o">.</span><span class="n">track_work</span><span class="p">(</span><span class="n">hours</span><span class="p">)</span>
11150
11151 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
11152 <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">payroll</span><span class="o">.</span><span class="n">calculate_payroll</span><span class="p">()</span>
11153 </pre></div>
11154
11155 <p>The <code>Employee</code> class is initialized with the <code>id</code>, <code>name</code>, and <code>address</code> attributes. It also requires the productivity <code>role</code> for the employee and the <code>payroll</code> policy.</p>
11156 <p>The class exposes a <code>.work()</code> method that takes the hours worked. This method first retrieves the <code>duties</code> from the <code>role</code>. In other words, it delegates to the <code>role</code> object to perform its duties.</p>
11157 <p>In the same way, it delegates to the <code>payroll</code> object to track the work <code>hours</code>. The <code>payroll</code>, as you saw, uses those hours to calculate the payroll if needed.</p>
11158 <p>The following diagram shows the composition design used:</p>
11159 <p><a href="https://files.realpython.com/media/ic-policy-based-composition.6e78bdb5824f.jpg" target="_blank"><img class="img-fluid mx-auto d-block " src="https://files.realpython.com/media/ic-policy-based-composition.6e78bdb5824f.jpg" width="1219" height="1381" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/ic-policy-based-composition.6e78bdb5824f.jpg&w=304&sig=8baa24bb7a519fc76f40f994c95c05833c8ded67 304w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/ic-policy-based-composition.6e78bdb5824f.jpg&w=609&sig=2a14268f94d82ac5f5c64d115f02f8eeea808580 609w, https://files.realpython.com/media/ic-policy-based-composition.6e78bdb5824f.jpg 1219w" sizes="75vw" alt="Policy based design using composition"/></a></p>
11160 <p>The diagram shows the design of composition based policies. There is a single <code>Employee</code> that is composed of other data objects like <code>Address</code> and depends on the <code>IRole</code> and <code>IPayrollCalculator</code> interfaces to delegate the work. There are multiple implementations of these interfaces.</p>
11161 <p>You can now use this design in your program:</p>
11162 <div class="highlight python"><pre><span></span><span class="c1"># In program.py</span>
11163
11164 <span class="kn">from</span> <span class="nn">hr</span> <span class="k">import</span> <span class="n">PayrollSystem</span>
11165 <span class="kn">from</span> <span class="nn">productivity</span> <span class="k">import</span> <span class="n">ProductivitySystem</span>
11166 <span class="kn">from</span> <span class="nn">employees</span> <span class="k">import</span> <span class="n">EmployeeDatabase</span>
11167
11168 <span class="n">productivity_system</span> <span class="o">=</span> <span class="n">ProductivitySystem</span><span class="p">()</span>
11169 <span class="n">payroll_system</span> <span class="o">=</span> <span class="n">PayrollSystem</span><span class="p">()</span>
11170 <span class="n">employee_database</span> <span class="o">=</span> <span class="n">EmployeeDatabase</span><span class="p">()</span>
11171 <span class="n">employees</span> <span class="o">=</span> <span class="n">employee_database</span><span class="o">.</span><span class="n">employees</span>
11172 <span class="n">productivity_system</span><span class="o">.</span><span class="n">track</span><span class="p">(</span><span class="n">employees</span><span class="p">,</span> <span class="mi">40</span><span class="p">)</span>
11173 <span class="n">payroll_system</span><span class="o">.</span><span class="n">calculate_payroll</span><span class="p">(</span><span class="n">employees</span><span class="p">)</span>
11174 </pre></div>
11175
11176 <p>You can run the program to see its output:</p>
11177 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python program.py
11178
11179 <span class="go">Tracking Employee Productivity</span>
11180 <span class="go">==============================</span>
11181 <span class="go">Employee 1 - Mary Poppins:</span>
11182 <span class="go">- screams and yells for 40 hours.</span>
11183
11184 <span class="go">Employee 2 - John Smith:</span>
11185 <span class="go">- does paperwork for 40 hours.</span>
11186
11187 <span class="go">Employee 3 - Kevin Bacon:</span>
11188 <span class="go">- expends 40 hours on the phone.</span>
11189
11190 <span class="go">Employee 4 - Jane Doe:</span>
11191 <span class="go">- manufactures gadgets for 40 hours.</span>
11192
11193 <span class="go">Employee 5 - Robin Williams:</span>
11194 <span class="go">- does paperwork for 40 hours.</span>
11195
11196
11197 <span class="go">Calculating Payroll</span>
11198 <span class="go">===================</span>
11199 <span class="go">Payroll for: 1 - Mary Poppins</span>
11200 <span class="go">- Check amount: 3000</span>
11201 <span class="go">- Sent to:</span>
11202 <span class="go">121 Admin Rd.</span>
11203 <span class="go">Concord, NH 03301</span>
11204
11205 <span class="go">Payroll for: 2 - John Smith</span>
11206 <span class="go">- Check amount: 1500</span>
11207 <span class="go">- Sent to:</span>
11208 <span class="go">67 Paperwork Ave</span>
11209 <span class="go">Manchester, NH 03101</span>
11210
11211 <span class="go">Payroll for: 3 - Kevin Bacon</span>
11212 <span class="go">- Check amount: 1800.0</span>
11213 <span class="go">- Sent to:</span>
11214 <span class="go">15 Rose St</span>
11215 <span class="go">Apt. B-1</span>
11216 <span class="go">Concord, NH 03301</span>
11217
11218 <span class="go">Payroll for: 4 - Jane Doe</span>
11219 <span class="go">- Check amount: 600</span>
11220 <span class="go">- Sent to:</span>
11221 <span class="go">39 Sole St.</span>
11222 <span class="go">Concord, NH 03301</span>
11223
11224 <span class="go">Payroll for: 5 - Robin Williams</span>
11225 <span class="go">- Check amount: 360</span>
11226 <span class="go">- Sent to:</span>
11227 <span class="go">99 Mountain Rd.</span>
11228 <span class="go">Concord, NH 03301</span>
11229 </pre></div>
11230
11231 <p>This design is what is called <a href="https://en.wikipedia.org/wiki/Modern_C%2B%2B_Design#Policy-based_design">policy-based design</a>, where classes are composed of policies, and they delegate to those policies to do the work.</p>
11232 <p>Policy-based design was introduced in the book <a href="https://realpython.com/asins/B00AU3JUHG">Modern C++ Design</a>, and it uses template metaprogramming in C++ to achieve the results.</p>
11233 <p>Python does not support templates, but you can achieve similar results using composition, as you saw in the example above.</p>
11234 <p>This type of design gives you all the flexibility you’ll need as requirements change. Imagine you need to change the way payroll is calculated for an object at run-time.</p>
11235 <h3 id="customizing-behavior-with-composition">Customizing Behavior With Composition</h3>
11236 <p>If your design relies on inheritance, you need to find a way to change the type of an object to change its behavior. With composition, you just need to change the policy the object uses.</p>
11237 <p>Imagine that our <code>manager</code> all of a sudden becomes a temporary employee that gets paid by the hour. You can modify the object during the execution of the program in the following way:</p>
11238 <div class="highlight python"><pre><span></span><span class="c1"># In program.py</span>
11239
11240 <span class="kn">from</span> <span class="nn">hr</span> <span class="k">import</span> <span class="n">PayrollSystem</span><span class="p">,</span> <span class="n">HourlyPolicy</span>
11241 <span class="kn">from</span> <span class="nn">productivity</span> <span class="k">import</span> <span class="n">ProductivitySystem</span>
11242 <span class="kn">from</span> <span class="nn">employees</span> <span class="k">import</span> <span class="n">EmployeeDatabase</span>
11243
11244 <span class="n">productivity_system</span> <span class="o">=</span> <span class="n">ProductivitySystem</span><span class="p">()</span>
11245 <span class="n">payroll_system</span> <span class="o">=</span> <span class="n">PayrollSystem</span><span class="p">()</span>
11246 <span class="n">employee_database</span> <span class="o">=</span> <span class="n">EmployeeDatabase</span><span class="p">()</span>
11247 <span class="n">employees</span> <span class="o">=</span> <span class="n">employee_database</span><span class="o">.</span><span class="n">employees</span>
11248 <span class="n">manager</span> <span class="o">=</span> <span class="n">employees</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
11249 <span class="n">manager</span><span class="o">.</span><span class="n">payroll</span> <span class="o">=</span> <span class="n">HourlyPolicy</span><span class="p">(</span><span class="mi">55</span><span class="p">)</span>
11250
11251 <span class="n">productivity_system</span><span class="o">.</span><span class="n">track</span><span class="p">(</span><span class="n">employees</span><span class="p">,</span> <span class="mi">40</span><span class="p">)</span>
11252 <span class="n">payroll_system</span><span class="o">.</span><span class="n">calculate_payroll</span><span class="p">(</span><span class="n">employees</span><span class="p">)</span>
11253 </pre></div>
11254
11255 <p>The program gets the employee list from the <code>EmployeeDatabase</code> and retrieves the first employee, which is the manager we want. Then it creates a new <code>HourlyPolicy</code> initialized at $55 per hour and assigns it to the manager object.</p>
11256 <p>The new policy is now used by the <code>PayrollSystem</code> modifying the existing behavior. You can run the program again to see the result:</p>
11257 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python program.py
11258
11259 <span class="go">Tracking Employee Productivity</span>
11260 <span class="go">==============================</span>
11261 <span class="go">Employee 1 - Mary Poppins:</span>
11262 <span class="go">- screams and yells for 40 hours.</span>
11263
11264 <span class="go">Employee 2 - John Smith:</span>
11265 <span class="go">- does paperwork for 40 hours.</span>
11266
11267 <span class="go">Employee 3 - Kevin Bacon:</span>
11268 <span class="go">- expends 40 hours on the phone.</span>
11269
11270 <span class="go">Employee 4 - Jane Doe:</span>
11271 <span class="go">- manufactures gadgets for 40 hours.</span>
11272
11273 <span class="go">Employee 5 - Robin Williams:</span>
11274 <span class="go">- does paperwork for 40 hours.</span>
11275
11276
11277 <span class="go">Calculating Payroll</span>
11278 <span class="go">===================</span>
11279 <span class="go">Payroll for: 1 - Mary Poppins</span>
11280 <span class="go">- Check amount: 2200</span>
11281 <span class="go">- Sent to:</span>
11282 <span class="go">121 Admin Rd.</span>
11283 <span class="go">Concord, NH 03301</span>
11284
11285 <span class="go">Payroll for: 2 - John Smith</span>
11286 <span class="go">- Check amount: 1500</span>
11287 <span class="go">- Sent to:</span>
11288 <span class="go">67 Paperwork Ave</span>
11289 <span class="go">Manchester, NH 03101</span>
11290
11291 <span class="go">Payroll for: 3 - Kevin Bacon</span>
11292 <span class="go">- Check amount: 1800.0</span>
11293 <span class="go">- Sent to:</span>
11294 <span class="go">15 Rose St</span>
11295 <span class="go">Apt. B-1</span>
11296 <span class="go">Concord, NH 03301</span>
11297
11298 <span class="go">Payroll for: 4 - Jane Doe</span>
11299 <span class="go">- Check amount: 600</span>
11300 <span class="go">- Sent to:</span>
11301 <span class="go">39 Sole St.</span>
11302 <span class="go">Concord, NH 03301</span>
11303
11304 <span class="go">Payroll for: 5 - Robin Williams</span>
11305 <span class="go">- Check amount: 360</span>
11306 <span class="go">- Sent to:</span>
11307 <span class="go">99 Mountain Rd.</span>
11308 <span class="go">Concord, NH 03301</span>
11309 </pre></div>
11310
11311 <p>The check for Mary Poppins, our manager, is now for $2200 instead of the fixed salary of $3000 that she had per week.</p>
11312 <p>Notice how we added that business rule to the program without changing any of the existing classes. Consider what type of changes would’ve been required with an inheritance design. </p>
11313 <p>You would’ve had to create a new class and change the type of the manager employee. There is no chance you could’ve changed the policy at run-time.</p>
11314 <h2 id="choosing-between-inheritance-and-composition-in-python">Choosing Between Inheritance and Composition in Python</h2>
11315 <p>So far, you’ve seen how inheritance and composition work in Python. You’ve seen that derived classes inherit the interface and implementation of their base classes. You’ve also seen that composition allows you to reuse the implementation of another class.</p>
11316 <p>You’ve implemented two solutions to the same problem. The first solution used multiple inheritance, and the second one used composition.</p>
11317 <p>You’ve also seen that Python’s duck typing allows you to reuse objects with existing parts of a program by implementing the desired interface. In Python, it isn’t necessary to derive from a base class for your classes to be reused.</p>
11318 <p>At this point, you might be asking when to use inheritance vs composition in Python. They both enable code reuse. Inheritance and composition can tackle similar problems in your Python programs.</p>
11319 <p>The general advice is to use the relationship that creates fewer dependencies between two classes. This relation is composition. Still, there will be times where inheritance will make more sense.</p>
11320 <p>The following sections provide some guidelines to help you make the right choice between inheritance and composition in Python.</p>
11321 <h3 id="inheritance-to-model-is-a-relationship">Inheritance to Model “Is A” Relationship</h3>
11322 <p>Inheritance should only be used to model an <strong>is a</strong> relationship. Liskov’s substitution principle says that an object of type <code>Derived</code>, which inherits from <code>Base</code>, can replace an object of type <code>Base</code> without altering the desirable properties of a program.</p>
11323 <p>Liskov’s substitution principle is the most important guideline to determine if inheritance is the appropriate design solution. Still, the answer might not be straightforward in all situations. Fortunately, there is a simple test you can use to determine if your design follows Liskov’s substitution principle.</p>
11324 <p>Let’s say you have a class <code>A</code> that provides an implementation and interface you want to reuse in another class <code>B</code>. Your initial thought is that you can derive <code>B</code> from <code>A</code> and inherit both the interface and implementation. To be sure this is the right design, you follow theses steps:</p>
11325 <ol>
11326 <li>
11327 <p><strong>Evaluate <code>B</code> is an <code>A</code>:</strong> Think about this relationship and justify it. Does it make sense?</p>
11328 </li>
11329 <li>
11330 <p><strong>Evaluate <code>A</code> is a <code>B</code>:</strong> Reverse the relationship and justify it. Does it also make sense?</p>
11331 </li>
11332 </ol>
11333 <p>If you can justify both relationships, then you should never inherit those classes from one another. Let’s look at a more concrete example.</p>
11334 <p>You have a class <code>Rectangle</code> which exposes an <code>.area</code> property. You need a class <code>Square</code>, which also has an <code>.area</code>. It seems that a <code>Square</code> is a special type of <code>Rectangle</code>, so maybe you can derive from it and leverage both the interface and implementation.</p>
11335 <p>Before you jump into the implementation, you use Liskov’s substitution principle to evaluate the relationship.</p>
11336 <p>A <code>Square</code> <strong>is a</strong> <code>Rectangle</code> because its area is calculated from the product of its <code>height</code> times its <code>length</code>. The constraint is that <code>Square.height</code> and <code>Square.length</code> must be equal.</p>
11337 <p>It makes sense. You can justify the relationship and explain why a <code>Square</code> <strong>is a</strong> <code>Rectangle</code>. Let’s reverse the relationship to see if it makes sense.</p>
11338 <p>A <code>Rectangle</code> <strong>is a</strong> <code>Square</code> because its area is calculated from the product of its <code>height</code> times its <code>length</code>. The difference is that <code>Rectangle.height</code> and <code>Rectangle.width</code> can change independently.</p>
11339 <p>It also makes sense. You can justify the relationship and describe the special constraints for each class. This is a good sign that these two classes should never derive from each other.</p>
11340 <p>You might have seen other examples that derive <code>Square</code> from <code>Rectangle</code> to explain inheritance. You might be skeptical with the little test you just did. Fair enough. Let’s write a program that illustrates the problem with deriving <code>Square</code> from <code>Rectangle</code>.</p>
11341 <p>First, you implement <code>Rectangle</code>. You’re even going to <a href="https://en.wikipedia.org/wiki/Encapsulation_(computer_programming)">encapsulate</a> the attributes to ensure that all the constraints are met:</p>
11342 <div class="highlight python"><pre><span></span><span class="c1"># In rectangle_square_demo.py</span>
11343
11344 <span class="k">class</span> <span class="nc">Rectangle</span><span class="p">:</span>
11345 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">length</span><span class="p">,</span> <span class="n">height</span><span class="p">):</span>
11346 <span class="bp">self</span><span class="o">.</span><span class="n">_length</span> <span class="o">=</span> <span class="n">length</span>
11347 <span class="bp">self</span><span class="o">.</span><span class="n">_height</span> <span class="o">=</span> <span class="n">height</span>
11348
11349 <span class="nd">@property</span>
11350 <span class="k">def</span> <span class="nf">area</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
11351 <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_length</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">_height</span>
11352 </pre></div>
11353
11354 <p>The <code>Rectangle</code> class is initialized with a <code>length</code> and a <code>height</code>, and it provides an <code>.area</code> property that returns the area. The <code>length</code> and <code>height</code> are encapsulated to avoid changing them directly.</p>
11355 <p>Now, you derive <code>Square</code> from <code>Rectangle</code> and override the necessary interface to meet the constraints of a <code>Square</code>:</p>
11356 <div class="highlight python"><pre><span></span><span class="c1"># In rectangle_square_demo.py</span>
11357
11358 <span class="k">class</span> <span class="nc">Square</span><span class="p">(</span><span class="n">Rectangle</span><span class="p">):</span>
11359 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">side_size</span><span class="p">):</span>
11360 <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">side_size</span><span class="p">,</span> <span class="n">side_size</span><span class="p">)</span>
11361 </pre></div>
11362
11363 <p>The <code>Square</code> class is initialized with a <code>side_size</code>, which is used to initialize both components of the base class. Now, you write a small program to test the behavior:</p>
11364 <div class="highlight python"><pre><span></span><span class="c1"># In rectangle_square_demo.py</span>
11365
11366 <span class="n">rectangle</span> <span class="o">=</span> <span class="n">Rectangle</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">4</span><span class="p">)</span>
11367 <span class="k">assert</span> <span class="n">rectangle</span><span class="o">.</span><span class="n">area</span> <span class="o">==</span> <span class="mi">8</span>
11368
11369 <span class="n">square</span> <span class="o">=</span> <span class="n">Square</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
11370 <span class="k">assert</span> <span class="n">square</span><span class="o">.</span><span class="n">area</span> <span class="o">==</span> <span class="mi">4</span>
11371
11372 <span class="nb">print</span><span class="p">(</span><span class="s1">'OK!'</span><span class="p">)</span>
11373 </pre></div>
11374
11375 <p>The program creates a <code>Rectangle</code> and a <code>Square</code> and asserts that their <code>.area</code> is calculated correctly. You can run the program and see that everything is <code>OK</code> so far:</p>
11376 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python rectangle_square_demo.py
11377
11378 <span class="go">OK!</span>
11379 </pre></div>
11380
11381 <p>The program executes correctly, so it seems that <code>Square</code> is just a special case of a <code>Rectangle</code>.</p>
11382 <p>Later on, you need to support resizing <code>Rectangle</code> objects, so you make the appropriate changes to the class:</p>
11383 <div class="highlight python"><pre><span></span><span class="c1"># In rectangle_square_demo.py</span>
11384
11385 <span class="k">class</span> <span class="nc">Rectangle</span><span class="p">:</span>
11386 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">length</span><span class="p">,</span> <span class="n">height</span><span class="p">):</span>
11387 <span class="bp">self</span><span class="o">.</span><span class="n">_length</span> <span class="o">=</span> <span class="n">length</span>
11388 <span class="bp">self</span><span class="o">.</span><span class="n">_height</span> <span class="o">=</span> <span class="n">height</span>
11389
11390 <span class="nd">@property</span>
11391 <span class="k">def</span> <span class="nf">area</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
11392 <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_length</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">_height</span>
11393
11394 <span class="k">def</span> <span class="nf">resize</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">new_length</span><span class="p">,</span> <span class="n">new_height</span><span class="p">):</span>
11395 <span class="bp">self</span><span class="o">.</span><span class="n">_length</span> <span class="o">=</span> <span class="n">new_length</span>
11396 <span class="bp">self</span><span class="o">.</span><span class="n">_height</span> <span class="o">=</span> <span class="n">new_height</span>
11397 </pre></div>
11398
11399 <p><code>.resize()</code> takes the <code>new_length</code> and <code>new_width</code> for the object. You can add the following code to the program to verify that it works correctly:</p>
11400 <div class="highlight python"><pre><span></span><span class="c1"># In rectangle_square_demo.py</span>
11401
11402 <span class="n">rectangle</span><span class="o">.</span><span class="n">resize</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">5</span><span class="p">)</span>
11403 <span class="k">assert</span> <span class="n">rectangle</span><span class="o">.</span><span class="n">area</span> <span class="o">==</span> <span class="mi">15</span>
11404
11405 <span class="nb">print</span><span class="p">(</span><span class="s1">'OK!'</span><span class="p">)</span>
11406 </pre></div>
11407
11408 <p>You resize the rectangle object and assert that the new area is correct. You can run the program to verify the behavior:</p>
11409 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python rectangle_square_demo.py
11410
11411 <span class="go">OK!</span>
11412 </pre></div>
11413
11414 <p>The assertion passes, and you see that the program runs correctly.</p>
11415 <p>So, what happens if you resize a square? Modify the program, and try to modify the <code>square</code> object:</p>
11416 <div class="highlight python"><pre><span></span><span class="c1"># In rectangle_square_demo.py</span>
11417
11418 <span class="n">square</span><span class="o">.</span><span class="n">resize</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">5</span><span class="p">)</span>
11419 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Square area: </span><span class="si">{square.area}</span><span class="s1">'</span><span class="p">)</span>
11420 </pre></div>
11421
11422 <p>You pass the same parameters to <code>square.resize()</code> that you used with <code>rectangle</code>, and print the area. When you run the program you see:</p>
11423 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python rectangle_square_demo.py
11424
11425 <span class="go">Square area: 15</span>
11426 <span class="go">OK!</span>
11427 </pre></div>
11428
11429 <p>The program shows that the new area is <code>15</code> like the <code>rectangle</code> object. The problem now is that the <code>square</code> object no longer meets the <code>Square</code> class constraint that the <code>length</code> and <code>height</code> must be equal.</p>
11430 <p>How can you fix that problem? You can try several approaches, but all of them will be awkward. You can override <code>.resize()</code> in <code>square</code> and ignore the <code>height</code> parameter, but that will be confusing for people looking at other parts of the program where <code>rectangles</code> are being resized and some of them are not getting the expected areas because they are really <code>squares</code>.</p>
11431 <p>In a small program like this one, it might be easy to spot the causes of the weird behavior, but in a more complex program, the problem will be harder to find.</p>
11432 <p>The reality is that if you’re able to justify an inheritance relationship between two classes both ways, you should not derive one class from another. </p>
11433 <p>In the example, it doesn’t make sense that <code>Square</code> inherits the interface and implementation of <code>.resize()</code> from <code>Rectangle</code>. That doesn’t mean that <code>Square</code> objects can’t be resized. It means that the interface is different because it only needs a <code>side_size</code> parameter.</p>
11434 <p>This difference in interface justifies not deriving <code>Square</code> from <code>Rectangle</code> like the test above advised.</p>
11435 <h3 id="mixing-features-with-mixin-classes">Mixing Features With Mixin Classes</h3>
11436 <p>One of the uses of multiple inheritance in Python is to extend a class features through <a href="https://en.wikipedia.org/wiki/Mixin">mixins</a>. A <strong>mixin</strong> is a class that provides methods to other classes but are not considered a base class.</p>
11437 <p>A mixin allows other classes to reuse its interface and implementation without becoming a super class. They implement a unique behavior that can be aggregated to other unrelated classes. They are similar to composition but they create a stronger relationship.</p>
11438 <p>Let’s say you want to convert objects of certain types in your application to a dictionary representation of the object. You could provide a <code>.to_dict()</code> method in every class that you want to support this feature, but the implementation of <code>.to_dict()</code> seems to be very similar.</p>
11439 <p>This could be a good candidate for a mixin. You start by slightly modifying the <code>Employee</code> class from the composition example:</p>
11440 <div class="highlight python"><pre><span></span><span class="c1"># In employees.py</span>
11441
11442 <span class="k">class</span> <span class="nc">Employee</span><span class="p">:</span>
11443 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">address</span><span class="p">,</span> <span class="n">role</span><span class="p">,</span> <span class="n">payroll</span><span class="p">):</span>
11444 <span class="bp">self</span><span class="o">.</span><span class="n">id</span> <span class="o">=</span> <span class="nb">id</span>
11445 <span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">name</span>
11446 <span class="bp">self</span><span class="o">.</span><span class="n">address</span> <span class="o">=</span> <span class="n">address</span>
11447 <span class="hll"> <span class="bp">self</span><span class="o">.</span><span class="n">_role</span> <span class="o">=</span> <span class="n">role</span>
11448 </span><span class="hll"> <span class="bp">self</span><span class="o">.</span><span class="n">_payroll</span> <span class="o">=</span> <span class="n">payroll</span>
11449 </span>
11450
11451 <span class="k">def</span> <span class="nf">work</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hours</span><span class="p">):</span>
11452 <span class="n">duties</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_role</span><span class="o">.</span><span class="n">perform_duties</span><span class="p">(</span><span class="n">hours</span><span class="p">)</span>
11453 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Employee </span><span class="si">{self.id}</span><span class="s1"> - </span><span class="si">{self.name}</span><span class="s1">:'</span><span class="p">)</span>
11454 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'- </span><span class="si">{duties}</span><span class="s1">'</span><span class="p">)</span>
11455 <span class="nb">print</span><span class="p">(</span><span class="s1">''</span><span class="p">)</span>
11456 <span class="bp">self</span><span class="o">.</span><span class="n">_payroll</span><span class="o">.</span><span class="n">track_work</span><span class="p">(</span><span class="n">hours</span><span class="p">)</span>
11457
11458 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
11459 <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_payroll</span><span class="o">.</span><span class="n">calculate_payroll</span><span class="p">()</span>
11460 </pre></div>
11461
11462 <p>The change is very small. You just changed the <code>role</code> and <code>payroll</code> attributes to be internal by adding a leading underscore to their name. You will see soon why you are making that change.</p>
11463 <p>Now, you add the <code>AsDictionaryMixin</code> class:</p>
11464 <div class="highlight python"><pre><span></span><span class="c1"># In representations.py</span>
11465
11466 <span class="k">class</span> <span class="nc">AsDictionaryMixin</span><span class="p">:</span>
11467 <span class="k">def</span> <span class="nf">to_dict</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
11468 <span class="k">return</span> <span class="p">{</span>
11469 <span class="n">prop</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">_represent</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
11470 <span class="k">for</span> <span class="n">prop</span><span class="p">,</span> <span class="n">value</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="vm">__dict__</span><span class="o">.</span><span class="n">items</span><span class="p">()</span>
11471 <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">_is_internal</span><span class="p">(</span><span class="n">prop</span><span class="p">)</span>
11472 <span class="p">}</span>
11473
11474 <span class="k">def</span> <span class="nf">_represent</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">value</span><span class="p">):</span>
11475 <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="nb">object</span><span class="p">):</span>
11476 <span class="k">if</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="s1">'to_dict'</span><span class="p">):</span>
11477 <span class="k">return</span> <span class="n">value</span><span class="o">.</span><span class="n">to_dict</span><span class="p">()</span>
11478 <span class="k">else</span><span class="p">:</span>
11479 <span class="k">return</span> <span class="nb">str</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
11480 <span class="k">else</span><span class="p">:</span>
11481 <span class="k">return</span> <span class="n">value</span>
11482
11483 <span class="k">def</span> <span class="nf">_is_internal</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">prop</span><span class="p">):</span>
11484 <span class="k">return</span> <span class="n">prop</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'_'</span><span class="p">)</span>
11485 </pre></div>
11486
11487 <p>The <code>AsDictionaryMixin</code> class exposes a <code>.to_dict()</code> method that returns the representation of itself as a dictionary. The method is implemented as a <a href="https://www.python.org/dev/peps/pep-0274/"><code>dict</code> comprehension</a> that says, “Create a dictionary mapping <code>prop</code> to <code>value</code> for each item in <code>self.__dict__.items()</code> if the <code>prop</code> is not internal.”</p>
11488 <div class="alert alert-primary" role="alert">
11489 <p><strong>Note:</strong> This is why we made the role and payroll attributes internal in the <code>Employee</code> class, because we don’t want to represent them in the dictionary.</p>
11490 </div>
11491 <p>As you saw at the beginning, creating a class inherits some members from <code>object</code>, and one of those members is <code>__dict__</code>, which is basically a mapping of all the attributes in an object to their value.</p>
11492 <p>You iterate through all the items in <code>__dict__</code> and filter out the ones that have a name that starts with an underscore using <code>._is_internal()</code>.</p>
11493 <p><code>._represent()</code> checks the specified value. If the value <strong>is an</strong> <code>object</code>, then it looks to see if it also has a <code>.to_dict()</code> member and uses it to represent the object. Otherwise, it returns a string representation. If the value is not an <code>object</code>, then it simply returns the value.</p>
11494 <p>You can modify the <code>Employee</code> class to support this mixin:</p>
11495 <div class="highlight python"><pre><span></span><span class="c1"># In employees.py</span>
11496
11497 <span class="kn">from</span> <span class="nn">representations</span> <span class="k">import</span> <span class="n">AsDictionaryMixin</span>
11498
11499 <span class="k">class</span> <span class="nc">Employee</span><span class="p">(</span><span class="n">AsDictionaryMixin</span><span class="p">):</span>
11500 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">address</span><span class="p">,</span> <span class="n">role</span><span class="p">,</span> <span class="n">payroll</span><span class="p">):</span>
11501 <span class="bp">self</span><span class="o">.</span><span class="n">id</span> <span class="o">=</span> <span class="nb">id</span>
11502 <span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">name</span>
11503 <span class="bp">self</span><span class="o">.</span><span class="n">address</span> <span class="o">=</span> <span class="n">address</span>
11504 <span class="bp">self</span><span class="o">.</span><span class="n">_role</span> <span class="o">=</span> <span class="n">role</span>
11505 <span class="bp">self</span><span class="o">.</span><span class="n">_payroll</span> <span class="o">=</span> <span class="n">payroll</span>
11506
11507 <span class="k">def</span> <span class="nf">work</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hours</span><span class="p">):</span>
11508 <span class="n">duties</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_role</span><span class="o">.</span><span class="n">perform_duties</span><span class="p">(</span><span class="n">hours</span><span class="p">)</span>
11509 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Employee </span><span class="si">{self.id}</span><span class="s1"> - </span><span class="si">{self.name}</span><span class="s1">:'</span><span class="p">)</span>
11510 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'- </span><span class="si">{duties}</span><span class="s1">'</span><span class="p">)</span>
11511 <span class="nb">print</span><span class="p">(</span><span class="s1">''</span><span class="p">)</span>
11512 <span class="bp">self</span><span class="o">.</span><span class="n">_payroll</span><span class="o">.</span><span class="n">track_work</span><span class="p">(</span><span class="n">hours</span><span class="p">)</span>
11513
11514 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
11515 <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_payroll</span><span class="o">.</span><span class="n">calculate_payroll</span><span class="p">()</span>
11516 </pre></div>
11517
11518 <p>All you have to do is inherit the <code>AsDictionaryMixin</code> to support the functionality. It will be nice to support the same functionality in the <code>Address</code> class, so the <code>Employee.address</code> attribute is represented in the same way:</p>
11519 <div class="highlight python"><pre><span></span><span class="c1"># In contacts.py</span>
11520
11521 <span class="kn">from</span> <span class="nn">representations</span> <span class="k">import</span> <span class="n">AsDictionaryMixin</span>
11522
11523 <span class="k">class</span> <span class="nc">Address</span><span class="p">(</span><span class="n">AsDictionaryMixin</span><span class="p">):</span>
11524 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">street</span><span class="p">,</span> <span class="n">city</span><span class="p">,</span> <span class="n">state</span><span class="p">,</span> <span class="n">zipcode</span><span class="p">,</span> <span class="n">street2</span><span class="o">=</span><span class="s1">''</span><span class="p">):</span>
11525 <span class="bp">self</span><span class="o">.</span><span class="n">street</span> <span class="o">=</span> <span class="n">street</span>
11526 <span class="bp">self</span><span class="o">.</span><span class="n">street2</span> <span class="o">=</span> <span class="n">street2</span>
11527 <span class="bp">self</span><span class="o">.</span><span class="n">city</span> <span class="o">=</span> <span class="n">city</span>
11528 <span class="bp">self</span><span class="o">.</span><span class="n">state</span> <span class="o">=</span> <span class="n">state</span>
11529 <span class="bp">self</span><span class="o">.</span><span class="n">zipcode</span> <span class="o">=</span> <span class="n">zipcode</span>
11530
11531 <span class="k">def</span> <span class="nf">__str__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
11532 <span class="n">lines</span> <span class="o">=</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">street</span><span class="p">]</span>
11533 <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">street2</span><span class="p">:</span>
11534 <span class="n">lines</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">street2</span><span class="p">)</span>
11535 <span class="n">lines</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">f</span><span class="s1">'</span><span class="si">{self.city}</span><span class="s1">, </span><span class="si">{self.state}</span><span class="s1"> </span><span class="si">{self.zipcode}</span><span class="s1">'</span><span class="p">)</span>
11536 <span class="k">return</span> <span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">lines</span><span class="p">)</span>
11537 </pre></div>
11538
11539 <p>You apply the mixin to the <code>Address</code> class to support the feature. Now, you can write a small program to test it:</p>
11540 <div class="highlight python"><pre><span></span> <span class="c1"># In program.py</span>
11541
11542 <span class="kn">import</span> <span class="nn">json</span>
11543 <span class="kn">from</span> <span class="nn">employees</span> <span class="k">import</span> <span class="n">EmployeeDatabase</span>
11544
11545 <span class="k">def</span> <span class="nf">print_dict</span><span class="p">(</span><span class="n">d</span><span class="p">):</span>
11546 <span class="nb">print</span><span class="p">(</span><span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">d</span><span class="p">,</span> <span class="n">indent</span><span class="o">=</span><span class="mi">2</span><span class="p">))</span>
11547
11548 <span class="k">for</span> <span class="n">employee</span> <span class="ow">in</span> <span class="n">EmployeeDatabase</span><span class="p">()</span><span class="o">.</span><span class="n">employees</span><span class="p">:</span>
11549 <span class="n">print_dict</span><span class="p">(</span><span class="n">employee</span><span class="o">.</span><span class="n">to_dict</span><span class="p">())</span>
11550 </pre></div>
11551
11552 <p>The program implements a <code>print_dict()</code> that converts the dictionary to a <a href="http://json.org/">JSON</a> string using indentation so the output looks better.</p>
11553 <p>Then, it iterates through all the employees, printing the dictionary representation provided by <code>.to_dict()</code>. You can run the program to see its output:</p>
11554 <div class="highlight sh"><pre><span></span><span class="gp"> $</span> python program.py
11555
11556 <span class="go"> {</span>
11557 <span class="go"> "id": "1",</span>
11558 <span class="go"> "name": "Mary Poppins",</span>
11559 <span class="go"> "address": {</span>
11560 <span class="go"> "street": "121 Admin Rd.",</span>
11561 <span class="go"> "street2": "",</span>
11562 <span class="go"> "city": "Concord",</span>
11563 <span class="go"> "state": "NH",</span>
11564 <span class="go"> "zipcode": "03301"</span>
11565 <span class="go"> }</span>
11566 <span class="go">}</span>
11567 <span class="go">{</span>
11568 <span class="go"> "id": "2",</span>
11569 <span class="go"> "name": "John Smith",</span>
11570 <span class="go"> "address": {</span>
11571 <span class="go"> "street": "67 Paperwork Ave",</span>
11572 <span class="go"> "street2": "",</span>
11573 <span class="go"> "city": "Manchester",</span>
11574 <span class="go"> "state": "NH",</span>
11575 <span class="go"> "zipcode": "03101"</span>
11576 <span class="go"> }</span>
11577 <span class="go">}</span>
11578 <span class="go">{</span>
11579 <span class="go"> "id": "3",</span>
11580 <span class="go"> "name": "Kevin Bacon",</span>
11581 <span class="go"> "address": {</span>
11582 <span class="go"> "street": "15 Rose St",</span>
11583 <span class="go"> "street2": "Apt. B-1",</span>
11584 <span class="go"> "city": "Concord",</span>
11585 <span class="go"> "state": "NH",</span>
11586 <span class="go"> "zipcode": "03301"</span>
11587 <span class="go"> }</span>
11588 <span class="go">}</span>
11589 <span class="go">{</span>
11590 <span class="go"> "id": "4",</span>
11591 <span class="go"> "name": "Jane Doe",</span>
11592 <span class="go"> "address": {</span>
11593 <span class="go"> "street": "39 Sole St.",</span>
11594 <span class="go"> "street2": "",</span>
11595 <span class="go"> "city": "Concord",</span>
11596 <span class="go"> "state": "NH",</span>
11597 <span class="go"> "zipcode": "03301"</span>
11598 <span class="go"> }</span>
11599 <span class="go">}</span>
11600 <span class="go">{</span>
11601 <span class="go"> "id": "5",</span>
11602 <span class="go"> "name": "Robin Williams",</span>
11603 <span class="go"> "address": {</span>
11604 <span class="go"> "street": "99 Mountain Rd.",</span>
11605 <span class="go"> "street2": "",</span>
11606 <span class="go"> "city": "Concord",</span>
11607 <span class="go"> "state": "NH",</span>
11608 <span class="go"> "zipcode": "03301"</span>
11609 <span class="go"> }</span>
11610 <span class="go">}</span>
11611 </pre></div>
11612
11613 <p>You leveraged the implementation of <code>AsDictionaryMixin</code> in both <code>Employee</code> and <code>Address</code> classes even when they are not related. Because <code>AsDictionaryMixin</code> only provides behavior, it is easy to reuse with other classes without causing problems.</p>
11614 <h3 id="composition-to-model-has-a-relationship">Composition to Model “Has A” Relationship</h3>
11615 <p>Composition models a <strong>has a</strong> relationship. With composition, a class <code>Composite</code> <strong>has an</strong> instance of class <code>Component</code> and can leverage its implementation. The <code>Component</code> class can be reused in other classes completely unrelated to the <code>Composite</code>.</p>
11616 <p>In the composition example above, the <code>Employee</code> class <strong>has an</strong> <code>Address</code> object. <code>Address</code> implements all the functionality to handle addresses, and it can be reused by other classes.</p>
11617 <p>Other classes like <code>Customer</code> or <code>Vendor</code> can reuse <code>Address</code> without being related to <code>Employee</code>. They can leverage the same implementation ensuring that addresses are handled consistently across the application.</p>
11618 <p>A problem you may run into when using composition is that some of your classes may start growing by using multiple components. Your classes may require multiple parameters in the constructor just to pass in the components they are made of. This can make your classes hard to use.</p>
11619 <p>A way to avoid the problem is by using the <a href="https://realpython.com/factory-method-python/">Factory Method</a> to construct your objects. You did that with the composition example.</p>
11620 <p>If you look at the implementation of the <code>EmployeeDatabase</code> class, you’ll notice that it uses <code>._create_employee()</code> to construct an <code>Employee</code> object with the right parameters.</p>
11621 <p>This design will work, but ideally, you should be able to construct an <code>Employee</code> object just by specifying an <code>id</code>, for example <code>employee = Employee(1)</code>.</p>
11622 <p>The following changes might improve your design. You can start with the <code>productivity</code> module:</p>
11623 <div class="highlight python"><pre><span></span><span class="c1"># In productivity.py</span>
11624
11625 <span class="k">class</span> <span class="nc">_ProductivitySystem</span><span class="p">:</span>
11626 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
11627 <span class="bp">self</span><span class="o">.</span><span class="n">_roles</span> <span class="o">=</span> <span class="p">{</span>
11628 <span class="s1">'manager'</span><span class="p">:</span> <span class="n">ManagerRole</span><span class="p">,</span>
11629 <span class="s1">'secretary'</span><span class="p">:</span> <span class="n">SecretaryRole</span><span class="p">,</span>
11630 <span class="s1">'sales'</span><span class="p">:</span> <span class="n">SalesRole</span><span class="p">,</span>
11631 <span class="s1">'factory'</span><span class="p">:</span> <span class="n">FactoryRole</span><span class="p">,</span>
11632 <span class="p">}</span>
11633
11634 <span class="k">def</span> <span class="nf">get_role</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">role_id</span><span class="p">):</span>
11635 <span class="n">role_type</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_roles</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">role_id</span><span class="p">)</span>
11636 <span class="k">if</span> <span class="ow">not</span> <span class="n">role_type</span><span class="p">:</span>
11637 <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'role_id'</span><span class="p">)</span>
11638 <span class="k">return</span> <span class="n">role_type</span><span class="p">()</span>
11639
11640 <span class="k">def</span> <span class="nf">track</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">employees</span><span class="p">,</span> <span class="n">hours</span><span class="p">):</span>
11641 <span class="nb">print</span><span class="p">(</span><span class="s1">'Tracking Employee Productivity'</span><span class="p">)</span>
11642 <span class="nb">print</span><span class="p">(</span><span class="s1">'=============================='</span><span class="p">)</span>
11643 <span class="k">for</span> <span class="n">employee</span> <span class="ow">in</span> <span class="n">employees</span><span class="p">:</span>
11644 <span class="n">employee</span><span class="o">.</span><span class="n">work</span><span class="p">(</span><span class="n">hours</span><span class="p">)</span>
11645 <span class="nb">print</span><span class="p">(</span><span class="s1">''</span><span class="p">)</span>
11646
11647 <span class="c1"># Role classes implementation omitted</span>
11648
11649 <span class="n">_productivity_system</span> <span class="o">=</span> <span class="n">_ProductivitySystem</span><span class="p">()</span>
11650
11651 <span class="k">def</span> <span class="nf">get_role</span><span class="p">(</span><span class="n">role_id</span><span class="p">):</span>
11652 <span class="k">return</span> <span class="n">_productivity_system</span><span class="o">.</span><span class="n">get_role</span><span class="p">(</span><span class="n">role_id</span><span class="p">)</span>
11653
11654 <span class="k">def</span> <span class="nf">track</span><span class="p">(</span><span class="n">employees</span><span class="p">,</span> <span class="n">hours</span><span class="p">):</span>
11655 <span class="n">_productivity_system</span><span class="o">.</span><span class="n">track</span><span class="p">(</span><span class="n">employees</span><span class="p">,</span> <span class="n">hours</span><span class="p">)</span>
11656 </pre></div>
11657
11658 <p>First, you make the <code>_ProductivitySystem</code> class internal, and then provide a <code>_productivity_system</code> internal variable to the module. You are communicating to other developers that they should not create or use the <code>_ProductivitySystem</code> directly. Instead, you provide two functions, <code>get_role()</code> and <code>track()</code>, as the public interface to the module. This is what other modules should use.</p>
11659 <p>What you are saying is that the <code>_ProductivitySystem</code> is a <a href="https://en.wikipedia.org/wiki/Singleton_pattern">Singleton</a>, and there should only be one object created from it.</p>
11660 <p>Now, you can do the same with the <code>hr</code> module:</p>
11661 <div class="highlight python"><pre><span></span><span class="c1"># In hr.py</span>
11662
11663 <span class="k">class</span> <span class="nc">_PayrollSystem</span><span class="p">:</span>
11664 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
11665 <span class="bp">self</span><span class="o">.</span><span class="n">_employee_policies</span> <span class="o">=</span> <span class="p">{</span>
11666 <span class="mi">1</span><span class="p">:</span> <span class="n">SalaryPolicy</span><span class="p">(</span><span class="mi">3000</span><span class="p">),</span>
11667 <span class="mi">2</span><span class="p">:</span> <span class="n">SalaryPolicy</span><span class="p">(</span><span class="mi">1500</span><span class="p">),</span>
11668 <span class="mi">3</span><span class="p">:</span> <span class="n">CommissionPolicy</span><span class="p">(</span><span class="mi">1000</span><span class="p">,</span> <span class="mi">100</span><span class="p">),</span>
11669 <span class="mi">4</span><span class="p">:</span> <span class="n">HourlyPolicy</span><span class="p">(</span><span class="mi">15</span><span class="p">),</span>
11670 <span class="mi">5</span><span class="p">:</span> <span class="n">HourlyPolicy</span><span class="p">(</span><span class="mi">9</span><span class="p">)</span>
11671 <span class="p">}</span>
11672
11673 <span class="k">def</span> <span class="nf">get_policy</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">employee_id</span><span class="p">):</span>
11674 <span class="n">policy</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_employee_policies</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">employee_id</span><span class="p">)</span>
11675 <span class="k">if</span> <span class="ow">not</span> <span class="n">policy</span><span class="p">:</span>
11676 <span class="k">return</span> <span class="ne">ValueError</span><span class="p">(</span><span class="n">employee_id</span><span class="p">)</span>
11677 <span class="k">return</span> <span class="n">policy</span>
11678
11679 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">employees</span><span class="p">):</span>
11680 <span class="nb">print</span><span class="p">(</span><span class="s1">'Calculating Payroll'</span><span class="p">)</span>
11681 <span class="nb">print</span><span class="p">(</span><span class="s1">'==================='</span><span class="p">)</span>
11682 <span class="k">for</span> <span class="n">employee</span> <span class="ow">in</span> <span class="n">employees</span><span class="p">:</span>
11683 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Payroll for: </span><span class="si">{employee.id}</span><span class="s1"> - </span><span class="si">{employee.name}</span><span class="s1">'</span><span class="p">)</span>
11684 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'- Check amount: {employee.calculate_payroll()}'</span><span class="p">)</span>
11685 <span class="k">if</span> <span class="n">employee</span><span class="o">.</span><span class="n">address</span><span class="p">:</span>
11686 <span class="nb">print</span><span class="p">(</span><span class="s1">'- Sent to:'</span><span class="p">)</span>
11687 <span class="nb">print</span><span class="p">(</span><span class="n">employee</span><span class="o">.</span><span class="n">address</span><span class="p">)</span>
11688 <span class="nb">print</span><span class="p">(</span><span class="s1">''</span><span class="p">)</span>
11689
11690 <span class="c1"># Policy classes implementation omitted</span>
11691
11692 <span class="n">_payroll_system</span> <span class="o">=</span> <span class="n">_PayrollSystem</span><span class="p">()</span>
11693
11694 <span class="k">def</span> <span class="nf">get_policy</span><span class="p">(</span><span class="n">employee_id</span><span class="p">):</span>
11695 <span class="k">return</span> <span class="n">_payroll_system</span><span class="o">.</span><span class="n">get_policy</span><span class="p">(</span><span class="n">employee_id</span><span class="p">)</span>
11696
11697 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="n">employees</span><span class="p">):</span>
11698 <span class="n">_payroll_system</span><span class="o">.</span><span class="n">calculate_payroll</span><span class="p">(</span><span class="n">employees</span><span class="p">)</span>
11699 </pre></div>
11700
11701 <p>Again, you make the <code>_PayrollSystem</code> internal and provide a public interface to it. The application will use the public interface to get policies and calculate payroll.</p>
11702 <p>You will now do the same with the <code>contacts</code> module:</p>
11703 <div class="highlight python"><pre><span></span><span class="c1"># In contacts.py</span>
11704
11705 <span class="k">class</span> <span class="nc">_AddressBook</span><span class="p">:</span>
11706 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
11707 <span class="bp">self</span><span class="o">.</span><span class="n">_employee_addresses</span> <span class="o">=</span> <span class="p">{</span>
11708 <span class="mi">1</span><span class="p">:</span> <span class="n">Address</span><span class="p">(</span><span class="s1">'121 Admin Rd.'</span><span class="p">,</span> <span class="s1">'Concord'</span><span class="p">,</span> <span class="s1">'NH'</span><span class="p">,</span> <span class="s1">'03301'</span><span class="p">),</span>
11709 <span class="mi">2</span><span class="p">:</span> <span class="n">Address</span><span class="p">(</span><span class="s1">'67 Paperwork Ave'</span><span class="p">,</span> <span class="s1">'Manchester'</span><span class="p">,</span> <span class="s1">'NH'</span><span class="p">,</span> <span class="s1">'03101'</span><span class="p">),</span>
11710 <span class="mi">3</span><span class="p">:</span> <span class="n">Address</span><span class="p">(</span><span class="s1">'15 Rose St'</span><span class="p">,</span> <span class="s1">'Concord'</span><span class="p">,</span> <span class="s1">'NH'</span><span class="p">,</span> <span class="s1">'03301'</span><span class="p">,</span> <span class="s1">'Apt. B-1'</span><span class="p">),</span>
11711 <span class="mi">4</span><span class="p">:</span> <span class="n">Address</span><span class="p">(</span><span class="s1">'39 Sole St.'</span><span class="p">,</span> <span class="s1">'Concord'</span><span class="p">,</span> <span class="s1">'NH'</span><span class="p">,</span> <span class="s1">'03301'</span><span class="p">),</span>
11712 <span class="mi">5</span><span class="p">:</span> <span class="n">Address</span><span class="p">(</span><span class="s1">'99 Mountain Rd.'</span><span class="p">,</span> <span class="s1">'Concord'</span><span class="p">,</span> <span class="s1">'NH'</span><span class="p">,</span> <span class="s1">'03301'</span><span class="p">),</span>
11713 <span class="p">}</span>
11714
11715 <span class="k">def</span> <span class="nf">get_employee_address</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">employee_id</span><span class="p">):</span>
11716 <span class="n">address</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_employee_addresses</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">employee_id</span><span class="p">)</span>
11717 <span class="k">if</span> <span class="ow">not</span> <span class="n">address</span><span class="p">:</span>
11718 <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="n">employee_id</span><span class="p">)</span>
11719 <span class="k">return</span> <span class="n">address</span>
11720
11721 <span class="c1"># Implementation of Address class omitted</span>
11722
11723 <span class="n">_address_book</span> <span class="o">=</span> <span class="n">_AddressBook</span><span class="p">()</span>
11724
11725 <span class="k">def</span> <span class="nf">get_employee_address</span><span class="p">(</span><span class="n">employee_id</span><span class="p">):</span>
11726 <span class="k">return</span> <span class="n">_address_book</span><span class="o">.</span><span class="n">get_employee_address</span><span class="p">(</span><span class="n">employee_id</span><span class="p">)</span>
11727 </pre></div>
11728
11729 <p>You are basically saying that there should only be one <code>_AddressBook</code>, one <code>_PayrollSystem</code>, and one <code>_ProductivitySystem</code>. Again, this design pattern is called the <a href="https://en.wikipedia.org/wiki/Singleton_pattern">Singleton</a> design pattern, which comes in handy for classes from which there should only be one, single instance.</p>
11730 <p>Now, you can work on the <code>employees</code> module. You will also make a Singleton out of the <code>_EmployeeDatabase</code>, but you will make some additional changes:</p>
11731 <div class="highlight python"><pre><span></span><span class="c1"># In employees.py</span>
11732
11733 <span class="kn">from</span> <span class="nn">productivity</span> <span class="k">import</span> <span class="n">get_role</span>
11734 <span class="kn">from</span> <span class="nn">hr</span> <span class="k">import</span> <span class="n">get_policy</span>
11735 <span class="kn">from</span> <span class="nn">contacts</span> <span class="k">import</span> <span class="n">get_employee_address</span>
11736 <span class="kn">from</span> <span class="nn">representations</span> <span class="k">import</span> <span class="n">AsDictionaryMixin</span>
11737
11738 <span class="k">class</span> <span class="nc">_EmployeeDatabase</span><span class="p">:</span>
11739 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
11740 <span class="bp">self</span><span class="o">.</span><span class="n">_employees</span> <span class="o">=</span> <span class="p">{</span>
11741 <span class="mi">1</span><span class="p">:</span> <span class="p">{</span>
11742 <span class="s1">'name'</span><span class="p">:</span> <span class="s1">'Mary Poppins'</span><span class="p">,</span>
11743 <span class="s1">'role'</span><span class="p">:</span> <span class="s1">'manager'</span>
11744 <span class="p">},</span>
11745 <span class="mi">2</span><span class="p">:</span> <span class="p">{</span>
11746 <span class="s1">'name'</span><span class="p">:</span> <span class="s1">'John Smith'</span><span class="p">,</span>
11747 <span class="s1">'role'</span><span class="p">:</span> <span class="s1">'secretary'</span>
11748 <span class="p">},</span>
11749 <span class="mi">3</span><span class="p">:</span> <span class="p">{</span>
11750 <span class="s1">'name'</span><span class="p">:</span> <span class="s1">'Kevin Bacon'</span><span class="p">,</span>
11751 <span class="s1">'role'</span><span class="p">:</span> <span class="s1">'sales'</span>
11752 <span class="p">},</span>
11753 <span class="mi">4</span><span class="p">:</span> <span class="p">{</span>
11754 <span class="s1">'name'</span><span class="p">:</span> <span class="s1">'Jane Doe'</span><span class="p">,</span>
11755 <span class="s1">'role'</span><span class="p">:</span> <span class="s1">'factory'</span>
11756 <span class="p">},</span>
11757 <span class="mi">5</span><span class="p">:</span> <span class="p">{</span>
11758 <span class="s1">'name'</span><span class="p">:</span> <span class="s1">'Robin Williams'</span><span class="p">,</span>
11759 <span class="s1">'role'</span><span class="p">:</span> <span class="s1">'secretary'</span>
11760 <span class="p">}</span>
11761 <span class="p">}</span>
11762
11763
11764 <span class="nd">@property</span>
11765 <span class="k">def</span> <span class="nf">employees</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
11766 <span class="k">return</span> <span class="p">[</span><span class="n">Employee</span><span class="p">(</span><span class="n">id_</span><span class="p">)</span> <span class="k">for</span> <span class="n">id_</span> <span class="ow">in</span> <span class="nb">sorted</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_employees</span><span class="p">)]</span>
11767
11768 <span class="k">def</span> <span class="nf">get_employee_info</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">employee_id</span><span class="p">):</span>
11769 <span class="n">info</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_employees</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">employee_id</span><span class="p">)</span>
11770 <span class="k">if</span> <span class="ow">not</span> <span class="n">info</span><span class="p">:</span>
11771 <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="n">employee_id</span><span class="p">)</span>
11772 <span class="k">return</span> <span class="n">info</span>
11773
11774 <span class="k">class</span> <span class="nc">Employee</span><span class="p">(</span><span class="n">AsDictionaryMixin</span><span class="p">):</span>
11775 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">):</span>
11776 <span class="bp">self</span><span class="o">.</span><span class="n">id</span> <span class="o">=</span> <span class="nb">id</span>
11777 <span class="n">info</span> <span class="o">=</span> <span class="n">employee_database</span><span class="o">.</span><span class="n">get_employee_info</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">id</span><span class="p">)</span>
11778 <span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">info</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'name'</span><span class="p">)</span>
11779 <span class="bp">self</span><span class="o">.</span><span class="n">address</span> <span class="o">=</span> <span class="n">get_employee_address</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">id</span><span class="p">)</span>
11780 <span class="bp">self</span><span class="o">.</span><span class="n">_role</span> <span class="o">=</span> <span class="n">get_role</span><span class="p">(</span><span class="n">info</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'role'</span><span class="p">))</span>
11781 <span class="bp">self</span><span class="o">.</span><span class="n">_payroll</span> <span class="o">=</span> <span class="n">get_policy</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">id</span><span class="p">)</span>
11782
11783 <span class="k">def</span> <span class="nf">work</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hours</span><span class="p">):</span>
11784 <span class="n">duties</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_role</span><span class="o">.</span><span class="n">perform_duties</span><span class="p">(</span><span class="n">hours</span><span class="p">)</span>
11785 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Employee </span><span class="si">{self.id}</span><span class="s1"> - </span><span class="si">{self.name}</span><span class="s1">:'</span><span class="p">)</span>
11786 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'- </span><span class="si">{duties}</span><span class="s1">'</span><span class="p">)</span>
11787 <span class="nb">print</span><span class="p">(</span><span class="s1">''</span><span class="p">)</span>
11788 <span class="bp">self</span><span class="o">.</span><span class="n">_payroll</span><span class="o">.</span><span class="n">track_work</span><span class="p">(</span><span class="n">hours</span><span class="p">)</span>
11789
11790 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
11791 <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_payroll</span><span class="o">.</span><span class="n">calculate_payroll</span><span class="p">()</span>
11792
11793
11794 <span class="n">employee_database</span> <span class="o">=</span> <span class="n">_EmployeeDatabase</span><span class="p">()</span>
11795 </pre></div>
11796
11797 <p>You first import the relevant functions and classes from other modules. The <code>_EmployeeDatabase</code> is made internal, and at the bottom, you create a single instance. This instance is public and part of the interface because you will want to use it in the application.</p>
11798 <p>You changed the <code>_EmployeeDatabase._employees</code> attribute to be a dictionary where the key is the employee <code>id</code> and the value is the employee information. You also exposed a <code>.get_employee_info()</code> method to return the information for the specified employee <code>employee_id</code>.</p>
11799 <p>The <code>_EmployeeDatabase.employees</code> property now sorts the keys to return the employees sorted by their <code>id</code>. You replaced the method that constructed the <code>Employee</code> objects with calls to the <code>Employee</code> initializer directly.</p>
11800 <p>The <code>Employee</code> class now is initialized with the <code>id</code> and uses the public functions exposed in the other modules to initialize its attributes.</p>
11801 <p>You can now change the program to test the changes:</p>
11802 <div class="highlight python"><pre><span></span><span class="c1"># In program.py</span>
11803
11804 <span class="kn">import</span> <span class="nn">json</span>
11805
11806 <span class="kn">from</span> <span class="nn">hr</span> <span class="k">import</span> <span class="n">calculate_payroll</span>
11807 <span class="kn">from</span> <span class="nn">productivity</span> <span class="k">import</span> <span class="n">track</span>
11808 <span class="kn">from</span> <span class="nn">employees</span> <span class="k">import</span> <span class="n">employee_database</span><span class="p">,</span> <span class="n">Employee</span>
11809
11810 <span class="k">def</span> <span class="nf">print_dict</span><span class="p">(</span><span class="n">d</span><span class="p">):</span>
11811 <span class="nb">print</span><span class="p">(</span><span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">d</span><span class="p">,</span> <span class="n">indent</span><span class="o">=</span><span class="mi">2</span><span class="p">))</span>
11812
11813 <span class="n">employees</span> <span class="o">=</span> <span class="n">employee_database</span><span class="o">.</span><span class="n">employees</span>
11814
11815 <span class="n">track</span><span class="p">(</span><span class="n">employees</span><span class="p">,</span> <span class="mi">40</span><span class="p">)</span>
11816 <span class="n">calculate_payroll</span><span class="p">(</span><span class="n">employees</span><span class="p">)</span>
11817
11818 <span class="n">temp_secretary</span> <span class="o">=</span> <span class="n">Employee</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>
11819 <span class="nb">print</span><span class="p">(</span><span class="s1">'Temporary Secretary:'</span><span class="p">)</span>
11820 <span class="n">print_dict</span><span class="p">(</span><span class="n">temp_secretary</span><span class="o">.</span><span class="n">to_dict</span><span class="p">())</span>
11821 </pre></div>
11822
11823 <p>You import the relevant functions from the <code>hr</code> and <code>productivity</code> modules, as well as the <code>employee_database</code> and <code>Employee</code> class. The program is cleaner because you exposed the required interface and encapsulated how objects are accessed.</p>
11824 <p>Notice that you can now create an <code>Employee</code> object directly just using its <code>id</code>. You can run the program to see its output:</p>
11825 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python program.py
11826
11827 <span class="go">Tracking Employee Productivity</span>
11828 <span class="go">==============================</span>
11829 <span class="go">Employee 1 - Mary Poppins:</span>
11830 <span class="go">- screams and yells for 40 hours.</span>
11831
11832 <span class="go">Employee 2 - John Smith:</span>
11833 <span class="go">- does paperwork for 40 hours.</span>
11834
11835 <span class="go">Employee 3 - Kevin Bacon:</span>
11836 <span class="go">- expends 40 hours on the phone.</span>
11837
11838 <span class="go">Employee 4 - Jane Doe:</span>
11839 <span class="go">- manufactures gadgets for 40 hours.</span>
11840
11841 <span class="go">Employee 5 - Robin Williams:</span>
11842 <span class="go">- does paperwork for 40 hours.</span>
11843
11844 <span class="go">Calculating Payroll</span>
11845 <span class="go">===================</span>
11846 <span class="go">Payroll for: 1 - Mary Poppins</span>
11847 <span class="go">- Check amount: 3000</span>
11848 <span class="go">- Sent to:</span>
11849 <span class="go">121 Admin Rd.</span>
11850 <span class="go">Concord, NH 03301</span>
11851
11852 <span class="go">Payroll for: 2 - John Smith</span>
11853 <span class="go">- Check amount: 1500</span>
11854 <span class="go">- Sent to:</span>
11855 <span class="go">67 Paperwork Ave</span>
11856 <span class="go">Manchester, NH 03101</span>
11857
11858 <span class="go">Payroll for: 3 - Kevin Bacon</span>
11859 <span class="go">- Check amount: 1800.0</span>
11860 <span class="go">- Sent to:</span>
11861 <span class="go">15 Rose St</span>
11862 <span class="go">Apt. B-1</span>
11863 <span class="go">Concord, NH 03301</span>
11864
11865 <span class="go">Payroll for: 4 - Jane Doe</span>
11866 <span class="go">- Check amount: 600</span>
11867 <span class="go">- Sent to:</span>
11868 <span class="go">39 Sole St.</span>
11869 <span class="go">Concord, NH 03301</span>
11870
11871 <span class="go">Payroll for: 5 - Robin Williams</span>
11872 <span class="go">- Check amount: 360</span>
11873 <span class="go">- Sent to:</span>
11874 <span class="go">99 Mountain Rd.</span>
11875 <span class="go">Concord, NH 03301</span>
11876
11877 <span class="go">Temporary Secretary:</span>
11878 <span class="go">{</span>
11879 <span class="go"> "id": "5",</span>
11880 <span class="go"> "name": "Robin Williams",</span>
11881 <span class="go"> "address": {</span>
11882 <span class="go"> "street": "99 Mountain Rd.",</span>
11883 <span class="go"> "street2": "",</span>
11884 <span class="go"> "city": "Concord",</span>
11885 <span class="go"> "state": "NH",</span>
11886 <span class="go"> "zipcode": "03301"</span>
11887 <span class="go"> }</span>
11888 <span class="go">}</span>
11889 </pre></div>
11890
11891 <p>The program works the same as before, but now you can see that a single <code>Employee</code> object can be created from its <code>id</code> and display its dictionary representation.</p>
11892 <p>Take a closer look at the <code>Employee</code> class:</p>
11893 <div class="highlight python"><pre><span></span><span class="c1"># In employees.py</span>
11894
11895 <span class="k">class</span> <span class="nc">Employee</span><span class="p">(</span><span class="n">AsDictionaryMixin</span><span class="p">):</span>
11896 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">):</span>
11897 <span class="bp">self</span><span class="o">.</span><span class="n">id</span> <span class="o">=</span> <span class="nb">id</span>
11898 <span class="n">info</span> <span class="o">=</span> <span class="n">employee_database</span><span class="o">.</span><span class="n">get_employee_info</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">id</span><span class="p">)</span>
11899 <span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">info</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'name'</span><span class="p">)</span>
11900 <span class="bp">self</span><span class="o">.</span><span class="n">address</span> <span class="o">=</span> <span class="n">get_employee_address</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">id</span><span class="p">)</span>
11901 <span class="bp">self</span><span class="o">.</span><span class="n">_role</span> <span class="o">=</span> <span class="n">get_role</span><span class="p">(</span><span class="n">info</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'role'</span><span class="p">))</span>
11902 <span class="bp">self</span><span class="o">.</span><span class="n">_payroll</span> <span class="o">=</span> <span class="n">get_policy</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">id</span><span class="p">)</span>
11903
11904 <span class="k">def</span> <span class="nf">work</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hours</span><span class="p">):</span>
11905 <span class="n">duties</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_role</span><span class="o">.</span><span class="n">perform_duties</span><span class="p">(</span><span class="n">hours</span><span class="p">)</span>
11906 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Employee </span><span class="si">{self.id}</span><span class="s1"> - </span><span class="si">{self.name}</span><span class="s1">:'</span><span class="p">)</span>
11907 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'- </span><span class="si">{duties}</span><span class="s1">'</span><span class="p">)</span>
11908 <span class="nb">print</span><span class="p">(</span><span class="s1">''</span><span class="p">)</span>
11909 <span class="bp">self</span><span class="o">.</span><span class="n">_payroll</span><span class="o">.</span><span class="n">track_work</span><span class="p">(</span><span class="n">hours</span><span class="p">)</span>
11910
11911 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
11912 <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_payroll</span><span class="o">.</span><span class="n">calculate_payroll</span><span class="p">()</span>
11913 </pre></div>
11914
11915 <p>The <code>Employee</code> class is a composite that contains multiple objects providing different functionality. It contains an <code>Address</code> that implements all the functionality related to where the employee lives.</p>
11916 <p><code>Employee</code> also contains a productivity role provided by the <code>productivity</code> module, and a payroll policy provided by the <code>hr</code> module. These two objects provide implementations that are leveraged by the <code>Employee</code> class to track work in the <code>.work()</code> method and to calculate the payroll in the <code>.calculate_payroll()</code> method.</p>
11917 <p>You are using composition in two different ways. The <code>Address</code> class provides additional data to <code>Employee</code> where the role and payroll objects provide additional behavior.</p>
11918 <p>Still, the relationship between <code>Employee</code> and those objects is loosely coupled, which provides some interesting capabilities that you’ll see in the next section.</p>
11919 <h3 id="composition-to-change-run-time-behavior">Composition to Change Run-Time Behavior</h3>
11920 <p>Inheritance, as opposed to composition, is a tightly couple relationship. With inheritance, there is only one way to change and customize behavior. Method overriding is the only way to customize the behavior of a base class. This creates rigid designs that are difficult to change.</p>
11921 <p>Composition, on the other hand, provides a loosely coupled relationship that enables flexible designs and can be used to change behavior at run-time.</p>
11922 <p>Imagine you need to support a long-term disability (LTD) policy when calculating payroll. The policy states that an employee on LTD should be paid 60% of their weekly salary assuming 40 hours of work.</p>
11923 <p>With an inheritance design, this can be a very difficult requirement to support. Adding it to the composition example is a lot easier. Let’s start by adding the policy class:</p>
11924 <div class="highlight python"><pre><span></span><span class="c1"># In hr.py</span>
11925
11926 <span class="k">class</span> <span class="nc">LTDPolicy</span><span class="p">:</span>
11927 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
11928 <span class="bp">self</span><span class="o">.</span><span class="n">_base_policy</span> <span class="o">=</span> <span class="kc">None</span>
11929
11930 <span class="k">def</span> <span class="nf">track_work</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hours</span><span class="p">):</span>
11931 <span class="bp">self</span><span class="o">.</span><span class="n">_check_base_policy</span><span class="p">()</span>
11932 <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_base_policy</span><span class="o">.</span><span class="n">track_work</span><span class="p">(</span><span class="n">hours</span><span class="p">)</span>
11933
11934 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
11935 <span class="bp">self</span><span class="o">.</span><span class="n">_check_base_policy</span><span class="p">()</span>
11936 <span class="n">base_salary</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_base_policy</span><span class="o">.</span><span class="n">calculate_payroll</span><span class="p">()</span>
11937 <span class="k">return</span> <span class="n">base_salary</span> <span class="o">*</span> <span class="mf">0.6</span>
11938
11939 <span class="k">def</span> <span class="nf">apply_to_policy</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">base_policy</span><span class="p">):</span>
11940 <span class="bp">self</span><span class="o">.</span><span class="n">_base_policy</span> <span class="o">=</span> <span class="n">base_policy</span>
11941
11942 <span class="k">def</span> <span class="nf">_check_base_policy</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
11943 <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">_base_policy</span><span class="p">:</span>
11944 <span class="k">raise</span> <span class="ne">RuntimeError</span><span class="p">(</span><span class="s1">'Base policy missing'</span><span class="p">)</span>
11945 </pre></div>
11946
11947 <p>Notice that <code>LTDPolicy</code> doesn’t inherit <code>PayrollPolicy</code>, but implements the same interface. This is because the implementation is completely different, so we don’t want to inherit any of the <code>PayrollPolicy</code> implementation.</p>
11948 <p>The <code>LTDPolicy</code> initializes <code>_base_policy</code> to <code>None</code>, and provides an internal <code>._check_base_policy()</code> method that raises an exception if the <code>._base_policy</code> has not been applied. Then, it provides a <code>.apply_to_policy()</code> method to assign the <code>_base_policy</code>.</p>
11949 <p>The public interface first checks that the <code>_base_policy</code> has been applied, and then implements the functionality in terms of that base policy. The <code>.track_work()</code> method just delegates to the base policy, and <code>.calculate_payroll()</code> uses it to calculate the <code>base_salary</code> and then return the 60%.</p>
11950 <p>You can now make a small change to the <code>Employee</code> class:</p>
11951 <div class="highlight python"><pre><span></span><span class="c1"># In employees.py</span>
11952
11953 <span class="k">class</span> <span class="nc">Employee</span><span class="p">(</span><span class="n">AsDictionaryMixin</span><span class="p">):</span>
11954 <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="p">):</span>
11955 <span class="bp">self</span><span class="o">.</span><span class="n">id</span> <span class="o">=</span> <span class="nb">id</span>
11956 <span class="n">info</span> <span class="o">=</span> <span class="n">employee_database</span><span class="o">.</span><span class="n">get_employee_info</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">id</span><span class="p">)</span>
11957 <span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">info</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'name'</span><span class="p">)</span>
11958 <span class="bp">self</span><span class="o">.</span><span class="n">address</span> <span class="o">=</span> <span class="n">get_employee_address</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">id</span><span class="p">)</span>
11959 <span class="bp">self</span><span class="o">.</span><span class="n">_role</span> <span class="o">=</span> <span class="n">get_role</span><span class="p">(</span><span class="n">info</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'role'</span><span class="p">))</span>
11960 <span class="bp">self</span><span class="o">.</span><span class="n">_payroll</span> <span class="o">=</span> <span class="n">get_policy</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">id</span><span class="p">)</span>
11961
11962
11963 <span class="k">def</span> <span class="nf">work</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hours</span><span class="p">):</span>
11964 <span class="n">duties</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_role</span><span class="o">.</span><span class="n">perform_duties</span><span class="p">(</span><span class="n">hours</span><span class="p">)</span>
11965 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Employee </span><span class="si">{self.id}</span><span class="s1"> - </span><span class="si">{self.name}</span><span class="s1">:'</span><span class="p">)</span>
11966 <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'- </span><span class="si">{duties}</span><span class="s1">'</span><span class="p">)</span>
11967 <span class="nb">print</span><span class="p">(</span><span class="s1">''</span><span class="p">)</span>
11968 <span class="bp">self</span><span class="o">.</span><span class="n">_payroll</span><span class="o">.</span><span class="n">track_work</span><span class="p">(</span><span class="n">hours</span><span class="p">)</span>
11969
11970 <span class="k">def</span> <span class="nf">calculate_payroll</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
11971 <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_payroll</span><span class="o">.</span><span class="n">calculate_payroll</span><span class="p">()</span>
11972
11973 <span class="k">def</span> <span class="nf">apply_payroll_policy</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">new_policy</span><span class="p">):</span>
11974 <span class="n">new_policy</span><span class="o">.</span><span class="n">apply_to_policy</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_payroll</span><span class="p">)</span>
11975 <span class="bp">self</span><span class="o">.</span><span class="n">_payroll</span> <span class="o">=</span> <span class="n">new_policy</span>
11976 </pre></div>
11977
11978 <p>You added an <code>.apply_payroll_policy()</code> method that applies the existing payroll policy to the new policy and then substitutes it. You can now modify the program to apply the policy to an <code>Employee</code> object:</p>
11979 <div class="highlight python"><pre><span></span><span class="c1"># In program.py</span>
11980
11981 <span class="kn">from</span> <span class="nn">hr</span> <span class="k">import</span> <span class="n">calculate_payroll</span><span class="p">,</span> <span class="n">LTDPolicy</span>
11982 <span class="kn">from</span> <span class="nn">productivity</span> <span class="k">import</span> <span class="n">track</span>
11983 <span class="kn">from</span> <span class="nn">employees</span> <span class="k">import</span> <span class="n">employee_database</span>
11984
11985 <span class="n">employees</span> <span class="o">=</span> <span class="n">employee_database</span><span class="o">.</span><span class="n">employees</span>
11986
11987 <span class="n">sales_employee</span> <span class="o">=</span> <span class="n">employees</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span>
11988 <span class="n">ltd_policy</span> <span class="o">=</span> <span class="n">LTDPolicy</span><span class="p">()</span>
11989 <span class="n">sales_employee</span><span class="o">.</span><span class="n">apply_payroll_policy</span><span class="p">(</span><span class="n">ltd_policy</span><span class="p">)</span>
11990
11991 <span class="n">track</span><span class="p">(</span><span class="n">employees</span><span class="p">,</span> <span class="mi">40</span><span class="p">)</span>
11992 <span class="n">calculate_payroll</span><span class="p">(</span><span class="n">employees</span><span class="p">)</span>
11993 </pre></div>
11994
11995 <p>The program accesses <code>sales_employee</code>, which is located at index <code>2</code>, creates the <code>LTDPolicy</code> object, and applies the policy to the employee. When <code>.calculate_payroll()</code> is called, the change is reflected. You can run the program to evaluate the output:</p>
11996 <div class="highlight sh"><pre><span></span><span class="gp">$</span> python program.py
11997
11998 <span class="go">Tracking Employee Productivity</span>
11999 <span class="go">==============================</span>
12000 <span class="go">Employee 1 - Mary Poppins:</span>
12001 <span class="go">- screams and yells for 40 hours.</span>
12002
12003 <span class="go">Employee 2 - John Smith:</span>
12004 <span class="go">- Does paperwork for 40 hours.</span>
12005
12006 <span class="go">Employee 3 - Kevin Bacon:</span>
12007 <span class="go">- Expends 40 hours on the phone.</span>
12008
12009 <span class="go">Employee 4 - Jane Doe:</span>
12010 <span class="go">- Manufactures gadgets for 40 hours.</span>
12011
12012 <span class="go">Employee 5 - Robin Williams:</span>
12013 <span class="go">- Does paperwork for 40 hours.</span>
12014
12015
12016 <span class="go">Calculating Payroll</span>
12017 <span class="go">===================</span>
12018 <span class="go">Payroll for: 1 - Mary Poppins</span>
12019 <span class="go">- Check amount: 3000</span>
12020 <span class="go">- Sent to:</span>
12021 <span class="go">121 Admin Rd.</span>
12022 <span class="go">Concord, NH 03301</span>
12023
12024 <span class="go">Payroll for: 2 - John Smith</span>
12025 <span class="go">- Check amount: 1500</span>
12026 <span class="go">- Sent to:</span>
12027 <span class="go">67 Paperwork Ave</span>
12028 <span class="go">Manchester, NH 03101</span>
12029
12030 <span class="go">Payroll for: 3 - Kevin Bacon</span>
12031 <span class="go">- Check amount: 1080.0</span>
12032 <span class="go">- Sent to:</span>
12033 <span class="go">15 Rose St</span>
12034 <span class="go">Apt. B-1</span>
12035 <span class="go">Concord, NH 03301</span>
12036
12037 <span class="go">Payroll for: 4 - Jane Doe</span>
12038 <span class="go">- Check amount: 600</span>
12039 <span class="go">- Sent to:</span>
12040 <span class="go">39 Sole St.</span>
12041 <span class="go">Concord, NH 03301</span>
12042
12043 <span class="go">Payroll for: 5 - Robin Williams</span>
12044 <span class="go">- Check amount: 360</span>
12045 <span class="go">- Sent to:</span>
12046 <span class="go">99 Mountain Rd.</span>
12047 <span class="go">Concord, NH 03301</span>
12048 </pre></div>
12049
12050 <p>The check amount for employee Kevin Bacon, who is the sales employee, is now for $1080 instead of $1800. That’s because the <code>LTDPolicy</code> has been applied to the salary.</p>
12051 <p>As you can see, you were able to support the changes just by adding a new policy and modifying a couple interfaces. This is the kind of flexibility that policy design based on composition gives you.</p>
12052 <h3 id="choosing-between-inheritance-and-composition-in-python_1">Choosing Between Inheritance and Composition in Python</h3>
12053 <p>Python, as an object oriented programming language, supports both inheritance and composition. You saw that inheritance is best used to model an <strong>is a</strong> relationship, whereas composition models a <strong>has a</strong> relationship.</p>
12054 <p>Sometimes, it’s hard to see what the relationship between two classes should be, but you can follow these guidelines:</p>
12055 <ul>
12056 <li>
12057 <p><strong>Use inheritance over composition in Python</strong> to model a clear <strong>is a</strong> relationship. First, justify the relationship between the derived class and its base. Then, reverse the relationship and try to justify it. If you can justify the relationship in both directions, then you should not use inheritance between them.</p>
12058 </li>
12059 <li>
12060 <p><strong>Use inheritance over composition in Python</strong> to leverage both the interface and implementation of the base class.</p>
12061 </li>
12062 <li>
12063 <p><strong>Use inheritance over composition in Python</strong> to provide <strong>mixin</strong> features to several unrelated classes when there is only one implementation of that feature.</p>
12064 </li>
12065 <li>
12066 <p><strong>Use composition over inheritance in Python</strong> to model a <strong>has a</strong> relationship that leverages the implementation of the component class.</p>
12067 </li>
12068 <li>
12069 <p><strong>Use composition over inheritance in Python</strong> to create components that can be reused by multiple classes in your Python applications.</p>
12070 </li>
12071 <li>
12072 <p><strong>Use composition over inheritance in Python</strong> to implement groups of behaviors and policies that can be applied interchangeably to other classes to customize their behavior.</p>
12073 </li>
12074 <li>
12075 <p><strong>Use composition over inheritance in Python</strong> to enable run-time behavior changes without affecting existing classes.</p>
12076 </li>
12077 </ul>
12078 <h2 id="conclusion">Conclusion</h2>
12079 <p>You explored <strong>inheritance and composition in Python</strong>. You learned about the type of relationships that inheritance and composition create. You also went through a series of exercises to understand how inheritance and composition are implemented in Python.</p>
12080 <p>In this article, you learned how to:</p>
12081 <ul>
12082 <li>Use inheritance to express an <strong>is a</strong> relationship between two classes</li>
12083 <li>Evaluate if inheritance is the right relationship</li>
12084 <li>Use multiple inheritance in Python and evaluate Python’s MRO to troubleshoot multiple inheritance problems</li>
12085 <li>Extend classes with mixins and reuse their implementation</li>
12086 <li>Use composition to express a <strong>has a</strong> relationship between two classes</li>
12087 <li>Provide flexible designs using composition</li>
12088 <li>Reuse existing code through policy design based on composition</li>
12089 </ul>
12090 <h2 id="recommended-reading">Recommended Reading</h2>
12091 <p>Here are some books and articles that further explore object oriented design and can be useful to help you understand the correct use of inheritance and composition in Python or other languages:</p>
12092 <ul>
12093 <li><a href="https://realpython.com/asins/B000SEIBB8">Design Patterns: Elements of Reusable Object-Oriented Software</a></li>
12094 <li><a href="https://realpython.com/asins/B00AA36RZY">Head First Design Patterns: A Brain-Friendly Guide</a></li>
12095 <li><a href="https://realpython.com/asins/B001GSTOAM">Clean Code: A Handbook of Agile Software Craftsmanship</a></li>
12096 <li><a href="https://en.wikipedia.org/wiki/SOLID">SOLID Principles</a></li>
12097 <li><a href="https://en.wikipedia.org/wiki/Liskov_substitution_principle">Liskov Substitution Principle</a></li>
12098 </ul>
12099 <hr />
12100 <p><em>[ Improve Your Python With π Python Tricks π β Get a short & sweet Python Trick delivered to your inbox every couple of days. <a href="https://realpython.com/python-tricks/?utm_source=realpython&utm_medium=rss&utm_campaign=footer">>> Click here to learn more and see examples</a> ]</em></p>#
12101 datePublished: #Wed Aug 7 14:00:00 2019#
12102 dateUpdated: #Wed Aug 7 14:00:00 2019#
12103 # Person begin ####################
12104 name: #Real Python#
12105 # Person end ######################
12106 # Item end ########################
12107 # Item begin ######################
12108 id: #https://realpython.com/courses/python-beginner-tips/#
12109 title: #11 Beginner Tips for Learning Python#
12110 link: #https://realpython.com/courses/python-beginner-tips/#
12111 description: #In this course, you'll see several learning strategies and tips that will help you jumpstart your journey towards becoming a rockstar Python programmer!#
12112 content: #<p>We are so excited that you have decided to embark on the journey of learning Python! One of the most common questions we receive from our readers is βWhatβs the best way to learn Python?β</p>
12113 <p>The first step in learning any programming language is making sure that you understand how to learn. Learning how to learn is arguably the most critical skill involved in computer programming.</p>
12114 <p>Why is knowing how to learn so important? Languages evolve, libraries are created, and tools are upgraded. Knowing how to learn will be essential to keeping up with these changes and becoming a successful programmer.</p>
12115 <p>In this course, you’ll see several learning strategies that will help you jumpstart your journey towards becoming a rockstar Python programmer!</p>
12116 <hr />
12117 <p><em>[ Improve Your Python With π Python Tricks π β Get a short & sweet Python Trick delivered to your inbox every couple of days. <a href="https://realpython.com/python-tricks/?utm_source=realpython&utm_medium=rss&utm_campaign=footer">>> Click here to learn more and see examples</a> ]</em></p>#
12118 datePublished: #Tue Aug 6 14:00:00 2019#
12119 dateUpdated: #Tue Aug 6 14:00:00 2019#
12120 # Person begin ####################
12121 name: #Real Python#
12122 # Person end ######################
12123 # Item end ########################
12124 # Item begin ######################
12125 id: #https://realpython.com/manage-users-in-django-admin/#
12126 title: #What You Need to Know to Manage Users in Django Admin#
12127 link: #https://realpython.com/manage-users-in-django-admin/#
12128 description: #In this Python tutorial, you'll learn what you need to know to manage users in Django admin. Out of the box, Django admin doesn't enforce special restrictions on the user admin. This can lead to dangerous scenarios that might compromise your system.#
12129 content: #<p>User management in Django admin is a tricky subject. If you enforce too many permissions, then you might interfere with day-to-day operations. If you allow for permissions to be granted freely without supervision, then you put your system at risk.</p>
12130 <p>Django provides a good authentication framework with tight integration to Django admin. Out of the box, Django admin does not enforce special restrictions on the user admin. This can lead to dangerous scenarios that might compromise your system.</p>
12131 <p>Did you know staff users that manage other users in the admin can edit their own permissions? Did you know they can also make themselves superusers? There is nothing in Django admin that prevents that, so it’s up to you!</p>
12132 <p><strong>By the end of this tutorial, you’ll know how to protect your system:</strong></p>
12133 <ul>
12134 <li><strong>Protect against permission escalation</strong> by preventing users from editing their own permissions</li>
12135 <li><strong>Keep permissions tidy and maintainable</strong> by only forcing users to manage permissions only using groups</li>
12136 <li><strong>Prevent permissions from leaking through custom actions</strong> by explicitly enforcing the necessary permissions</li>
12137 </ul>
12138 <div class="alert alert-primary" role="alert">
12139 <p><strong>Follow Along:</strong></p>
12140 <p>To follow along with this tutorial, it’s best to setup a small project to play with. If you aren’t sure how to do that, then check out <a href="https://realpython.com/get-started-with-django-1/#hello-world">Get Started With Django</a>.</p>
12141 <p>This tutorial also assumes a basic understanding of user management in Django. If you aren’t familiar with that, then check out <a href="https://docs.djangoproject.com/en/2.2/topics/auth/">the official documentation</a>.</p>
12142 </div>
12143 <div class="alert alert-warning" role="alert"><p><strong>Free Bonus:</strong> <a href="#" class="alert-link" data-toggle="modal" data-target="#modal-django-resources-learing-guide" data-focus="false">Click here to get access to a free Django Learning Resources Guide (PDF)</a> that shows you tips and tricks as well as common pitfalls to avoid when building Python + Django web applications.</p></div>
12144
12145 <h2 id="model-permissions">Model Permissions</h2>
12146 <p>Permissions are tricky. If you don’t set permissions, then you put your system at risk of intruders, data leaks, and human errors. If you abuse permissions or use them too much, then you risk interfering with day-to-day operations.</p>
12147 <p>Django comes with a built-in <a href="https://docs.djangoproject.com/en/2.2/topics/auth/">authentication system</a>. The authentication system includes users, groups, and permissions.</p>
12148 <p>When a model is created, Django will automatically create four <a href="https://docs.djangoproject.com/en/2.2/topics/auth/default/#default-permissions">default permissions</a> for the following actions:</p>
12149 <ol>
12150 <li><strong><code>add</code>:</strong> Users with this permission can add an instance of the model.</li>
12151 <li><strong><code>delete</code>:</strong> Users with this permission can delete an instance of the model.</li>
12152 <li><strong><code>change</code>:</strong> Users with this permission can update an instance of the model.</li>
12153 <li><strong><code>view</code>:</strong> Users with this permission can view instances of this model. This permission was a much anticipated one, and it was finally added in Django 2.1.</li>
12154 </ol>
12155 <p>Permission names follow a very specific naming convention: <code><app>.<action>_<modelname></code>.</p>
12156 <p>Let’s break that down:</p>
12157 <ul>
12158 <li><strong><code><app></code></strong> is the name of the app. For example, the <code>User</code> model is imported from the <code>auth</code> app (<code>django.contrib.auth</code>).</li>
12159 <li><strong><code><action></code></strong> is one of the actions above (<code>add</code>, <code>delete</code>, <code>change</code>, or <code>view</code>).</li>
12160 <li><strong><code><modelname></code></strong> is the name of the model, in all lowercase letters.</li>
12161 </ul>
12162 <p>Knowing this naming convention can help you manage permissions more easily. For example, the name of the permission to change a user is <code>auth.change_user</code>.</p>
12163 <h3 id="how-to-check-permissions">How to Check Permissions</h3>
12164 <p>Model permissions are granted to users or groups. To check if a user has a certain permission, you can do the following:</p>
12165 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">django.contrib.auth.models</span> <span class="k">import</span> <span class="n">User</span>
12166 <span class="gp">>>> </span><span class="n">u</span> <span class="o">=</span> <span class="n">User</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">create_user</span><span class="p">(</span><span class="n">username</span><span class="o">=</span><span class="s1">'haki'</span><span class="p">)</span>
12167 <span class="hll"><span class="gp">>>> </span><span class="n">u</span><span class="o">.</span><span class="n">has_perm</span><span class="p">(</span><span class="s1">'auth.change_user'</span><span class="p">)</span>
12168 </span><span class="go">False</span>
12169 </pre></div>
12170
12171 <p>It’s worth mentioning that <a href="https://github.com/django/django/blob/bf9e0e342da3ed2f74ee0ec34e75bdcbedde40a9/django/contrib/auth/models.py#L255"><code>.has_perm()</code> will always return <code>True</code></a> for active superuser, even if the permission doesn’t really exist:</p>
12172 <div class="highlight python repl"><span class="repl-toggle" title="Toggle REPL prompts and output">>>></span><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">django.contrib.auth.models</span> <span class="k">import</span> <span class="n">User</span>
12173 <span class="gp">>>> </span><span class="n">superuser</span> <span class="o">=</span> <span class="n">User</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">create_superuser</span><span class="p">(</span>
12174 <span class="gp">... </span> <span class="n">username</span><span class="o">=</span><span class="s1">'superhaki'</span><span class="p">,</span>
12175 <span class="gp">... </span> <span class="n">email</span><span class="o">=</span><span class="s1">'me@hakibenita.com'</span><span class="p">,</span>
12176 <span class="gp">... </span> <span class="n">password</span><span class="o">=</span><span class="s1">'secret'</span><span class="p">,</span>
12177 <span class="go">)</span>
12178 <span class="hll"><span class="gp">>>> </span><span class="n">superuser</span><span class="o">.</span><span class="n">has_perm</span><span class="p">(</span><span class="s1">'does.not.exist'</span><span class="p">)</span>
12179 </span><span class="go">True</span>
12180 </pre></div>
12181
12182 <p>As you can see, when you’re checking permissions for a superuser, the permissions are not really being checked.</p>
12183 <h3 id="how-to-enforce-permissions">How to Enforce Permissions</h3>
12184 <p>Django models don’t enforce permissions themselves. The only place permissions are enforced out of the box by default is Django Admin.</p>
12185 <p>The reason models don’t enforce permissions is that, normally, the model is unaware of the user performing the action. In Django apps, the user is usually obtained from the request. This is why, most of the time, permissions are enforced at the view layer.</p>
12186 <p>For example, to prevent a user without view permissions on the <code>User</code> model from accessing a view that shows user information, do the following:</p>
12187 <div class="highlight python"><pre><span></span><span class="kn">from</span> <span class="nn">django.core.exceptions</span> <span class="k">import</span> <span class="n">PermissionDenied</span>
12188
12189 <span class="k">def</span> <span class="nf">users_list_view</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
12190 <span class="k">if</span> <span class="ow">not</span> <span class="n">request</span><span class="o">.</span><span class="n">user</span><span class="o">.</span><span class="n">has_perm</span><span class="p">(</span><span class="s1">'auth.view_user'</span><span class="p">):</span>
12191 <span class="k">raise</span> <span class="n">PermissionDenied</span><span class="p">()</span>
12192 </pre></div>
12193
12194 <p>If the user making the request logged in and was authenticated, then <a href="https://docs.djangoproject.com/en/2.2/ref/request-response/#django.http.HttpRequest.user"><code>request.user</code></a> will hold an instance of <code>User</code>. If the user did not login, then <code>request.user</code> will be an instance of <a href="https://docs.djangoproject.com/en/2.2/ref/contrib/auth/#anonymoususer-object"><code>AnonymousUser</code></a>. This is a special object used by Django to indicate an unauthenticated user. Using <code>has_perm</code> on <code>AnonymousUser</code> will always return <code>False</code>.</p>
12195 <p>If the user making the request doesn’t have the <code>view_user</code> permission, then you raise a <code>PermissionDenied</code> exception, and a response with status <code>403</code> is returned to the client.</p>
12196 <p>To make it easier to enforce permissions in views, Django provides a shortcut <a href="https://realpython.com/primer-on-python-decorators/">decorator</a> called <a href="https://docs.djangoproject.com/en/2.2/topics/auth/default/#the-permission-required-decorator"><code>permission_required</code></a> that does the same thing:</p>
12197 <div class="highlight python"><pre><span></span><span class="kn">from</span> <span class="nn">django.contrib.auth.decorators</span> <span class="k">import</span> <span class="n">permission_required</span>
12198
12199 <span class="nd">@permission_required</span><span class="p">(</span><span class="s1">'auth.view_user'</span><span class="p">)</span>
12200 <span class="k">def</span> <span class="nf">users_list_view</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
12201 <span class="k">pass</span>
12202 </pre></div>
12203
12204 <p>To enforce permissions in templates, you can access the current user permissions through a special template variable called <a href="https://docs.djangoproject.com/en/2.2/topics/auth/default/#permissions"><code>perms</code></a>. For example, if you want to show a delete button only to users with delete permission, then do the following:</p>
12205 <div class="highlight htmldjango"><pre><span></span><span class="cp">{%</span> <span class="k">if</span> <span class="nv">perms.auth.delete_user</span> <span class="cp">%}</span>
12206 <span class="p"><</span><span class="nt">button</span><span class="p">></span>Delete user!<span class="p"></</span><span class="nt">button</span><span class="p">></span>
12207 <span class="cp">{%</span> <span class="k">endif</span> <span class="cp">%}</span>
12208 </pre></div>
12209
12210 <p>Some popular third party apps such as the <a href="https://www.django-rest-framework.org/">Django rest framework</a> also provide <a href="https://www.django-rest-framework.org/api-guide/permissions/#djangomodelpermissions">useful integration with Django model permissions</a>.</p>
12211 <h3 id="django-admin-and-model-permissions">Django Admin and Model Permissions</h3>
12212 <p>Django admin has a very <a href="https://docs.djangoproject.com/en/2.2/topics/auth/default/#permissions-and-authorization">tight integration</a> with the built-in authentication system, and model permissions in particular. Out of the box, Django admin is enforcing model permissions:</p>
12213 <ul>
12214 <li>If the user has no permissions on a model, then they won’t be able to see it or access it in the admin.</li>
12215 <li>If the user has view and change permissions on a model, then they will be able to view and update instances, but they won’t be able to add new instances or delete existing ones.</li>
12216 </ul>
12217 <p>With proper permissions in place, admin users are less likely to make mistakes, and intruders will have a harder time causing harm.</p>
12218 <h2 id="implement-custom-business-roles-in-django-admin">Implement Custom Business Roles in Django Admin</h2>
12219 <p>One of the most vulnerable places in every app is the authentication system. In Django apps, this is the <code>User</code> model. So, to better protect your app, you are going to start with the <code>User</code> model.</p>
12220 <p>First, you need to take control over the <code>User</code> model admin page. Django already comes with a very nice admin page to manage users. To take advantage of that great work, you are going to extend the built-in <code>User</code> admin model.</p>
12221 <h3 id="setup-a-custom-user-admin">Setup: A Custom User Admin</h3>
12222 <p>To provide a custom admin for the <code>User</code> model, you need to unregister the existing model admin provided by Django, and register one of your own:</p>
12223 <div class="highlight python"><pre><span></span><span class="kn">from</span> <span class="nn">django.contrib</span> <span class="k">import</span> <span class="n">admin</span>
12224 <span class="kn">from</span> <span class="nn">django.contrib.auth.models</span> <span class="k">import</span> <span class="n">User</span>
12225 <span class="kn">from</span> <span class="nn">django.contrib.auth.admin</span> <span class="k">import</span> <span class="n">UserAdmin</span>
12226
12227 <span class="c1"># Unregister the provided model admin</span>
12228 <span class="n">admin</span><span class="o">.</span><span class="n">site</span><span class="o">.</span><span class="n">unregister</span><span class="p">(</span><span class="n">User</span><span class="p">)</span>
12229
12230 <span class="c1"># Register out own model admin, based on the default UserAdmin</span>
12231 <span class="nd">@admin</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">User</span><span class="p">)</span>
12232 <span class="k">class</span> <span class="nc">CustomUserAdmin</span><span class="p">(</span><span class="n">UserAdmin</span><span class="p">):</span>
12233 <span class="k">pass</span>
12234 </pre></div>
12235
12236 <p>Your <code>CustomUserAdmin</code> is extending Django’s <code>UserAdmin</code>. You did that so you can take advantage of all the work already done by the Django developers.</p>
12237 <p>At this point, if you log into your Django admin at <code>http://127.0.0.1:8000/admin/auth/user</code>, you should see the user admin unchanged:</p>
12238 <p><a href="https://files.realpython.com/media/django-bare-boned-user-admin.4ac55297d529.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/django-bare-boned-user-admin.4ac55297d529.png" width="861" height="671" srcset="https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/django-bare-boned-user-admin.4ac55297d529.png&w=215&sig=602db70adb2b1b9f4747c90ae4ba0c6e4860f4ad 215w, https://robocrop.realpython.net/?url=https%3A//files.realpython.com/media/django-bare-boned-user-admin.4ac55297d529.png&w=430&sig=c425295f2d8e0abb534dfe50a5f84dc40876726b 430w, https://files.realpython.com/media/django-bare-boned-user-admin.4ac55297d529.png 861w" sizes="75vw" alt="Django bare boned user admin"/></a></p>
12239 <p>By extending <code>UserAdmin</code>, you are able to use all the built-in features provided by Django admin.</p>
12240 <h3 id="prevent-update-of-fields">Prevent Update of Fields</h3>
12241 <p>Unattended admin forms are a prime candidate for horrible mistakes. A staff user can easily update a model instance through the admin in a way the app does not expect. Most of the time, the user won’t even notice something is wrong. Such mistakes are usually very hard to track down and fix.</p>
12242 <p>To prevent such mistakes from happening, you can prevent admin users from modifying certain fields in the model.</p>
12243 <p>If you want to prevent any user, including superusers, from updating a field, you can mark the field as read only. For example, the field <code>date_joined</code> is set when a user registers. This information should never be changed by any user, so you mark it as read only:</p>
12244 <div class="highlight python"><pre><span></span><span class="kn">from</span> <span class="nn">django.contrib</span> <span class="k">import</span> <span class="n">admin</span>
12245 <span class="kn">from</span> <span class="nn">django.contrib.auth.models</span> <span class="k">import</span> <span class="n">User</span>
12246 <span class="kn">from</span> <span class="nn">django.contrib.auth.admin</span> <span class="k">import</span> <span class="n">UserAdmin</span>
12247
12248 <span class="nd">@admin</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">User</span><span class="p">)</span>
12249 <span class="k">class</span> <span class="nc">CustomUserAdmin</span><span class="p">(</span><span class="n">UserAdmin</span><span class="p">):</span>
12250 <span class="n">readonly_fields</span> <span class="o">=</span> <span class="p">[</span>
12251 <span class="s1">'date_joined'</span><span class="p">,</span>
12252 <span class="p">]</span>
12253 </pre></div>
12254
12255 <p>When a field is added to <code>readonly_fields</code>, it will not be editable in the admin default change form. When a field is marked as read only, Django will render the input element as disabled.</p>
12256 <p>But, what if you want to prevent only some users from updating a field?</p>
12257 <h3 id="conditionally-prevent-update-of-fields">Conditionally Prevent Update of Fields</h3>
12258 <p>Sometimes it’s useful to update fields directly in the admin. But you don’t want to let any user do it: you want to allow only superusers to do it.</p>
12259 <p>Let’s say you want to prevent non-superusers from changing a user’s username. To do that, you need to modify the change form generated by Django, and disable the username field based on the current user:</p>
12260 <div class="highlight python"><pre><span></span><span class="kn">from</span> <span class="nn">django.contrib</span> <span class="k">import</span> <span class="n">admin</span>
12261 <span class="kn">from</span> <span class="nn">django.contrib.auth.models</span> <span class="k">import</span> <span class="n">User</span>
12262 <span class="kn">from</span> <span class="nn">django.contrib.auth.admin</span> <span class="k">import</span> <span class="n">UserAdmin</span>
12263
12264 <span class="nd">@admin</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">User</span><span class="p">)</span>
12265 <span class="k">class</span> <span class="nc">CustomUserAdmin</span><span class="p">(</span><span class="n">UserAdmin</span><span class="p">):</span>
12266 <span class="k">def</span> <span class="nf">get_form</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">request</span><span class="p">,</span> <span class="n">obj</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
12267 <span class="n">form</span> <span class="o">=</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">get_form</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="n">obj</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
12268 <span class="n">is_superuser</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">user</span><span class="o">.</span><span class="n">is_superuser</span>
12269
12270 <span class="hll"> <span class="k">if</span> <span class="ow">not</span> <span class="n">is_superuser</span><span class="p">:</span>
12271 </span><span class="hll"> <span class="n">form</span><span class="o">.</span><span class="n">base_fields</span><span class="p">[</span><span class="s1">'username'</span><span class="p">]</span><span class="o">.</span><span class="n">disabled</span> <span class="o">=</span> <span class="kc">True</span>
12272 </span>
12273 <span class="k">return</span> <span class="n">form</span>
12274 </pre></div>
12275
12276 <p>Let’s break it down:</p>
12277 <ul>
12278 <li>To make adjustments to the form, you override <a href="https://docs.djangoproject.com/en/2.1/ref/contrib/admin/#django.contrib.admin.ModelAdmin.get_form"><code>get_form()</code></a>. This function is used by Django to generate a default change form for a model.</li>
12279 <li>To conditionally disable the field, you first fetch the default form generated by Django, and then if the user is not a superuser, disable the username field.</li>
12280 </ul>
12281 <p>Now, when a non-superuser tries to edit a user, the username field will be disabled. Any attempt to modify the username through Django Admin will fail. When a superuser tries to edit the user, the username field will be editable and behave as expected.</p>
12282 <h3 id="prevent-non-superusers-from-granting-superuser-rights">Prevent Non-Superusers From Granting Superuser Rights</h3>
12283 <p>Superuser is a very strong permission that should not be granted lightly. However, any user with a change permission on the <code>User</code> model can make any user a superuser, including themselves. This goes against the whole purpose of the permission system, so you want to close this hole.</p>
12284 <p>Based on the previous example, to prevent non-superusers from making themselves superusers, you add the following restriction:</p>
12285 <div class="highlight python"><pre><span></span><span class="kn">from</span> <span class="nn">typing</span> <span class="k">import</span> <span class="n">Set</span>
12286
12287 <span class="kn">from</span> <span class="nn">django.contrib</span> <span class="k">import</span> <span class="n">admin</span>
12288 <span class="kn">from</span> <span class="nn">django.contrib.auth.models</span> <span class="k">import</span> <span class="n">User</span>
12289 <span class="kn">from</span> <span class="nn">django.contrib.auth.admin</span> <span class="k">import</span> <span class="n">UserAdmin</span>
12290
12291 <span class="nd">@admin</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">User</span><span class="p">)</span>
12292 <span class="k">class</span> <span class="nc">CustomUserAdmin</span><span class="p">(</span><span class="n">UserAdmin</span><span class="p">):</span>
12293 <span class="k">def</span> <span class="nf">get_form</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">request</span><span class="p">,</span> <span class="n">obj</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
12294 <span class="n">form</span> <span class="o">=</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">get_form</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="n">obj</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
12295 <span class="n">is_superuser</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">user</span><span class="o">.</span><span class="n">is_superuser</span>
12296 <span class="n">disabled_fields</span> <span class="o">=</span> <span class="nb">set</span><span class="p">()</span> <span class="c1"># type: Set[str]</span>
12297
12298 <span class="k">if</span> <span class="ow">not</span> <span class="n">is_superuser</span><span class="p">:</span>
12299 <span class="n">disabled_fields</span> <span class="o">|=</span> <span class="p">{</span>
12300 <span class="s1">'username'</span><span class="p">,</span>
12301 <span class="hll"> <span class="s1">'is_superuser'</span><span class="p">,</span>
12302 </span> <span class="p">}</span>
12303
12304 <span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">disabled_fields</span><span class="p">:</span>
12305 <span class="k">if</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">form</span><span class="o">.</span><span class="n">base_fields</span><span class="p">:</span>
12306 <span class="n">form</span><span class="o">.</span><span class="n">base_fields</span><span class="p">[</span><span class="n">f</span><span class="p">]</span><span class="o">.</span><span class="n">disabled</span> <span class="o">=</span> <span class="kc">True</span>
12307
12308 <span class="k">return</span> <span class="n">form</span>
12309 </pre></div>
12310
12311 <p>In addition to the previous example, you made the following additions:</p>
12312 <ol>
12313 <li>
12314 <p>You initialized an empty set <code>disabled_fields</code> that will hold the fields to disable. <code>set</code> is a data structure that holds unique values. It makes sense to use a set in this case, because you only need to disable a field once. The operator <code>|=</code> is used to perform an in-place <code>OR</code> update. For more information about sets, check out <a href="https://realpython.com/python-sets/">Sets in Python</a>.</p>
12315 </li>
12316 <li>
12317 <p>Next, if the user is a superuser, you add two fields to the set (<code>username</code> from the previous example, and <code>is_superuser</code>). They will prevent non-superusers from making themselves superusers.</p>
12318 </li>
12319 <li>
12320 <p>Lastly, you iterate over the fields in the set, mark all of them as disabled, and return the form.</p>
12321 </li>
12322 </ol>
12323 <p><strong>Django User Admin Two-Step Form</strong></p>
12324 <p>When you create a new user in Django admin, you go through a two-step form. In the first form, you fill in the username and password. In the second form, you update the rest of the fields.</p>
12325 <p>This two-step process is unique to the <code>User</code> model. To accommodate this unique process, you must verify that the field exists before you try to disable it. Otherwise, you might get a <code>KeyError</code>. This is not necessary if you customize other model admins.</p>
12326 <p>For more information about <code>KeyError</code>, check out <a href="https://realpython.com/python-keyerror/">Python KeyError Exceptions and How to Handle Them</a>.</p>
12327 <h3 id="grant-permissions-only-using-groups">Grant Permissions Only Using Groups</h3>
12328 <p>The way permissions are managed is very specific to each team, product, and company. I found that it’s easier to manage permissions in groups. In my own projects, I create groups for support, content editors, analysts, and so on. I found that managing permissions at the user level can be a real hassle. When new models are added, or when business requirements change, it’s tedious to update each individual user.</p>
12329 <p>To manage permissions only using groups, you need to prevent users from granting permissions to specific users. Instead, you want to only allow associating users to groups. To do that, disable the field <code>user_permissions</code> for all non-superusers:</p>
12330 <div class="highlight python"><pre><span></span><span class="kn">from</span> <span class="nn">typing</span> <span class="k">import</span> <span class="n">Set</span>
12331
12332 <span class="kn">from</span> <span class="nn">django.contrib</span> <span class="k">import</span> <span class="n">admin</span>
12333 <span class="kn">from</span> <span class="nn">django.contrib.auth.models</span> <span class="k">import</span> <span class="n">User</span>
12334 <span class="kn">from</span> <span class="nn">django.contrib.auth.admin</span> <span class="k">import</span> <span class="n">UserAdmin</span>
12335
12336 <span class="nd">@admin</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">User</span><span class="p">)</span>
12337 <span class="k">class</span> <span class="nc">CustomUserAdmin</span><span class="p">(</span><span class="n">UserAdmin</span><span class="p">):</span>
12338 <span class="k">def</span> <span class="nf">get_form</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">request</span><span class="p">,</span> <span class="n">obj</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
12339 <span class="n">form</span> <span class="o">=</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">get_form</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="n">obj</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
12340 <span class="n">is_superuser</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">user</span><span class="o">.</span><span class="n">is_superuser</span>
12341 <span class="n">disabled_fields</span> <span class="o">=</span> <span class="nb">set</span><span class="p">()</span> <span class="c1"># type: Set[str]</span>
12342
12343 <span class="k">if</span> <span class="ow">not</span> <span class="n">is_superuser</span><span class="p">:</span>
12344 <span class="n">disabled_fields</span> <span class="o">|=</span> <span class="p">{</span>
12345 <span class="s1">'username'</span><span class="p">,</span>
12346 <span class="s1">'is_superuser'</span><span class="p">,</span>
12347 <span class="hll"> <span class="s1">'user_permissions'</span><span class="p">,</span>
12348 </span> <span class="p">}</span>
12349
12350 <span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">disabled_fields</span><span class="p">:</span>
12351 <span class="k">if</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">form</span><span class="o">.</span><span class="n">base_fields</span><span class="p">:</span>
12352 <span class="n">form</span><span class="o">.</span><span class="n">base_fields</span><span class="p">[</span><span class="n">f</span><span class="p">]</span><span class="o">.</span><span class="n">disabled</span> <span class="o">=</span> <span class="kc">True</span>
12353
12354 <span class="k">return</span> <span class="n">form</span>
12355 </pre></div>
12356
12357 <p>You used the exact same technique as in the previous sections to implement another business rule. In the next sections, you’re going to implement more complex business rules to protect your system.</p>
12358 <h3 id="prevent-non-superusers-from-editing-their-own-permissions">Prevent Non-Superusers From Editing Their Own Permissions</h3>
12359 <p>Strong users are often a weak spot. They possess strong permissions, and the potential damage they can cause is significant. To prevent permission escalation in case of intrusion, you can prevent users from editing their own permissions:</p>
12360 <div class="highlight python"><pre><span></span><span class="kn">from</span> <span class="nn">typing</span> <span class="k">import</span> <span class="n">Set</span>
12361
12362 <span class="kn">from</span> <span class="nn">django.contrib</span> <span class="k">import</span> <span class="n">admin</span>
12363 <span class="kn">from</span> <span class="nn">django.contrib.auth.models</span> <span class="k">import</span> <span class="n">User</span>
12364 <span class="kn">from</span> <span class="nn">django.contrib.auth.admin</span> <span class="k">import</span> <span class="n">UserAdmin</span>
12365
12366 <span class="nd">@admin</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">User</span><span class="p">)</span>
12367 <span class="k">class</span> <span class="nc">CustomUserAdmin</span><span class="p">(</span><span class="n">UserAdmin</span><span class="p">):</span>
12368 <span class="k">def</span> <span class="nf">get_form</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">request</span><span class="p">,</span> <span class="n">obj</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
12369 <span class="n">form</span> <span class="o">=</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">get_form</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="n">obj</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
12370 <span class="n">is_superuser</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">user</span><span class="o">.</span><span class="n">is_superuser</span>
12371 <span class="n">disabled_fields</span> <span class="o">=</span> <span class="nb">set</span><span class="p">()</span> <span class="c1"># type: Set[str]</span>
12372
12373 <span class="k">if</span> <span class="ow">not</span> <span class="n">is_superuser</span><span class="p">:</span>
12374 <span class="n">disabled_fields</span> <span class="o">|=</span> <span class="p">{</span>
12375 <span class="s1">'username'</span><span class="p">,</span>
12376 <span class="s1">'is_superuser'</span><span class="p">,</span>
12377 <span class="s1">'user_permissions'</span><span class="p">,</span>
12378 <span class="p">}</span>
12379
12380 <span class="c1"># Prevent non-superusers from editing their own permissions</span>
12381 <span class="k">if</span> <span class="p">(</span>
12382 <span class="hll"> <span class="ow">not</span> <span class="n">is_superuser</span>
12383 </span><span class="hll"> <span class="ow">and</span> <span class="n">obj</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span>
12384 </span><span class="hll"> <span class="ow">and</span> <span class="n">obj</span> <span class="o">==</span> <span class="n">request</span><span class="o">.</span><span class="n">user</span>
12385 </span> <span class="p">):</span>
12386 <span class="n">disabled_fields</span> <span class="o">|=</span> <span class="p">{</span>
12387 <span class="s1">'is_staff'</span><span class="p">,</span>
12388 <span class="s1">'is_superuser'</span><span class="p">,</span>
12389 <span class="s1">'groups'</span><span class="p">,</span>
12390 <span class="s1">'user_permissions'</span><span class="p">,</span>
12391 <span class="p">}</span>
12392
12393 <span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">disabled_fields</span><span class="p">:</span>
12394 <span class="k">if</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">form</span><span class="o">.</span><span class="n">base_fields</span><span class="p">:</span>
12395 <span class="n">form</span><span class="o">.</span><span class="n">base_fields</span><span class="p">[</span><span class="n">f</span><span class="p">]</span><span class="o">.</span><span class="n">disabled</span> <span class="o">=</span> <span class="kc">True</span>
12396
12397 <span class="k">return</span> <span class="n">form</span>
12398 </pre></div>
12399
12400 <p>The argument <code>obj</code> is the instance of the object you are currently operating on:</p>
12401 <ul>
12402 <li><strong>When <code>obj</code> is None</strong>, the form is used to create a new user.</li>
12403 <li><strong>When <code>obj</code> is not <code>None</code></strong>, the form is used to edit an existing user.</li>
12404 </ul>
12405 <p>To check if the user making the request is operating on themselves, you compare <code>request.user</code> with <code>obj</code>. Because this is the user admin, <code>obj</code> is either an instance of <code>User</code>, or <code>None</code>. When the user making the request, <code>request.user</code>, is equal to <code>obj</code>, then it means that the user is updating themselves. In this case, you disable all sensitive fields that can be used to gain permissions.</p>
12406 <p>The ability to customize the form based on the object is very useful. It can be used to implement elaborate business roles.</p>
12407 <h3 id="override-permissions">Override Permissions</h3>
12408 <p>It can sometimes be useful to completely override the permissions in Django admin. A common scenario is when you use permissions in other places, and you don’t want staff users to make changes in the admin.</p>
12409 <p>Django uses <a href="https://docs.djangoproject.com/en/2.1/ref/contrib/admin/#django.contrib.admin.ModelAdmin.has_view_permission">hooks for the four built-in permissions</a>. Internally, the hooks use the current user’s permissions to make a decision. You can override these hooks, and provide a different decision.</p>
12410 <p>To prevent staff users from deleting a model instance, regardless of their permissions, you can do the following:</p>
12411 <div class="highlight python"><pre><span></span><span class="kn">from</span> <span class="nn">django.contrib</span> <span class="k">import</span> <span class="n">admin</span>
12412 <span class="kn">from</span> <span class="nn">django.contrib.auth.models</span> <span class="k">import</span> <span class="n">User</span>
12413 <span class="kn">from</span> <span class="nn">django.contrib.auth.admin</span> <span class="k">import</span> <span class="n">UserAdmin</span>
12414
12415 <span class="nd">@admin</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">User</span><span class="p">)</span>
12416 <span class="k">class</span> <span class="nc">CustomUserAdmin</span><span class="p">(</span><span class="n">UserAdmin</span><span class="p">):</span>
12417 <span class="hll"> <span class="k">def</span> <span class="nf">has_delete_permission</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">request</span><span class="p">,</span> <span class="n">obj</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
12418 </span><span class="hll"> <span class="k">return</span> <span class="kc">False</span>
12419 </span></pre></div>
12420
12421 <p>Just like with <code>get_form()</code>, <code>obj</code> is the instance you currently operate on:</p>
12422 <ul>
12423 <li><strong>When <code>obj</code> is <code>None</code></strong>, the user requested the list view.</li>
12424 <li><strong>When <code>obj</code> is not <code>None</code></strong>, the user requested the change view of a specific instance.</li>
12425 </ul>
12426 <p>Having the instance of the object in this hook is very useful for implementing object-level permissions for different types of actions. Here are other use cases:</p>
12427 <ul>
12428 <li>Preventing changes during business hours</li>
12429 <li>Implementing object-level permissions</li>
12430 </ul>
12431 <h3 id="restrict-access-to-custom-actions">Restrict Access to Custom Actions</h3>
12432 <p><a href="https://docs.djangoproject.com/en/2.2/ref/contrib/admin/actions/#adding-actions-to-the-modeladmin">Custom admin actions</a> require special attention. Django is not familiar with them, so it can’t restrict access to them by default. A custom action will be accessible to any admin user with any permission on the model.</p>
12433 <p>To illustrate, add a handy admin action to mark multiple users as active:</p>
12434 <div class="highlight python"><pre><span></span><span class="kn">from</span> <span class="nn">django.contrib</span> <span class="k">import</span> <span class="n">admin</span>
12435 <span class="kn">from</span> <span class="nn">django.contrib.auth.models</span> <span class="k">import</span> <span class="n">User</span>
12436 <span class="kn">from</span> <span class="nn">django.contrib.auth.admin</span> <span class="k">import</span> <span class="n">UserAdmin</span>
12437
12438 <span class="nd">@admin</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">User</span><span class="p">)</span>
12439 <span class="k">class</span> <span class="nc">CustomUserAdmin</span><span class="p">(</span><span class="n">UserAdmin</span><span class="p">):</span>
12440 <span class="n">actions</span> <span class="o">=</span> <span class="p">[</span>
12441 <span class="s1">'activate_users'</span><span class="p">,</span>
12442 <span class="p">]</span>
12443
12444 <span class="k">def</span> <span class="nf">activate_users</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">request</span><span class="p">,</span> <span class="n">queryset</span><span class="p">):</span>
12445 <span class="n">cnt</span> <span class="o">=</span> <span class="n">queryset</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">is_active</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">is_active</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
12446 <span class="bp">self</span><span class="o">.</span><span class="n">message_user</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="s1">'Activated </span><span class="si">{}</span><span class="s1"> users.'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">cnt</span><span class="p">))</span>
12447 <span class="n">activate_users</span><span class="o">.</span><span class="n">short_description</span> <span class="o">=</span> <span class="s1">'Activate Users'</span> <span class="c1"># type: ignore</span>
12448 </pre></div>
12449
12450 <p>Using this action, a staff user can mark one or more users, and activate them all at once. This is useful in all sorts of cases, such as if you had a bug in the registration process and needed to activate users in bulk.</p>
12451 <p>This action updates user information, so you want only users with change permissions to be able to use it.</p>
12452 <p>Django admin uses an internal function to get actions. To hide <code>activate_users()</code> from users without change permission, override <code>get_actions()</code>:</p>
12453 <div class="highlight python"><pre><span></span><span class="kn">from</span> <span class="nn">django.contrib</span> <span class="k">import</span> <span class="n">admin</span>
12454 <span class="kn">from</span> <span class="nn">django.contrib.auth.models</span> <span class="k">import</span> <span class="n">User</span>
12455 <span class="kn">from</span> <span class="nn">django.contrib.auth.admin</span> <span class="k">import</span> <span class="n">UserAdmin</span>
12456
12457 <span class="nd">@admin</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">User</span><span class="p">)</span>
12458 <span class="k">class</span> <span class="nc">CustomUserAdmin</span><span class="p">(</span><span class="n">UserAdmin</span><span class="p">):</span>
12459 <span class="n">actions</span> <span class="o">=</span> <span class="p">[</span>
12460 <span class="s1">'activate_users'</span><span class="p">,</span>
12461 <span class="p">]</span>
12462
12463 <span class="k">def</span> <span class="nf">activate_users</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">request</span><span class="p">,</span> <span class="n">queryset</span><span class="p">):</span>
12464 <span class="k">assert</span> <span class="n">request</span><span class="o">.</span><span class="n">user</span><span class="o">.</span><span class="n">has_perm</span><span class="p">(</span><span class="s1">'auth.change_user'</span><span class="p">)</span>
12465 <span class="n">cnt</span> <span class="o">=</span> <span class="n">queryset</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">is_active</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">is_active</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
12466 <span class="bp">self</span><span class="o">.</span><span class="n">message_user</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="s1">'Activated </span><span class="si">{}</span><span class="s1"> users.'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">cnt</span><span class="p">))</span>
12467 <span class="n">activate_users</span><span class="o">.</span><span class="n">short_description</span> <span class="o">=</span> <span class="s1">'Activate Users'</span> <span class="c1"># type: ignore</span>
12468
12469 <span class="k">def</span> <span class="nf">get_actions</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">request</span><span class="p">):</span>
12470 <span class="n">actions</span> <span class="o">=</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">get_actions</span><span class="p">(</span><span class="n">request</span><span class="p">)</span>
12471 <span class="hll"> <span class="k">if</span> <span class="ow">not</span> <span class="n">request</span><span class="o">.</span><span class="n">user</span><span class="o">.</span><span class="n">has_perm</span><span class="p">(</span><span class="s1">'auth.change_user'</span><span class="p">):</span>
12472 </span><span class="hll"> <span class="k">del</span> <span class="n">actions</span><span class="p">[</span><span class="s1">'activate_users'</span><span class="p">]</span>
12473 </span> <span class="k">return</span> <span class="n">actions</span>
12474 </pre></div>
12475
12476 <p><code>get_actions()</code> returns an <code>OrderedDict</code>. The key is the name of the action, and the value is the action function. To adjust the return value, you override the function, fetch the original value, and depending on the user permissions, remove the custom action <code>activate_users</code> from the <code>dict</code>. To be on the safe side, you assert the user permission in the action as well.</p>
12477 <p>For staff users without <code>change_user()</code> permissions, the action <code>activate_users</code> will not appear in the actions dropdown.</p>
12478 <h2 id="conclusion">Conclusion</h2>
12479 <p>Django admin is a great tool for managing a Django project. Many teams rely on it to stay productive in managing day-to-day operations. If you use Django admin to perform operations on models, then it’s important to be aware of permissions. The techniques described in this article are useful for any model admin, not just the <code>User</code> model.</p>
12480 <p>In this tutorial, you protected your system by making the following adjustments in Django Admin:</p>
12481 <ul>
12482 <li>You <strong>protected against permission escalation</strong> by preventing users from editing their own permissions.</li>
12483 <li>You <strong>kept permissions tidy and maintainable</strong> by only forcing users to manage permissions only using groups.</li>
12484 <li>You <strong>prevented permissions from leaking through custom actions</strong> by explicitly enforcing the necessary permissions.</li>
12485 </ul>
12486 <p>Your <code>User</code> model admin is now much safer than when you started!</p>
12487 <hr />
12488 <p><em>[ Improve Your Python With π Python Tricks π β Get a short & sweet Python Trick delivered to your inbox every couple of days. <a href="https://realpython.com/python-tricks/?utm_source=realpython&utm_medium=rss&utm_campaign=footer">>> Click here to learn more and see examples</a> ]</em></p>#
12489 datePublished: #Mon Aug 5 14:00:00 2019#
12490 dateUpdated: #Mon Aug 5 14:00:00 2019#
12491 # Person begin ####################
12492 name: #Real Python#
12493 # Person end ######################
12494 # Item end ########################
12495 # Item begin ######################
12496 id: #https://realpython.com/courses/dictionaries-python/#
12497 title: #Dictionaries in Python#
12498 link: #https://realpython.com/courses/dictionaries-python/#
12499 description: #In this course on Python dictionaries, you'll cover the basic characteristics of dictionaries and learn how to access and manage dictionary data. Once you've finished this course, you'll have a good sense of when a dictionary is the appropriate data type to use and know how to use it.#
12500 content: #<p>Python provides a composite <a href="https://realpython.com/python-data-types/">data type</a> called a <strong>dictionary</strong>, which is similar to a <a href="https://realpython.com/python-lists-tuples/">list</a> in that it is a collection of objects.</p>
12501 <p><strong>Here’s what you’ll learn in this course:</strong> You’ll cover the basic characteristics of Python dictionaries and learn how to access and manage dictionary data. Once you’ve finished this course, you’ll have a good sense of when a dictionary is the appropriate data type to use and know how to use it.</p>
12502 <p>Dictionaries and lists share the following characteristics:</p>
12503 <ul>
12504 <li>Both are mutable.</li>
12505 <li>Both are dynamic. They can grow and shrink as needed.</li>
12506 <li>Both can be nested. A list can contain another list. A dictionary can contain another dictionary. A dictionary can also contain a list, and vice versa.</li>
12507 </ul>
12508 <p>Dictionaries differ from lists primarily in how elements are accessed:</p>
12509 <ul>
12510 <li>List elements are accessed by their position in the list, via indexing.</li>
12511 <li>Dictionary elements are accessed via keys.</li>
12512 </ul>
12513 <hr />
12514 <p><em>[ Improve Your Python With π Python Tricks π β Get a short & sweet Python Trick delivered to your inbox every couple of days. <a href="https://realpython.com/python-tricks/?utm_source=realpython&utm_medium=rss&utm_campaign=footer">>> Click here to learn more and see examples</a> ]</em></p>#
12515 datePublished: #Tue Jul 30 14:00:00 2019#
12516 dateUpdated: #Tue Jul 30 14:00:00 2019#
12517 # Person begin ####################
12518 name: #Real Python#
12519 # Person end ######################
12520 # Item end ########################
12521 # Item begin ######################
12522 id: #https://realpython.com/courses/logging-python/#
12523 title: #Logging in Python#
12524 link: #https://realpython.com/courses/logging-python/#
12525 description: #In this video course, you'll learn why and how to get started with Python's powerful logging module to meet the needs of beginners and enterprise teams alike.#
12526 content: #<p>Logging is a very useful tool in a programmerβs toolbox. It can help you develop a better understanding of the flow of a program and discover scenarios that you might not even have thought of while developing.</p>
12527 <p>Logs provide developers with an extra set of eyes that are constantly looking at the flow that an application is going through. They can store information, like which user or IP accessed the application. If an error occurs, then they can provide more insights than a stack trace by telling you what the state of the program was before it arrived at the line of code where the error occurred.</p>
12528 <p>By logging useful data from the right places, you can not only debug errors easily but also use the data to analyze the performance of the application to plan for scaling or look at usage patterns to plan for marketing.</p>
12529 <p>Python provides a logging system as a part of its standard library, so you can quickly add logging to your application. In this course, you’ll learn why using this module is the best way to add logging to your application as well as how to get started quickly, and you will get an introduction to some of the advanced features available.</p>
12530 <hr />
12531 <p><em>[ Improve Your Python With π Python Tricks π β Get a short & sweet Python Trick delivered to your inbox every couple of days. <a href="https://realpython.com/python-tricks/?utm_source=realpython&utm_medium=rss&utm_campaign=footer">>> Click here to learn more and see examples</a> ]</em></p>#
12532 datePublished: #Tue Jul 23 14:00:00 2019#
12533 dateUpdated: #Tue Jul 23 14:00:00 2019#
12534 # Person begin ####################
12535 name: #Real Python#
12536 # Person end ######################
12537 # Item end ########################
12538 # Item begin ######################
12539 id: #https://realpython.com/courses/how-to-write-pythonic-loops/#
12540 title: #How to Write Pythonic Loops#
12541 link: #https://realpython.com/courses/how-to-write-pythonic-loops/#
12542 description: #In this course, you'll see how you can make your loops more Pythonic if you're coming to Python from a C-style language. You'll learn how you can get the most out of using range(), xrange(), and enumerate(). You'll also see how you can avoid having to keep track of loop indexes manually.#
12543 content: #<p>One of the easiest ways to spot a developer who has a background in C-style languages and only recently picked up Python is to look at how they loop through a list. In this course, you’ll learn how to take a C-style (Java, PHP, C, C++) loop and turn it into the sort of loop a Python developer would write.</p>
12544 <p>You can use these techniques to refactor your existing Python <a href="https://realpython.com/python-for-loop/"><code>for</code> loops</a> and <a href="https://realpython.com/python-while-loop/"><code>while</code> loops</a> in order to make them easier to read and more maintainable. You’ll learn how to use Python’s <a href="https://realpython.com/python-range/"><code>range()</code>, <code>xrange()</code></a>, and <code>enumerate()</code> built-ins to refactor your loops and how to avoid having to keep track of loop indexes manually.</p>
12545 <p>The main takeaways in this tutorial are that:</p>
12546 <ol>
12547 <li>
12548 <p>Writing C-style loops in Python is considered not <a href="https://realpython.com/learning-paths/writing-pythonic-code/">Pythonic</a>. Avoid managing loop indexes and stop conditions manually if possible.</p>
12549 </li>
12550 <li>
12551 <p>Pythonβs <code>for</code> loops are really βfor eachβ loops that can iterate over items from a container or sequence directly.</p>
12552 </li>
12553 </ol>
12554 <hr />
12555 <p><em>[ Improve Your Python With π Python Tricks π β Get a short & sweet Python Trick delivered to your inbox every couple of days. <a href="https://realpython.com/python-tricks/?utm_source=realpython&utm_medium=rss&utm_campaign=footer">>> Click here to learn more and see examples</a> ]</em></p>#
12556 datePublished: #Tue Jul 16 14:00:00 2019#
12557 dateUpdated: #Tue Jul 16 14:00:00 2019#
12558 # Person begin ####################
12559 name: #Real Python#
12560 # Person end ######################
12561 # Item end ########################
12562 # Item begin ######################
12563 id: #https://realpython.com/courses/reading-and-writing-files-python/#
12564 title: #Reading and Writing Files in Python#
12565 link: #https://realpython.com/courses/reading-and-writing-files-python/#
12566 description: #In this course, you'll learn about reading and writing files in Python. You'll cover everything from what a file is made up of to which libraries can help you along that way. You'll also take a look at some basic scenarios of file usage as well as some advanced techniques.#
12567 content: #<p>In this course, you’ll learn about reading and writing files in Python. You’ll cover everything from what a file is made up of to which libraries can help you along that way. You’ll also take a look at some basic scenarios of file usage as well as some advanced techniques.</p>
12568 <p>One of the most common tasks that you can do with Python is reading and writing files. Whether itβs writing to a simple text file, reading a complicated server log, or even analyzing raw byte data, all of these situations require reading or writing a file.</p>
12569 <p>By the end of this course, youβll know:</p>
12570 <ul>
12571 <li>What makes up a file and why thatβs important in Python</li>
12572 <li>The basics of reading and writing files in Python</li>
12573 <li>Some basic scenarios of reading and writing files</li>
12574 </ul>
12575 <p>This tutorial is mainly for beginner to intermediate Pythonistas, but there are some tips in here that more advanced programmers may appreciate as well.</p>
12576 <hr />
12577 <p><em>[ Improve Your Python With π Python Tricks π β Get a short & sweet Python Trick delivered to your inbox every couple of days. <a href="https://realpython.com/python-tricks/?utm_source=realpython&utm_medium=rss&utm_campaign=footer">>> Click here to learn more and see examples</a> ]</em></p>#
12578 datePublished: #Tue Jul 9 14:00:00 2019#
12579 dateUpdated: #Tue Jul 9 14:00:00 2019#
12580 # Person begin ####################
12581 name: #Real Python#
12582 # Person end ######################
12583 # Item end ########################
12584 # Item begin ######################
12585 id: #https://realpython.com/courses/functional-programming-python/#
12586 title: #Functional Programming in Python#
12587 link: #https://realpython.com/courses/functional-programming-python/#
12588 description: #In this course, you'll learn how to approach functional programming in Python. You'll cover what functional programming is, how you can use immutable data structures to represent your data, as well as how to use filter(), map(), and reduce().#
12589 content: #<p>In this course, you’ll learn how to approach functional programming in Python. You’ll start with the absolute basics of Functional Programming (FP). After that, you’ll see hands-on examples for common FP patterns available, like using immutable data structures and the <code>filter()</code>, <code>map()</code>, and <code>reduce()</code> functions. You’ll end the course with actionable tips for parallelizing your code to make it run faster.</p>
12590 <p>You’ll cover:</p>
12591 <ol>
12592 <li>What functional programming is</li>
12593 <li>How you can use immutable data structures to represent your data</li>
12594 <li>How to use <code>filter()</code>, <code>map()</code>, and <code>reduce()</code></li>
12595 <li>How to do parallel processing with <code>multiprocessing</code> and <code>concurrent.futures</code></li>
12596 </ol>
12597 <hr />
12598 <p><em>[ Improve Your Python With π Python Tricks π β Get a short & sweet Python Trick delivered to your inbox every couple of days. <a href="https://realpython.com/python-tricks/?utm_source=realpython&utm_medium=rss&utm_campaign=footer">>> Click here to learn more and see examples</a> ]</em></p>#
12599 datePublished: #Tue Jul 2 14:00:00 2019#
12600 dateUpdated: #Tue Jul 2 14:00:00 2019#
12601 # Person begin ####################
12602 name: #Real Python#
12603 # Person end ######################
12604 # Item end ########################
12605 # Feed end ########################
12606