Stefan Scherfkehttp://stefan.sofa-rockers.org/2014-06-25T10:15:00+02:00SimPy: Real-time simulations2014-06-25T10:15:00+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2014-06-25:2014/06/25/simpy-real-time-simulations/<p>This is another guide to SimPy simulations. SimPy ususally runs simulations as fast as possible, but sometimes, you might want your simulation to run synchronized with the wall-clock time. This kind of simulation is also called <em>real-time simulation</em>.</p> <p>Real-time simulations may be&nbsp;necessary</p> <ul class="simple"> <li>if you have&nbsp;hardware-in-the-loop,</li> <li>if there is human interaction with your simulation,&nbsp;or</li> <li>if you want to analyze the real-time behavior of an&nbsp;algorithm.</li> </ul> <p>To convert a simulation into a real-time simulation, you only need to replace SimPy&#8217;s default <tt class="docutils literal">Environment</tt> with a <tt class="docutils literal">simpy.rt.RealtimeEnvironment</tt>. Apart from the <em>initial_time</em> argument, there are two additional parameters: <em>factor</em> and <em>strict</em>: <tt class="docutils literal">RealtimeEnvironment(initial_time=0, factor=1.0, strict=True)</tt>.</p> <p>The <em>factor</em> defines how much <em>real time</em> passes with each step of simulation time. By default, this is one second. If you set <tt class="docutils literal">factor=0.1</tt>, a unit of simulation time will only take a tenth of a second; if you set <tt class="docutils literal">factor=60</tt>, it will take a&nbsp;minute.</p> <p>Here is a simple example for converting a normal simulation to a real-time simulation with a duration of one tenth of a second per simulation time&nbsp;unit:</p> <div class="highlight"><pre><span class="o">&gt;&gt;&gt;</span> <span class="kn">import</span> <span class="nn">time</span> <span class="o">&gt;&gt;&gt;</span> <span class="kn">import</span> <span class="nn">simpy</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">example</span><span class="p">(</span><span class="n">env</span><span class="p">):</span> <span class="o">...</span> <span class="n">start</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">perf_counter</span><span class="p">()</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="o">...</span> <span class="n">end</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">perf_counter</span><span class="p">()</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;Duration of one simulation time unit: </span><span class="si">%.2f</span><span class="s">s&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">end</span> <span class="o">-</span> <span class="n">start</span><span class="p">))</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span> <span class="o">=</span> <span class="n">simpy</span><span class="o">.</span><span class="n">Environment</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">proc</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="n">example</span><span class="p">(</span><span class="n">env</span><span class="p">))</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">until</span><span class="o">=</span><span class="n">proc</span><span class="p">)</span> <span class="n">Duration</span> <span class="n">of</span> <span class="n">one</span> <span class="n">simulation</span> <span class="n">time</span> <span class="n">unit</span><span class="p">:</span> <span class="mf">0.00</span><span class="n">s</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="kn">import</span> <span class="nn">simpy.rt</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span> <span class="o">=</span> <span class="n">simpy</span><span class="o">.</span><span class="n">rt</span><span class="o">.</span><span class="n">RealtimeEnvironment</span><span class="p">(</span><span class="n">factor</span><span class="o">=</span><span class="mf">0.1</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">proc</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="n">example</span><span class="p">(</span><span class="n">env</span><span class="p">))</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">until</span><span class="o">=</span><span class="n">proc</span><span class="p">)</span> <span class="n">Duration</span> <span class="n">of</span> <span class="n">one</span> <span class="n">simulation</span> <span class="n">time</span> <span class="n">unit</span><span class="p">:</span> <span class="mf">0.10</span><span class="n">s</span> </pre></div> <p>If the <em>strict</em> parameter is set to <tt class="docutils literal">True</tt> (the default), the <tt class="docutils literal">step()</tt> and <tt class="docutils literal">run()</tt> methods will raise a <tt class="docutils literal">RuntimeError</tt> if the computation within a simulation time step take more time than the real-time factor allows. In the following example, a process will perform a task that takes 0.02 seconds within a real-time environment with a time factor of 0.01&nbsp;seconds:</p> <div class="highlight"><pre><span class="o">&gt;&gt;&gt;</span> <span class="kn">import</span> <span class="nn">time</span> <span class="o">&gt;&gt;&gt;</span> <span class="kn">import</span> <span class="nn">simpy.rt</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">slow_proc</span><span class="p">(</span><span class="n">env</span><span class="p">):</span> <span class="o">...</span> <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.02</span><span class="p">)</span> <span class="c"># Heavy computation :-)</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span> <span class="o">=</span> <span class="n">simpy</span><span class="o">.</span><span class="n">rt</span><span class="o">.</span><span class="n">RealtimeEnvironment</span><span class="p">(</span><span class="n">factor</span><span class="o">=</span><span class="mf">0.01</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">proc</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="n">slow_proc</span><span class="p">(</span><span class="n">env</span><span class="p">))</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">try</span><span class="p">:</span> <span class="o">...</span> <span class="n">env</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">until</span><span class="o">=</span><span class="n">proc</span><span class="p">)</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;Everything alright&#39;</span><span class="p">)</span> <span class="o">...</span> <span class="k">except</span> <span class="ne">RuntimeError</span><span class="p">:</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;Simulation is too slow&#39;</span><span class="p">)</span> <span class="n">Simulation</span> <span class="ow">is</span> <span class="n">too</span> <span class="n">slow</span> </pre></div> <p>To suppress the error, simply set <tt class="docutils literal">strict=False</tt>:</p> <div class="highlight"><pre><span class="o">&gt;&gt;&gt;</span> <span class="n">env</span> <span class="o">=</span> <span class="n">simpy</span><span class="o">.</span><span class="n">rt</span><span class="o">.</span><span class="n">RealtimeEnvironment</span><span class="p">(</span><span class="n">factor</span><span class="o">=</span><span class="mf">0.01</span><span class="p">,</span> <span class="n">strict</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">proc</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="n">slow_proc</span><span class="p">(</span><span class="n">env</span><span class="p">))</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">try</span><span class="p">:</span> <span class="o">...</span> <span class="n">env</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">until</span><span class="o">=</span><span class="n">proc</span><span class="p">)</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;Everything alright&#39;</span><span class="p">)</span> <span class="o">...</span> <span class="k">except</span> <span class="ne">RuntimeError</span><span class="p">:</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;Simulation is too slow&#39;</span><span class="p">)</span> <span class="n">Everything</span> <span class="n">alright</span> </pre></div> <p>That&#8217;s it. Real-time simulations are that simple with SimPy! This guide is now also part of the <a class="reference external" href="http://simpy.readthedocs.org/en/latest/topical_guides/real-time-simulations.html">official SimPy documentation</a>.</p> SimPy: Shared Resources2014-06-13T21:18:00+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2014-06-13:2014/06/13/simpy-shared-resources/<p>Shared resources are another way to model <a class="reference external" href="http://simpy.readthedocs.org/en/latest/topical_guides/process_interaction.html">process interaction</a>. They form a congestion point where processes queue up in order to use&nbsp;them.</p> <p>SimPy defines three categories of&nbsp;resources:</p> <ul class="simple"> <li><a class="reference internal" href="#resources">Resources</a> – Resources that can be used by a limited number of processes at a time (e.g., a gas station with a limited number of fuel&nbsp;pumps).</li> <li><a class="reference internal" href="#containers">Containers</a> – Resources that model the production and consumption of a homogeneous, undifferentiated bulk. It may either be continuous (like water) or discrete (like&nbsp;apples).</li> <li><a class="reference internal" href="#stores">Stores</a> – Resources that allow the production and consumption of Python&nbsp;objects.</li> </ul> <div class="section" id="the-basic-concept-of-resources"> <h2>The basic concept of&nbsp;resources</h2> <p>All resources share the same basic concept: The resource itself is some kind of a container with a, usually limited, <em>capacity</em>. Processes can either try to <em>put</em> something into the resource or try to <em>get</em> something out. If the resource is full or empty, they have to <em>queue</em> up and&nbsp;wait.</p> <p>This is roughly, how every resource looks&nbsp;like:</p> <pre class="literal-block"> BaseResource(capacity): put_queue get_queue put(): event get(): event </pre> <p>Every resources a maximum capacity and two queues, one for processes that want to put something into it and one for processes that want to get something out. The <tt class="docutils literal">put()</tt> and <tt class="docutils literal">get()</tt> methods both return an event that is triggered when the corresponding action was&nbsp;successful.</p> <div class="section" id="resources-and-interrupts"> <h3>Resources and&nbsp;interrupts</h3> <p>While a process is waiting for a put or get event to succeed, it may be <a class="reference external" href="http://simpy.readthedocs.org/en/latest/topical_guides/process_interaction.html#interrupting-another-process">interrupted</a> by another process. After catching the interrupt, the process has two&nbsp;possibilities:</p> <ol class="arabic"> <li><p class="first">It may continue to wait for the request (by yielding the event&nbsp;again).</p> </li> <li><p class="first">It may stop waiting for the request. In this case, it has to call the event&#8217;s <tt class="docutils literal">cancel()</tt> method.</p> <p>Since you can easily forget this, all resources events are <em>context managers</em> (see the <a class="reference external" href="https://docs.python.org/3/reference/compound_stmts.html#with">Python docs</a> for&nbsp;details).</p> </li> </ol> <hr class="docutils" /> <p>The resource system is modular and extensible. Resources can, for example, use specialized queues and event types. This allows them to use sorted queues, to add priorities to events, or to offer&nbsp;preemption.</p> </div> </div> <div class="section" id="resources"> <h2>Resources</h2> <p>Resources can be used by a limited number of processes at a time (e.g., a gas station with a limited number of fuel pumps). Processes <em>request</em> these resources to become a user (or to &quot;own&quot; them) and have to <em>release</em> them once they are done (e.g., vehicles arrive at the gas station, use a fuel-pump, if one is available, and leave when they are&nbsp;done).</p> <p>Requesting a resources is modeled as &quot;putting a process&#8217; token into the resources&quot; and releasing a resources correspondingly as &quot;getting a process&#8217; token out of the resource&quot;. Thus, calling <tt class="docutils literal">request()</tt>/<tt class="docutils literal">release()</tt> is equivalent to calling <tt class="docutils literal">put()</tt>/<tt class="docutils literal">get()</tt>. Releasing a resource will always succeed&nbsp;immediately.</p> <p>SimPy implements three <em>resource</em>&nbsp;types:</p> <ol class="arabic simple"> <li><a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.resources.html#simpy.resources.resource.Resource">Resource</a></li> <li><a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.resources.html#simpy.resources.resource.PriorityResource">PriorityResource</a>, where queueing processes are sorted by&nbsp;priority</li> <li><a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.resources.html#simpy.resources.resource.PreemptiveResource">PreemptiveResource</a>, where processes additionally may preempt other processes with a lower&nbsp;priority</li> </ol> <div class="section" id="resource"> <h3>Resource</h3> <p>The <tt class="docutils literal">Resource</tt> is conceptually a <em>semaphore</em>. Its only parameter – apart from the obligatory reference to an <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.core.html#simpy.core.Environment">Environment</a> – is its <em>capacity</em>. It must be a positive number and defaults to 1: <tt class="docutils literal">Resource(env, capacity=1)</tt>.</p> <p>Instead of just counting its current users, it stores the request event as an &quot;access token&quot; for each user. This is, for example, useful for adding preemption (see&nbsp;below).</p> <p>Here is as basic example for using a&nbsp;resource:</p> <div class="highlight"><pre><span class="o">&gt;&gt;&gt;</span> <span class="kn">import</span> <span class="nn">simpy</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">resource_user</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">resource</span><span class="p">):</span> <span class="o">...</span> <span class="n">request</span> <span class="o">=</span> <span class="n">resource</span><span class="o">.</span><span class="n">request</span><span class="p">()</span> <span class="c"># Generate a request event</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">request</span> <span class="c"># Wait for access</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="c"># Do something</span> <span class="o">...</span> <span class="n">resource</span><span class="o">.</span><span class="n">release</span><span class="p">(</span><span class="n">request</span><span class="p">)</span> <span class="c"># Release the resource</span> <span class="o">...</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span> <span class="o">=</span> <span class="n">simpy</span><span class="o">.</span><span class="n">Environment</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">res</span> <span class="o">=</span> <span class="n">simpy</span><span class="o">.</span><span class="n">Resource</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">capacity</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">user</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="n">resource_user</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">res</span><span class="p">))</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span><span class="o">.</span><span class="n">run</span><span class="p">()</span> </pre></div> <p>Note, that you have to release the resource under all conditions; for example, if you got interrupted while waiting for or using the resource. In order to help you with that and to avoid too many <tt class="docutils literal">try: ... finally: ...</tt> constructs, request events can be used as context&nbsp;manager:</p> <div class="highlight"><pre><span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">resource_user</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">resource</span><span class="p">):</span> <span class="o">...</span> <span class="k">with</span> <span class="n">resource</span><span class="o">.</span><span class="n">request</span><span class="p">()</span> <span class="k">as</span> <span class="n">req</span><span class="p">:</span> <span class="c"># Generate a request event</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">req</span> <span class="c"># Wait for access</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="c"># Do something</span> <span class="o">...</span> <span class="c"># Resource released automatically</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">user</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="n">resource_user</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">res</span><span class="p">))</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span><span class="o">.</span><span class="n">run</span><span class="p">()</span> </pre></div> <p>Resources allow you retrieve the list of users and queued as well as the number of users and resource&#8217;s&nbsp;capacity:</p> <div class="highlight"><pre><span class="o">&gt;&gt;&gt;</span> <span class="n">res</span> <span class="o">=</span> <span class="n">simpy</span><span class="o">.</span><span class="n">Resource</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">capacity</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">print_stats</span><span class="p">(</span><span class="n">res</span><span class="p">):</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;</span><span class="si">%d</span><span class="s"> of </span><span class="si">%d</span><span class="s"> slots are allocated.&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">res</span><span class="o">.</span><span class="n">count</span><span class="p">,</span> <span class="n">res</span><span class="o">.</span><span class="n">capacity</span><span class="p">))</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39; Users:&#39;</span><span class="p">,</span> <span class="n">res</span><span class="o">.</span><span class="n">users</span><span class="p">)</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39; Queued events:&#39;</span><span class="p">,</span> <span class="n">res</span><span class="o">.</span><span class="n">queue</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">user</span><span class="p">(</span><span class="n">res</span><span class="p">):</span> <span class="o">...</span> <span class="n">print_stats</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">...</span> <span class="k">with</span> <span class="n">res</span><span class="o">.</span><span class="n">request</span><span class="p">()</span> <span class="k">as</span> <span class="n">req</span><span class="p">:</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">req</span> <span class="o">...</span> <span class="n">print_stats</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">...</span> <span class="n">print_stats</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">procs</span> <span class="o">=</span> <span class="p">[</span><span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="n">user</span><span class="p">(</span><span class="n">res</span><span class="p">)),</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="n">user</span><span class="p">(</span><span class="n">res</span><span class="p">))]</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span><span class="o">.</span><span class="n">run</span><span class="p">()</span> <span class="mi">0</span> <span class="n">of</span> <span class="mi">1</span> <span class="n">slots</span> <span class="n">are</span> <span class="n">allocated</span><span class="o">.</span> <span class="n">Users</span><span class="p">:</span> <span class="p">[]</span> <span class="n">Queued</span> <span class="n">events</span><span class="p">:</span> <span class="p">[]</span> <span class="mi">1</span> <span class="n">of</span> <span class="mi">1</span> <span class="n">slots</span> <span class="n">are</span> <span class="n">allocated</span><span class="o">.</span> <span class="n">Users</span><span class="p">:</span> <span class="p">[</span><span class="o">&lt;</span><span class="n">Request</span><span class="p">()</span> <span class="nb">object</span> <span class="n">at</span> <span class="mi">0</span><span class="n">x</span><span class="o">...&gt;</span><span class="p">]</span> <span class="n">Queued</span> <span class="n">events</span><span class="p">:</span> <span class="p">[</span><span class="o">&lt;</span><span class="n">Request</span><span class="p">()</span> <span class="nb">object</span> <span class="n">at</span> <span class="mi">0</span><span class="n">x</span><span class="o">...&gt;</span><span class="p">]</span> <span class="mi">1</span> <span class="n">of</span> <span class="mi">1</span> <span class="n">slots</span> <span class="n">are</span> <span class="n">allocated</span><span class="o">.</span> <span class="n">Users</span><span class="p">:</span> <span class="p">[</span><span class="o">&lt;</span><span class="n">Request</span><span class="p">()</span> <span class="nb">object</span> <span class="n">at</span> <span class="mi">0</span><span class="n">x</span><span class="o">...&gt;</span><span class="p">]</span> <span class="n">Queued</span> <span class="n">events</span><span class="p">:</span> <span class="p">[</span><span class="o">&lt;</span><span class="n">Request</span><span class="p">()</span> <span class="nb">object</span> <span class="n">at</span> <span class="mi">0</span><span class="n">x</span><span class="o">...&gt;</span><span class="p">]</span> <span class="mi">0</span> <span class="n">of</span> <span class="mi">1</span> <span class="n">slots</span> <span class="n">are</span> <span class="n">allocated</span><span class="o">.</span> <span class="n">Users</span><span class="p">:</span> <span class="p">[]</span> <span class="n">Queued</span> <span class="n">events</span><span class="p">:</span> <span class="p">[</span><span class="o">&lt;</span><span class="n">Request</span><span class="p">()</span> <span class="nb">object</span> <span class="n">at</span> <span class="mi">0</span><span class="n">x</span><span class="o">...&gt;</span><span class="p">]</span> <span class="mi">1</span> <span class="n">of</span> <span class="mi">1</span> <span class="n">slots</span> <span class="n">are</span> <span class="n">allocated</span><span class="o">.</span> <span class="n">Users</span><span class="p">:</span> <span class="p">[</span><span class="o">&lt;</span><span class="n">Request</span><span class="p">()</span> <span class="nb">object</span> <span class="n">at</span> <span class="mi">0</span><span class="n">x</span><span class="o">...&gt;</span><span class="p">]</span> <span class="n">Queued</span> <span class="n">events</span><span class="p">:</span> <span class="p">[]</span> <span class="mi">0</span> <span class="n">of</span> <span class="mi">1</span> <span class="n">slots</span> <span class="n">are</span> <span class="n">allocated</span><span class="o">.</span> <span class="n">Users</span><span class="p">:</span> <span class="p">[]</span> <span class="n">Queued</span> <span class="n">events</span><span class="p">:</span> <span class="p">[]</span> </pre></div> </div> <div class="section" id="priorityresource"> <h3>PriorityResource</h3> <p>As you may know from the real world, not every one is equally important. To map that to SimPy, there&#8217;s the <em>PriorityResource</em>. This subclass of <em>Resource</em> lets requesting processes provide a priority for each request. More important requests will gain access to the resource earlier than less important ones. Priority is expressed by integer numbers; smaller numbers mean a higher&nbsp;priority.</p> <p>Apart form that, it works like a normal <em>Resource</em>:</p> <div class="highlight"><pre><span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">resource_user</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">env</span><span class="p">,</span> <span class="n">resource</span><span class="p">,</span> <span class="n">wait</span><span class="p">,</span> <span class="n">prio</span><span class="p">):</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="n">wait</span><span class="p">)</span> <span class="o">...</span> <span class="k">with</span> <span class="n">resource</span><span class="o">.</span><span class="n">request</span><span class="p">(</span><span class="n">priority</span><span class="o">=</span><span class="n">prio</span><span class="p">)</span> <span class="k">as</span> <span class="n">req</span><span class="p">:</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;</span><span class="si">%s</span><span class="s"> requesting at </span><span class="si">%s</span><span class="s"> with priority=</span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">,</span> <span class="n">prio</span><span class="p">))</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">req</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;</span><span class="si">%s</span><span class="s"> got resource at </span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">))</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="o">...</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span> <span class="o">=</span> <span class="n">simpy</span><span class="o">.</span><span class="n">Environment</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">res</span> <span class="o">=</span> <span class="n">simpy</span><span class="o">.</span><span class="n">PriorityResource</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">capacity</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">p1</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="n">resource_user</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">env</span><span class="p">,</span> <span class="n">res</span><span class="p">,</span> <span class="n">wait</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="n">prio</span><span class="o">=</span><span class="mi">0</span><span class="p">))</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">p2</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="n">resource_user</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="n">env</span><span class="p">,</span> <span class="n">res</span><span class="p">,</span> <span class="n">wait</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">prio</span><span class="o">=</span><span class="mi">0</span><span class="p">))</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">p3</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="n">resource_user</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="n">env</span><span class="p">,</span> <span class="n">res</span><span class="p">,</span> <span class="n">wait</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">prio</span><span class="o">=-</span><span class="mi">1</span><span class="p">))</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span><span class="o">.</span><span class="n">run</span><span class="p">()</span> <span class="mi">1</span> <span class="n">requesting</span> <span class="n">at</span> <span class="mi">0</span> <span class="k">with</span> <span class="n">priority</span><span class="o">=</span><span class="mi">0</span> <span class="mi">1</span> <span class="n">got</span> <span class="n">resource</span> <span class="n">at</span> <span class="mi">0</span> <span class="mi">2</span> <span class="n">requesting</span> <span class="n">at</span> <span class="mi">1</span> <span class="k">with</span> <span class="n">priority</span><span class="o">=</span><span class="mi">0</span> <span class="mi">3</span> <span class="n">requesting</span> <span class="n">at</span> <span class="mi">2</span> <span class="k">with</span> <span class="n">priority</span><span class="o">=-</span><span class="mi">1</span> <span class="mi">3</span> <span class="n">got</span> <span class="n">resource</span> <span class="n">at</span> <span class="mi">3</span> <span class="mi">2</span> <span class="n">got</span> <span class="n">resource</span> <span class="n">at</span> <span class="mi">6</span> </pre></div> <p>Although <em>p3</em> requested the resource later than <em>p2</em>, it could use it earlier because its priority was&nbsp;higher.</p> </div> <div class="section" id="preemptiveresource"> <h3>PreemptiveResource</h3> <p>Sometimes, new requests are so important that queue-jumping is not enough and they need to kick existing users out of the resource (this is called <em>preemption</em>). The <em>PreemptiveResource</em> allows you to do exactly&nbsp;this:</p> <div class="highlight"><pre><span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">resource_user</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">env</span><span class="p">,</span> <span class="n">resource</span><span class="p">,</span> <span class="n">wait</span><span class="p">,</span> <span class="n">prio</span><span class="p">):</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="n">wait</span><span class="p">)</span> <span class="o">...</span> <span class="k">with</span> <span class="n">resource</span><span class="o">.</span><span class="n">request</span><span class="p">(</span><span class="n">priority</span><span class="o">=</span><span class="n">prio</span><span class="p">)</span> <span class="k">as</span> <span class="n">req</span><span class="p">:</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;</span><span class="si">%s</span><span class="s"> requesting at </span><span class="si">%s</span><span class="s"> with priority=</span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">,</span> <span class="n">prio</span><span class="p">))</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">req</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;</span><span class="si">%s</span><span class="s"> got resource at </span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">))</span> <span class="o">...</span> <span class="k">try</span><span class="p">:</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="o">...</span> <span class="k">except</span> <span class="n">simpy</span><span class="o">.</span><span class="n">Interrupt</span> <span class="k">as</span> <span class="n">interrupt</span><span class="p">:</span> <span class="o">...</span> <span class="n">by</span> <span class="o">=</span> <span class="n">interrupt</span><span class="o">.</span><span class="n">cause</span><span class="o">.</span><span class="n">by</span> <span class="o">...</span> <span class="n">usage</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span> <span class="o">-</span> <span class="n">interrupt</span><span class="o">.</span><span class="n">cause</span><span class="o">.</span><span class="n">usage_since</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;</span><span class="si">%s</span><span class="s"> got preempted by </span><span class="si">%s</span><span class="s"> at </span><span class="si">%s</span><span class="s"> after </span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="o">...</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">by</span><span class="p">,</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">,</span> <span class="n">usage</span><span class="p">))</span> <span class="o">...</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span> <span class="o">=</span> <span class="n">simpy</span><span class="o">.</span><span class="n">Environment</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">res</span> <span class="o">=</span> <span class="n">simpy</span><span class="o">.</span><span class="n">PreemptiveResource</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">capacity</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">p1</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="n">resource_user</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">env</span><span class="p">,</span> <span class="n">res</span><span class="p">,</span> <span class="n">wait</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="n">prio</span><span class="o">=</span><span class="mi">0</span><span class="p">))</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">p2</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="n">resource_user</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="n">env</span><span class="p">,</span> <span class="n">res</span><span class="p">,</span> <span class="n">wait</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">prio</span><span class="o">=</span><span class="mi">0</span><span class="p">))</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">p3</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="n">resource_user</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="n">env</span><span class="p">,</span> <span class="n">res</span><span class="p">,</span> <span class="n">wait</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">prio</span><span class="o">=-</span><span class="mi">1</span><span class="p">))</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span><span class="o">.</span><span class="n">run</span><span class="p">()</span> <span class="mi">1</span> <span class="n">requesting</span> <span class="n">at</span> <span class="mi">0</span> <span class="k">with</span> <span class="n">priority</span><span class="o">=</span><span class="mi">0</span> <span class="mi">1</span> <span class="n">got</span> <span class="n">resource</span> <span class="n">at</span> <span class="mi">0</span> <span class="mi">2</span> <span class="n">requesting</span> <span class="n">at</span> <span class="mi">1</span> <span class="k">with</span> <span class="n">priority</span><span class="o">=</span><span class="mi">0</span> <span class="mi">3</span> <span class="n">requesting</span> <span class="n">at</span> <span class="mi">2</span> <span class="k">with</span> <span class="n">priority</span><span class="o">=-</span><span class="mi">1</span> <span class="mi">1</span> <span class="n">got</span> <span class="n">preempted</span> <span class="n">by</span> <span class="o">&lt;</span><span class="n">Process</span><span class="p">(</span><span class="n">resource_user</span><span class="p">)</span> <span class="nb">object</span> <span class="n">at</span> <span class="mi">0</span><span class="n">x</span><span class="o">...&gt;</span> <span class="n">at</span> <span class="mi">2</span> <span class="n">after</span> <span class="mi">2</span> <span class="mi">3</span> <span class="n">got</span> <span class="n">resource</span> <span class="n">at</span> <span class="mi">2</span> <span class="mi">2</span> <span class="n">got</span> <span class="n">resource</span> <span class="n">at</span> <span class="mi">5</span> </pre></div> <p><em>PreemptiveResource</em> inherits from <em>PriorityResource</em> and adds a <em>preempt</em> flag (that defaults to <tt class="docutils literal">True</tt>) to <tt class="docutils literal">request()</tt>. By setting this to <tt class="docutils literal">False</tt> (<tt class="docutils literal">resource.request(priority=x, preempt=False)</tt>), a process can decide to not preempt another resource user. It will still be put in the queue according to its priority,&nbsp;though.</p> <p>The implementation of <em>PreemptiveResource</em> values priorities higher than preemption. That means preempt request are not allowed to cheat and jump over a higher prioritized request. The following example shows that preemptive low priority requests cannot queue-jump over high priority&nbsp;requests:</p> <div class="highlight"><pre><span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">user</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">env</span><span class="p">,</span> <span class="n">res</span><span class="p">,</span> <span class="n">prio</span><span class="p">,</span> <span class="n">preempt</span><span class="p">):</span> <span class="o">...</span> <span class="k">with</span> <span class="n">res</span><span class="o">.</span><span class="n">request</span><span class="p">(</span><span class="n">priority</span><span class="o">=</span><span class="n">prio</span><span class="p">,</span> <span class="n">preempt</span><span class="o">=</span><span class="n">preempt</span><span class="p">)</span> <span class="k">as</span> <span class="n">req</span><span class="p">:</span> <span class="o">...</span> <span class="k">try</span><span class="p">:</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;</span><span class="si">%s</span><span class="s"> requesting at </span><span class="si">%d</span><span class="s">&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">))</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">req</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;</span><span class="si">%s</span><span class="s"> got resource at </span><span class="si">%d</span><span class="s">&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">))</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="o">...</span> <span class="k">except</span> <span class="n">simpy</span><span class="o">.</span><span class="n">Interrupt</span><span class="p">:</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;</span><span class="si">%s</span><span class="s"> got preempted at </span><span class="si">%d</span><span class="s">&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">))</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span> <span class="o">=</span> <span class="n">simpy</span><span class="o">.</span><span class="n">Environment</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">res</span> <span class="o">=</span> <span class="n">simpy</span><span class="o">.</span><span class="n">PreemptiveResource</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">capacity</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">A</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="n">user</span><span class="p">(</span><span class="s">&#39;A&#39;</span><span class="p">,</span> <span class="n">env</span><span class="p">,</span> <span class="n">res</span><span class="p">,</span> <span class="n">prio</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="n">preempt</span><span class="o">=</span><span class="bp">True</span><span class="p">))</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">until</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> <span class="c"># Give A a head start</span> <span class="n">A</span> <span class="n">requesting</span> <span class="n">at</span> <span class="mi">0</span> <span class="n">A</span> <span class="n">got</span> <span class="n">resource</span> <span class="n">at</span> <span class="mi">0</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">B</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="n">user</span><span class="p">(</span><span class="s">&#39;B&#39;</span><span class="p">,</span> <span class="n">env</span><span class="p">,</span> <span class="n">res</span><span class="p">,</span> <span class="n">prio</span><span class="o">=-</span><span class="mi">2</span><span class="p">,</span> <span class="n">preempt</span><span class="o">=</span><span class="bp">False</span><span class="p">))</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">C</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="n">user</span><span class="p">(</span><span class="s">&#39;C&#39;</span><span class="p">,</span> <span class="n">env</span><span class="p">,</span> <span class="n">res</span><span class="p">,</span> <span class="n">prio</span><span class="o">=-</span><span class="mi">1</span><span class="p">,</span> <span class="n">preempt</span><span class="o">=</span><span class="bp">True</span><span class="p">))</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span><span class="o">.</span><span class="n">run</span><span class="p">()</span> <span class="n">B</span> <span class="n">requesting</span> <span class="n">at</span> <span class="mi">1</span> <span class="n">C</span> <span class="n">requesting</span> <span class="n">at</span> <span class="mi">1</span> <span class="n">B</span> <span class="n">got</span> <span class="n">resource</span> <span class="n">at</span> <span class="mi">3</span> <span class="n">C</span> <span class="n">got</span> <span class="n">resource</span> <span class="n">at</span> <span class="mi">6</span> </pre></div> <ol class="arabic simple"> <li>Process <em>A</em> requests the resource with priority 0. It immediately becomes a&nbsp;user.</li> <li>Process <em>B</em> requests the resource with priority -2 but sets <em>preempt</em> to <tt class="docutils literal">False</tt>. It will queue up and&nbsp;wait.</li> <li>Process <em>C</em> requests the resource with priority -1 but leaves <em>preempt</em> <tt class="docutils literal">True</tt>. Normally, it would preempt <em>A</em> but in this case, <em>B</em> is queued up before <em>C</em> and prevents <em>C</em> from preempting <em>A</em>. <em>C</em> can also not preempt <em>B</em> since its priority is not high&nbsp;enough.</li> </ol> <p>Thus, the behavior in the example is the same as if no preemption was used at all. Be careful when using mixed&nbsp;preemption!</p> <p>Due to the higher priority of process <em>B</em>, no preemption occurs in this example. Note that an additional request with a priority of -3 would be able to preempt <em>A</em>.</p> <p>If your use-case requires a different behaviour, for example queue-jumping or valuing preemption over priorities, you can subclass <em>PreemptiveResource</em> and override the default&nbsp;behaviour.</p> </div> </div> <div class="section" id="containers"> <h2>Containers</h2> <p>Containers help you modelling the production and consumption of a homogeneous, undifferentiated bulk. It may either be continuous (like water) or discrete (like&nbsp;apples).</p> <p>You can use this, for example, to model the gas / petrol tank of a gas station. Tankers increase the amount of gasoline in the tank while cars decrease&nbsp;it.</p> <p>The following example is a very simple model of a gas station with a limited number of fuel dispensers (modeled as <tt class="docutils literal">Resource</tt>) and a tank modeled as <tt class="docutils literal">Container</tt>:</p> <div class="highlight"><pre><span class="o">&gt;&gt;&gt;</span> <span class="k">class</span> <span class="nc">GasStation</span><span class="p">:</span> <span class="o">...</span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">env</span><span class="p">):</span> <span class="o">...</span> <span class="bp">self</span><span class="o">.</span><span class="n">fuel_dispensers</span> <span class="o">=</span> <span class="n">simpy</span><span class="o">.</span><span class="n">Resource</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">capacity</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span> <span class="o">...</span> <span class="bp">self</span><span class="o">.</span><span class="n">gas_tank</span> <span class="o">=</span> <span class="n">simpy</span><span class="o">.</span><span class="n">Container</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">init</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">capacity</span><span class="o">=</span><span class="mi">1000</span><span class="p">)</span> <span class="o">...</span> <span class="bp">self</span><span class="o">.</span><span class="n">mon_proc</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">monitor_tank</span><span class="p">(</span><span class="n">env</span><span class="p">))</span> <span class="o">...</span> <span class="o">...</span> <span class="k">def</span> <span class="nf">monitor_tank</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">env</span><span class="p">):</span> <span class="o">...</span> <span class="k">while</span> <span class="bp">True</span><span class="p">:</span> <span class="o">...</span> <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">gas_tank</span><span class="o">.</span><span class="n">level</span> <span class="o">&lt;</span> <span class="mi">100</span><span class="p">:</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;Calling tanker at </span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">)</span> <span class="o">...</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="n">tanker</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="bp">self</span><span class="p">))</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="mi">15</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">tanker</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">gas_station</span><span class="p">):</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> <span class="c"># Need 10 Minutes to arrive</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;Tanker arriving at </span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">)</span> <span class="o">...</span> <span class="n">amount</span> <span class="o">=</span> <span class="n">gas_station</span><span class="o">.</span><span class="n">gas_tank</span><span class="o">.</span><span class="n">capacity</span> <span class="o">-</span> <span class="n">gas_station</span><span class="o">.</span><span class="n">gas_tank</span><span class="o">.</span><span class="n">level</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">gas_station</span><span class="o">.</span><span class="n">gas_tank</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">amount</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">car</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">env</span><span class="p">,</span> <span class="n">gas_station</span><span class="p">):</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;Car </span><span class="si">%s</span><span class="s"> arriving at </span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">))</span> <span class="o">...</span> <span class="k">with</span> <span class="n">gas_station</span><span class="o">.</span><span class="n">fuel_dispensers</span><span class="o">.</span><span class="n">request</span><span class="p">()</span> <span class="k">as</span> <span class="n">req</span><span class="p">:</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">req</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;Car </span><span class="si">%s</span><span class="s"> starts refueling at </span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">))</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">gas_station</span><span class="o">.</span><span class="n">gas_tank</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="mi">40</span><span class="p">)</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;Car </span><span class="si">%s</span><span class="s"> done refueling at </span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">))</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">car_generator</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">gas_station</span><span class="p">):</span> <span class="o">...</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">4</span><span class="p">):</span> <span class="o">...</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="n">car</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">env</span><span class="p">,</span> <span class="n">gas_station</span><span class="p">))</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span> <span class="o">=</span> <span class="n">simpy</span><span class="o">.</span><span class="n">Environment</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">gas_station</span> <span class="o">=</span> <span class="n">GasStation</span><span class="p">(</span><span class="n">env</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">car_gen</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="n">car_generator</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">gas_station</span><span class="p">))</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="mi">35</span><span class="p">)</span> <span class="n">Car</span> <span class="mi">0</span> <span class="n">arriving</span> <span class="n">at</span> <span class="mi">0</span> <span class="n">Car</span> <span class="mi">0</span> <span class="n">starts</span> <span class="n">refueling</span> <span class="n">at</span> <span class="mi">0</span> <span class="n">Car</span> <span class="mi">1</span> <span class="n">arriving</span> <span class="n">at</span> <span class="mi">5</span> <span class="n">Car</span> <span class="mi">0</span> <span class="n">done</span> <span class="n">refueling</span> <span class="n">at</span> <span class="mi">5</span> <span class="n">Car</span> <span class="mi">1</span> <span class="n">starts</span> <span class="n">refueling</span> <span class="n">at</span> <span class="mi">5</span> <span class="n">Car</span> <span class="mi">2</span> <span class="n">arriving</span> <span class="n">at</span> <span class="mi">10</span> <span class="n">Car</span> <span class="mi">1</span> <span class="n">done</span> <span class="n">refueling</span> <span class="n">at</span> <span class="mi">10</span> <span class="n">Car</span> <span class="mi">2</span> <span class="n">starts</span> <span class="n">refueling</span> <span class="n">at</span> <span class="mi">10</span> <span class="n">Calling</span> <span class="n">tanker</span> <span class="n">at</span> <span class="mi">15</span> <span class="n">Car</span> <span class="mi">3</span> <span class="n">arriving</span> <span class="n">at</span> <span class="mi">15</span> <span class="n">Car</span> <span class="mi">3</span> <span class="n">starts</span> <span class="n">refueling</span> <span class="n">at</span> <span class="mi">15</span> <span class="n">Tanker</span> <span class="n">arriving</span> <span class="n">at</span> <span class="mi">25</span> <span class="n">Car</span> <span class="mi">2</span> <span class="n">done</span> <span class="n">refueling</span> <span class="n">at</span> <span class="mi">30</span> <span class="n">Car</span> <span class="mi">3</span> <span class="n">done</span> <span class="n">refueling</span> <span class="n">at</span> <span class="mi">30</span> </pre></div> <p>Containers allow you to retrieve their current <tt class="docutils literal">level</tt> as well as their <tt class="docutils literal">capacity</tt> (see <tt class="docutils literal">GasStation.monitor_tank()</tt> and <tt class="docutils literal">tanker()</tt>). You can also access the list of waiting events via the <tt class="docutils literal">put_queue</tt> and <tt class="docutils literal">get_queue</tt> attributes (similar to <tt class="docutils literal">Resource.queue</tt>).</p> </div> <div class="section" id="stores"> <h2>Stores</h2> <p>Using Stores you can model the production and consumption of concrete objects (in contrast to the rather abstract &quot;amount&quot; stored in containers). A single Store can even contain multiple types of&nbsp;objects.</p> <p>Beside <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.resources.html#simpy.resources.store.Store">Store</a>, there is a <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.resources.html#simpy.resources.store.FilterStore">FilterStore</a> that lets you use a custom function to filter the objects you get out of the&nbsp;store.</p> <p>Here is a simple example modelling a generic producer/consumer&nbsp;scenario:</p> <div class="highlight"><pre><span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">producer</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">store</span><span class="p">):</span> <span class="o">...</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">100</span><span class="p">):</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">store</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="s">&#39;spam </span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="n">i</span><span class="p">)</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;Produced spam at&#39;</span><span class="p">,</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">consumer</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">env</span><span class="p">,</span> <span class="n">store</span><span class="p">):</span> <span class="o">...</span> <span class="k">while</span> <span class="bp">True</span><span class="p">:</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="s">&#39;requesting spam at&#39;</span><span class="p">,</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">)</span> <span class="o">...</span> <span class="n">item</span> <span class="o">=</span> <span class="k">yield</span> <span class="n">store</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="s">&#39;got&#39;</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="s">&#39;at&#39;</span><span class="p">,</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span> <span class="o">=</span> <span class="n">simpy</span><span class="o">.</span><span class="n">Environment</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">store</span> <span class="o">=</span> <span class="n">simpy</span><span class="o">.</span><span class="n">Store</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">capacity</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">prod</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="n">producer</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">store</span><span class="p">))</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">consumers</span> <span class="o">=</span> <span class="p">[</span><span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="n">consumer</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">env</span><span class="p">,</span> <span class="n">store</span><span class="p">))</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">2</span><span class="p">)]</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">until</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span> <span class="mi">0</span> <span class="n">requesting</span> <span class="n">spam</span> <span class="n">at</span> <span class="mi">1</span> <span class="mi">1</span> <span class="n">requesting</span> <span class="n">spam</span> <span class="n">at</span> <span class="mi">1</span> <span class="n">Produced</span> <span class="n">spam</span> <span class="n">at</span> <span class="mi">2</span> <span class="mi">0</span> <span class="n">got</span> <span class="n">spam</span> <span class="mi">0</span> <span class="n">at</span> <span class="mi">2</span> <span class="mi">0</span> <span class="n">requesting</span> <span class="n">spam</span> <span class="n">at</span> <span class="mi">3</span> <span class="n">Produced</span> <span class="n">spam</span> <span class="n">at</span> <span class="mi">4</span> <span class="mi">1</span> <span class="n">got</span> <span class="n">spam</span> <span class="mi">1</span> <span class="n">at</span> <span class="mi">4</span> </pre></div> <p>As with the other resource types, you can get a store&#8217;s capacity via the <tt class="docutils literal">capacity</tt> attribute. The attribute <tt class="docutils literal">items</tt> points to the list of items currently available in the store. The put and get queues can be accessed via the <tt class="docutils literal">put_queue</tt> and <tt class="docutils literal">get_queue</tt> attributes.</p> <p><em>FilterStore</em> can, for example, be used to model machine shops where machines have varying attributes. This can be useful if the homogeneous slots of a <em>Resource</em> are not what you&nbsp;need:</p> <div class="highlight"><pre><span class="o">&gt;&gt;&gt;</span> <span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">namedtuple</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">Machine</span> <span class="o">=</span> <span class="n">namedtuple</span><span class="p">(</span><span class="s">&#39;Machine&#39;</span><span class="p">,</span> <span class="s">&#39;size, duration&#39;</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">m1</span> <span class="o">=</span> <span class="n">Machine</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span> <span class="c"># Small and slow</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">m2</span> <span class="o">=</span> <span class="n">Machine</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> <span class="c"># Big and fast</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span> <span class="o">=</span> <span class="n">simpy</span><span class="o">.</span><span class="n">Environment</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">machine_shop</span> <span class="o">=</span> <span class="n">simpy</span><span class="o">.</span><span class="n">FilterStore</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">capacity</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">machine_shop</span><span class="o">.</span><span class="n">items</span> <span class="o">=</span> <span class="p">[</span><span class="n">m1</span><span class="p">,</span> <span class="n">m2</span><span class="p">]</span> <span class="c"># Pre-populate the machine shop</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">user</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">env</span><span class="p">,</span> <span class="n">ms</span><span class="p">,</span> <span class="n">size</span><span class="p">):</span> <span class="o">...</span> <span class="n">machine</span> <span class="o">=</span> <span class="k">yield</span> <span class="n">ms</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="k">lambda</span> <span class="n">machine</span><span class="p">:</span> <span class="n">machine</span><span class="o">.</span><span class="n">size</span> <span class="o">==</span> <span class="n">size</span><span class="p">)</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="s">&#39;got&#39;</span><span class="p">,</span> <span class="n">machine</span><span class="p">,</span> <span class="s">&#39;at&#39;</span><span class="p">,</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">)</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="n">machine</span><span class="o">.</span><span class="n">duration</span><span class="p">)</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">ms</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">machine</span><span class="p">)</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="s">&#39;released&#39;</span><span class="p">,</span> <span class="n">machine</span><span class="p">,</span> <span class="s">&#39;at&#39;</span><span class="p">,</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">users</span> <span class="o">=</span> <span class="p">[</span><span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="n">user</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">env</span><span class="p">,</span> <span class="n">machine_shop</span><span class="p">,</span> <span class="p">(</span><span class="n">i</span> <span class="o">%</span> <span class="mi">2</span><span class="p">)</span> <span class="o">+</span> <span class="mi">1</span><span class="p">))</span> <span class="o">...</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">3</span><span class="p">)]</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span><span class="o">.</span><span class="n">run</span><span class="p">()</span> <span class="mi">0</span> <span class="n">got</span> <span class="n">Machine</span><span class="p">(</span><span class="n">size</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">duration</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span> <span class="n">at</span> <span class="mi">0</span> <span class="mi">1</span> <span class="n">got</span> <span class="n">Machine</span><span class="p">(</span><span class="n">size</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">duration</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> <span class="n">at</span> <span class="mi">0</span> <span class="mi">1</span> <span class="n">released</span> <span class="n">Machine</span><span class="p">(</span><span class="n">size</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">duration</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> <span class="n">at</span> <span class="mi">1</span> <span class="mi">0</span> <span class="n">released</span> <span class="n">Machine</span><span class="p">(</span><span class="n">size</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">duration</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span> <span class="n">at</span> <span class="mi">2</span> <span class="mi">2</span> <span class="n">got</span> <span class="n">Machine</span><span class="p">(</span><span class="n">size</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">duration</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span> <span class="n">at</span> <span class="mi">2</span> <span class="mi">2</span> <span class="n">released</span> <span class="n">Machine</span><span class="p">(</span><span class="n">size</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">duration</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span> <span class="n">at</span> <span class="mi">4</span> </pre></div> <p>This guide is now also <a class="reference external" href="http://simpy.readthedocs.org/en/latest/topical_guides/resources.html">part</a> of SimPy&#8217;s <a class="reference external" href="http://simpy.readthedocs.org/en/latest/">documentation</a>.</p> </div> SimPy: Process Interaction2014-04-11T21:08:00+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2014-04-11:2014/04/11/simpy-process-interaction/<p>As promised, the delay between this and the <a class="reference external" href="/2014/04/05/simpy-events/">last topical guide</a> was rather&nbsp;short.</p> <p>This one is about process interaction. This is what makes event discrete simulation interesting. Without it, you actually wouldn&#8217;t even need <a class="reference external" href="https://simpy.readthedocs.org">SimPy</a>.</p> <p>So this guide is&nbsp;about:</p> <ul class="simple"> <li><a class="reference internal" href="#sleep-until-woken-up">sleep-until-woken-up</a>&nbsp;(passivate/reactivate)</li> <li><a class="reference internal" href="#waiting-for-another-process-to-terminate">waiting-for-another-process-to-terminate</a></li> <li><a class="reference internal" href="#interrupting-another-process">interrupting-another-process</a></li> </ul> <p>Another possibility for processes to interact are resources. They will be discussed in the next&nbsp;guide.</p> <div class="section" id="id2"> <span id="sleep-until-woken-up"></span><h2>Sleep until woken&nbsp;up</h2> <p>Imagine you want to model an electric vehicle with an intelligent battery-charging controller. While the vehicle is driving, the controller can be passive but needs to be reactivate once the vehicle is connected to the power grid in order to charge the&nbsp;battery.</p> <p>In SimPy 2, this pattern was known as <em>passivate / reactivate</em>. In SimPy 3, you can accomplish that with a simple, shared <tt class="docutils literal">Event</tt>:</p> <div class="highlight"><pre><span class="o">&gt;&gt;&gt;</span> <span class="kn">from</span> <span class="nn">random</span> <span class="kn">import</span> <span class="n">seed</span><span class="p">,</span> <span class="n">randint</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">seed</span><span class="p">(</span><span class="mi">23</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="kn">import</span> <span class="nn">simpy</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">class</span> <span class="nc">EV</span><span class="p">:</span> <span class="o">...</span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">env</span><span class="p">):</span> <span class="o">...</span> <span class="bp">self</span><span class="o">.</span><span class="n">env</span> <span class="o">=</span> <span class="n">env</span> <span class="o">...</span> <span class="bp">self</span><span class="o">.</span><span class="n">drive_proc</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">drive</span><span class="p">(</span><span class="n">env</span><span class="p">))</span> <span class="o">...</span> <span class="bp">self</span><span class="o">.</span><span class="n">bat_ctrl_proc</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">bat_ctrl</span><span class="p">(</span><span class="n">env</span><span class="p">))</span> <span class="o">...</span> <span class="bp">self</span><span class="o">.</span><span class="n">bat_ctrl_reactivate</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">event</span><span class="p">()</span> <span class="o">...</span> <span class="o">...</span> <span class="k">def</span> <span class="nf">drive</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">env</span><span class="p">):</span> <span class="o">...</span> <span class="k">while</span> <span class="bp">True</span><span class="p">:</span> <span class="o">...</span> <span class="c"># Drive for 20-40 min</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="n">randint</span><span class="p">(</span><span class="mi">20</span><span class="p">,</span> <span class="mi">40</span><span class="p">))</span> <span class="o">...</span> <span class="o">...</span> <span class="c"># Park for 1–6 hours</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;Start parking at&#39;</span><span class="p">,</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">)</span> <span class="o">...</span> <span class="bp">self</span><span class="o">.</span><span class="n">bat_ctrl_reactivate</span><span class="o">.</span><span class="n">succeed</span><span class="p">()</span> <span class="c"># &quot;reactivate&quot;</span> <span class="o">...</span> <span class="bp">self</span><span class="o">.</span><span class="n">bat_ctrl_reactivate</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">event</span><span class="p">()</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="n">randint</span><span class="p">(</span><span class="mi">60</span><span class="p">,</span> <span class="mi">360</span><span class="p">))</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;Stop parking at&#39;</span><span class="p">,</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">)</span> <span class="o">...</span> <span class="o">...</span> <span class="k">def</span> <span class="nf">bat_ctrl</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">env</span><span class="p">):</span> <span class="o">...</span> <span class="k">while</span> <span class="bp">True</span><span class="p">:</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;Bat. ctrl. passivating at&#39;</span><span class="p">,</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">)</span> <span class="o">...</span> <span class="k">yield</span> <span class="bp">self</span><span class="o">.</span><span class="n">bat_ctrl_reactivate</span> <span class="c"># &quot;passivate&quot;</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;Bat. ctrl. reactivated at&#39;</span><span class="p">,</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">)</span> <span class="o">...</span> <span class="o">...</span> <span class="c"># Intelligent charging behavior here …</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="n">randint</span><span class="p">(</span><span class="mi">30</span><span class="p">,</span> <span class="mi">90</span><span class="p">))</span> <span class="o">...</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span> <span class="o">=</span> <span class="n">simpy</span><span class="o">.</span><span class="n">Environment</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">ev</span> <span class="o">=</span> <span class="n">EV</span><span class="p">(</span><span class="n">env</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">until</span><span class="o">=</span><span class="mi">150</span><span class="p">)</span> <span class="n">Bat</span><span class="o">.</span> <span class="n">ctrl</span><span class="o">.</span> <span class="n">passivating</span> <span class="n">at</span> <span class="mi">0</span> <span class="n">Start</span> <span class="n">parking</span> <span class="n">at</span> <span class="mi">29</span> <span class="n">Bat</span><span class="o">.</span> <span class="n">ctrl</span><span class="o">.</span> <span class="n">reactivated</span> <span class="n">at</span> <span class="mi">29</span> <span class="n">Bat</span><span class="o">.</span> <span class="n">ctrl</span><span class="o">.</span> <span class="n">passivating</span> <span class="n">at</span> <span class="mi">60</span> <span class="n">Stop</span> <span class="n">parking</span> <span class="n">at</span> <span class="mi">131</span> </pre></div> <p>Since <tt class="docutils literal">bat_ctrl()</tt> just waits for a normal event, we no longer call this pattern <em>passivate / reactivate</em> in SimPy&nbsp;3.</p> </div> <div class="section" id="id3"> <span id="waiting-for-another-process-to-terminate"></span><h2>Waiting for another process to&nbsp;terminate</h2> <p>The example above has a problem: it may happen that the vehicles wants to park for a shorter duration than it takes to charge the battery (this is the case if both, charging and parking would take 60 to 90&nbsp;minutes).</p> <p>To fix this problem we have to slightly change our model. A new <tt class="docutils literal">bat_ctrl()</tt> will be started every time the <span class="caps">EV</span> starts parking. The <span class="caps">EV</span> then waits until the parking duration is over <em>and</em> until the charging has&nbsp;stopped:</p> <div class="highlight"><pre><span class="o">&gt;&gt;&gt;</span> <span class="k">class</span> <span class="nc">EV</span><span class="p">:</span> <span class="o">...</span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">env</span><span class="p">):</span> <span class="o">...</span> <span class="bp">self</span><span class="o">.</span><span class="n">env</span> <span class="o">=</span> <span class="n">env</span> <span class="o">...</span> <span class="bp">self</span><span class="o">.</span><span class="n">drive_proc</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">drive</span><span class="p">(</span><span class="n">env</span><span class="p">))</span> <span class="o">...</span> <span class="o">...</span> <span class="k">def</span> <span class="nf">drive</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">env</span><span class="p">):</span> <span class="o">...</span> <span class="k">while</span> <span class="bp">True</span><span class="p">:</span> <span class="o">...</span> <span class="c"># Drive for 20-40 min</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="n">randint</span><span class="p">(</span><span class="mi">20</span><span class="p">,</span> <span class="mi">40</span><span class="p">))</span> <span class="o">...</span> <span class="o">...</span> <span class="c"># Park for 1–6 hours</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;Start parking at&#39;</span><span class="p">,</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">)</span> <span class="o">...</span> <span class="n">charging</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">bat_ctrl</span><span class="p">(</span><span class="n">env</span><span class="p">))</span> <span class="o">...</span> <span class="n">parking</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="n">randint</span><span class="p">(</span><span class="mi">60</span><span class="p">,</span> <span class="mi">360</span><span class="p">))</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">charging</span> <span class="o">&amp;</span> <span class="n">parking</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;Stop parking at&#39;</span><span class="p">,</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">)</span> <span class="o">...</span> <span class="o">...</span> <span class="k">def</span> <span class="nf">bat_ctrl</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">env</span><span class="p">):</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;Bat. ctrl. started at&#39;</span><span class="p">,</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">)</span> <span class="o">...</span> <span class="c"># Intelligent charging behavior here …</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="n">randint</span><span class="p">(</span><span class="mi">30</span><span class="p">,</span> <span class="mi">90</span><span class="p">))</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;Bat. ctrl. done at&#39;</span><span class="p">,</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">)</span> <span class="o">...</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span> <span class="o">=</span> <span class="n">simpy</span><span class="o">.</span><span class="n">Environment</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">ev</span> <span class="o">=</span> <span class="n">EV</span><span class="p">(</span><span class="n">env</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">until</span><span class="o">=</span><span class="mi">310</span><span class="p">)</span> <span class="n">Start</span> <span class="n">parking</span> <span class="n">at</span> <span class="mi">29</span> <span class="n">Bat</span><span class="o">.</span> <span class="n">ctrl</span><span class="o">.</span> <span class="n">started</span> <span class="n">at</span> <span class="mi">29</span> <span class="n">Bat</span><span class="o">.</span> <span class="n">ctrl</span><span class="o">.</span> <span class="n">done</span> <span class="n">at</span> <span class="mi">83</span> <span class="n">Stop</span> <span class="n">parking</span> <span class="n">at</span> <span class="mi">305</span> </pre></div> <p>Again, nothing new (if you&#8217;ve read the <a class="reference external" href="/2014/04/05/simpy-events/">events guide</a>) and special is happening. SimPy processes are events, too, so you can yield them and will thus wait for them to get triggered. You can also wait for two events at the same time by concatenating them with <tt class="docutils literal">&amp;</tt>.</p> </div> <div class="section" id="id4"> <span id="interrupting-another-process"></span><h2>Interrupting another&nbsp;process</h2> <p>As usual, we now have another problem: Imagine, a trip is very urgent, but with the current implementation, we always need to wait until the battery is fully charged. If we could somehow interrupt that&nbsp;&#8230;</p> <p>Fortunate coincidence, there is indeed a way to do exactly this. You can call <tt class="docutils literal">interrupt()</tt> on a <tt class="docutils literal">Process</tt>. This will throw an <tt class="docutils literal">Interrupt</tt> exception into that process, resuming it&nbsp;immediately:</p> <div class="highlight"><pre><span class="o">&gt;&gt;&gt;</span> <span class="k">class</span> <span class="nc">EV</span><span class="p">:</span> <span class="o">...</span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">env</span><span class="p">):</span> <span class="o">...</span> <span class="bp">self</span><span class="o">.</span><span class="n">env</span> <span class="o">=</span> <span class="n">env</span> <span class="o">...</span> <span class="bp">self</span><span class="o">.</span><span class="n">drive_proc</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">drive</span><span class="p">(</span><span class="n">env</span><span class="p">))</span> <span class="o">...</span> <span class="o">...</span> <span class="k">def</span> <span class="nf">drive</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">env</span><span class="p">):</span> <span class="o">...</span> <span class="k">while</span> <span class="bp">True</span><span class="p">:</span> <span class="o">...</span> <span class="c"># Drive for 20-40 min</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="n">randint</span><span class="p">(</span><span class="mi">20</span><span class="p">,</span> <span class="mi">40</span><span class="p">))</span> <span class="o">...</span> <span class="o">...</span> <span class="c"># Park for 1 hour</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;Start parking at&#39;</span><span class="p">,</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">)</span> <span class="o">...</span> <span class="n">charging</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">bat_ctrl</span><span class="p">(</span><span class="n">env</span><span class="p">))</span> <span class="o">...</span> <span class="n">parking</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="mi">60</span><span class="p">)</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">charging</span> <span class="o">|</span> <span class="n">parking</span> <span class="o">...</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">charging</span><span class="o">.</span><span class="n">triggered</span><span class="p">:</span> <span class="o">...</span> <span class="c"># Interrupt charging if not already done.</span> <span class="o">...</span> <span class="n">charging</span><span class="o">.</span><span class="n">interrupt</span><span class="p">(</span><span class="s">&#39;Need to go!&#39;</span><span class="p">)</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;Stop parking at&#39;</span><span class="p">,</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">)</span> <span class="o">...</span> <span class="o">...</span> <span class="k">def</span> <span class="nf">bat_ctrl</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">env</span><span class="p">):</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;Bat. ctrl. started at&#39;</span><span class="p">,</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">)</span> <span class="o">...</span> <span class="k">try</span><span class="p">:</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="n">randint</span><span class="p">(</span><span class="mi">60</span><span class="p">,</span> <span class="mi">90</span><span class="p">))</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;Bat. ctrl. done at&#39;</span><span class="p">,</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">)</span> <span class="o">...</span> <span class="k">except</span> <span class="n">simpy</span><span class="o">.</span><span class="n">Interrupt</span> <span class="k">as</span> <span class="n">i</span><span class="p">:</span> <span class="o">...</span> <span class="c"># Onoes! Got interrupted before the charging was done.</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;Bat. ctrl. interrupted at&#39;</span><span class="p">,</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">,</span> <span class="s">&#39;msg:&#39;</span><span class="p">,</span> <span class="o">...</span> <span class="n">i</span><span class="o">.</span><span class="n">cause</span><span class="p">)</span> <span class="o">...</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span> <span class="o">=</span> <span class="n">simpy</span><span class="o">.</span><span class="n">Environment</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">ev</span> <span class="o">=</span> <span class="n">EV</span><span class="p">(</span><span class="n">env</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">until</span><span class="o">=</span><span class="mi">100</span><span class="p">)</span> <span class="n">Start</span> <span class="n">parking</span> <span class="n">at</span> <span class="mi">31</span> <span class="n">Bat</span><span class="o">.</span> <span class="n">ctrl</span><span class="o">.</span> <span class="n">started</span> <span class="n">at</span> <span class="mi">31</span> <span class="n">Stop</span> <span class="n">parking</span> <span class="n">at</span> <span class="mi">91</span> <span class="n">Bat</span><span class="o">.</span> <span class="n">ctrl</span><span class="o">.</span> <span class="n">interrupted</span> <span class="n">at</span> <span class="mi">91</span> <span class="n">msg</span><span class="p">:</span> <span class="n">Need</span> <span class="n">to</span> <span class="n">go</span><span class="err">!</span> </pre></div> <p>What <tt class="docutils literal">process.interrupt()</tt> actually does is removing the process&#8217; <tt class="docutils literal">_resume()</tt> method from the callbacks of the event that it is currently waiting for. And it will schedule an event that will throw the <tt class="docutils literal">Interrupt</tt> exception into the interrupted process as soon as&nbsp;possible.</p> <p>Since we don&#8217;t to anything special to the event, the interrupted process can yield the same event again after catching the <tt class="docutils literal">Interrupt</tt> – Imagine someone waiting for a shop to open. The person may get interrupted by a phone call. After finishing the call, he or she checks if the shop already opened and either enters or continues to&nbsp;wait.</p> <p>This guide is now, of course, also part of the <a class="reference external" href="http://simpy.readthedocs.org/en/latest/topical_guides/process_interaction.html">SimPy documentation</a>.</p> </div> SimPy: Events2014-04-05T11:52:00+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2014-04-05:2014/04/05/simpy-events/<p>It&#8217;s been <a class="reference external" href="/2014/01/20/simpy-environments/">a while</a> since the last SimPy guide, but I&#8217;ve been quite busy with developing and Open Sourcing <a class="reference external" href="https://mosaik.offis.de/2014/03/27/mosaik-is-open-source/">mosaik 2</a>.</p> <p><em>&quot;What&#8217;s SimPy again?&quot;</em> <a class="reference external" href="https://simpy.readthedocs.org">SimPy</a> is a process-based discrete-event simulation framework based on standard Python. Its event dispatcher is based on Python’s generators and can also be used for <a class="reference external" href="https://bitbucket.org/simpy/simpy.io/">asynchronous networking</a> or to implement multi-agent systems (with both, simulated and real&nbsp;communication).</p> <p>The first to guides provided a <a class="reference external" href="/2013/12/03/how-simpy-works/">general description of how SimPy works</a> and an <a class="reference external" href="/2014/01/20/simpy-environments/">explanation of SimPy environments</a>.</p> <p>This guide is all about&nbsp;events.</p> <p>SimPy includes an extensive set of event types for various purposes. All of them inherit <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.events.html#simpy.events.Event">simpy.events.Event</a>. The listing below shows the hierarchy of events built into&nbsp;SimPy:</p> <pre class="literal-block"> events.Event ↑ +— events.Timeout | +— events.Initialize | +— events.Process | +— events.Condition | ↑ | +— events.AllOf | | | +— events.AnyOf ⋮ +– [resource events] </pre> <p>This is the set of basic events. Events are extensible and resources, for example, define additional events. In this guide, we&#8217;ll focus on the events in the <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.events.html#module-simpy.events">simpy.events</a>&nbsp;module.</p> <div class="section" id="event-basics"> <h2>Event&nbsp;basics</h2> <p>SimPy events are very similar – if not identical — to deferreds, futures or promises. Instances of the class <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.events.html#simpy.events.Event">Event</a> are used to describe any kind of events. Events can be in one of the following states. An&nbsp;event</p> <ul class="simple"> <li>might happen (not&nbsp;triggered),</li> <li>is going to happen (triggered)&nbsp;or</li> <li>has happened&nbsp;(processed).</li> </ul> <p>They traverse these states exactly once in that order. Events are also tightly bound to time and time causes events to advance their&nbsp;state.</p> <p>Initially, events are not triggered and just objects in&nbsp;memory.</p> <p>If an event gets triggered, it is scheduled at a given time and inserted into SimPy&#8217;s event queue. The property <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.events.html#simpy.events.Event.triggered">Event.triggered</a> becomes <tt class="docutils literal">True</tt>.</p> <p>As long as the event is not <em>processed</em>, you can add <em>callbacks</em> to an event. Callbacks are callables that accept an event as parameter and are stored in the <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.events.html#simpy.events.Event.callbacks">Event.callbacks</a>&nbsp;list.</p> <p>An event becomes <em>processed</em> when SimPy pops it from the event queue and calls all of its callbacks. It is now no longer possible to add callbacks. The property <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.events.html#simpy.events.Event.processed">Event.processed</a> becomes <tt class="docutils literal">True</tt>.</p> <p>Events also have a <em>value</em>. The value can be set before or when the event is triggered and can be retrieved via <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.events.html#simpy.events.Event.value">Event.value</a> or, within a process, by yielding the event (<tt class="docutils literal">value = yield event</tt>).</p> <div class="section" id="adding-callbacks-to-an-event"> <h3>Adding callbacks to an&nbsp;event</h3> <p>&quot;What? Callbacks? I&#8217;ve never seen no callbacks!&quot;, you might think if you have worked your way through the <a class="reference external" href="http://simpy.readthedocs.org/en/latest/simpy_intro/index.html">tutorial</a>.</p> <p>That&#8217;s on purpose. The most common way to add a callback to an event is yielding it from your process function (<tt class="docutils literal">yield event</tt>). This will add the process&#8217; <em>_resume()</em> method as a callback. That&#8217;s how your process gets resumed when it yielded an&nbsp;event.</p> <p>However, you can add any callable object (function) to the list of callbacks as long as it accepts an event instance as its single&nbsp;parameter:</p> <div class="highlight"><pre><span class="o">&gt;&gt;&gt;</span> <span class="kn">import</span> <span class="nn">simpy</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">my_callback</span><span class="p">(</span><span class="n">event</span><span class="p">):</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;Called back from&#39;</span><span class="p">,</span> <span class="n">event</span><span class="p">)</span> <span class="o">...</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span> <span class="o">=</span> <span class="n">simpy</span><span class="o">.</span><span class="n">Environment</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">event</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">event</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">event</span><span class="o">.</span><span class="n">callbacks</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">my_callback</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">event</span><span class="o">.</span><span class="n">callbacks</span> <span class="p">[</span><span class="o">&lt;</span><span class="n">function</span> <span class="n">my_callback</span> <span class="n">at</span> <span class="mi">0</span><span class="n">x</span><span class="o">...&gt;</span><span class="p">]</span> </pre></div> <p>If an event has been <em>processed</em>, all of its <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.events.html#simpy.events.Event.callbacks">Event.callbacks</a> have been executed and the attribute is set to <tt class="docutils literal">None</tt>. This is to prevent you from adding more callbacks – these would of course never get called because the event has already&nbsp;happened.</p> <p>Processes are smart about this, though. If you yield a processed event, <em>_resume()</em> will immediately resume your process with the value of the event (because there is nothing to wait&nbsp;for).</p> </div> <div class="section" id="triggering-events"> <h3>Triggering&nbsp;events</h3> <p>When events are triggered, they can either <em>succeed</em> or <em>fail</em>. For example, if an event is to be triggered at the end of a computation and everything works out fine, the event will <em>succeed</em>. If an exceptions occurs during that computation, the event will <em>fail</em>.</p> <p>To trigger an event and mark it as successful, you can use <tt class="docutils literal">Event.succeed(value=None)</tt>. You can optionally pass a <em>value</em> to it (e.g., the results of a&nbsp;computation).</p> <p>To trigger an event and mark it as failed, call <tt class="docutils literal">Event.fail(exception)</tt> and pass an <tt class="docutils literal">Exception</tt> instance to it (e.g., the exception you caught during your failed&nbsp;computation).</p> <p>There is also a generic way to trigger an event: <tt class="docutils literal">Event.trigger(event)</tt>. This will take the value and outcome (success or failure) of the event passed to&nbsp;it.</p> <p>All three methods return the event instance they are bound to. This allows you to do things like <tt class="docutils literal">yield <span class="pre">Event(env).succeed()</span></tt>.</p> </div> </div> <div class="section" id="example-usages-for-event"> <h2>Example usages for <tt class="docutils literal">Event</tt></h2> <p>The simple mechanics outlined above provide a great flexibility in the way events (even the basic <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.events.html#simpy.events.Event">Event</a>) can be&nbsp;used.</p> <p>One example for this is that events can be shared. They can be created by a process or outside of the context of a process. They can be passed to other processes and&nbsp;chained:</p> <div class="highlight"><pre><span class="o">&gt;&gt;&gt;</span> <span class="k">class</span> <span class="nc">School</span><span class="p">:</span> <span class="o">...</span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">env</span><span class="p">):</span> <span class="o">...</span> <span class="bp">self</span><span class="o">.</span><span class="n">env</span> <span class="o">=</span> <span class="n">env</span> <span class="o">...</span> <span class="bp">self</span><span class="o">.</span><span class="n">class_ends</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">event</span><span class="p">()</span> <span class="o">...</span> <span class="bp">self</span><span class="o">.</span><span class="n">pupil_procs</span> <span class="o">=</span> <span class="p">[</span><span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">pupil</span><span class="p">())</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">3</span><span class="p">)]</span> <span class="o">...</span> <span class="bp">self</span><span class="o">.</span><span class="n">bell_proc</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">bell</span><span class="p">())</span> <span class="o">...</span> <span class="o">...</span> <span class="k">def</span> <span class="nf">bell</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="o">...</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">2</span><span class="p">):</span> <span class="o">...</span> <span class="k">yield</span> <span class="bp">self</span><span class="o">.</span><span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="mi">45</span><span class="p">)</span> <span class="o">...</span> <span class="bp">self</span><span class="o">.</span><span class="n">class_ends</span><span class="o">.</span><span class="n">succeed</span><span class="p">()</span> <span class="o">...</span> <span class="bp">self</span><span class="o">.</span><span class="n">class_ends</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">env</span><span class="o">.</span><span class="n">event</span><span class="p">()</span> <span class="o">...</span> <span class="k">print</span><span class="p">()</span> <span class="o">...</span> <span class="o">...</span> <span class="k">def</span> <span class="nf">pupil</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="o">...</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">2</span><span class="p">):</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39; \o/&#39;</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s">&#39;&#39;</span><span class="p">)</span> <span class="o">...</span> <span class="k">yield</span> <span class="bp">self</span><span class="o">.</span><span class="n">class_ends</span> <span class="o">...</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">school</span> <span class="o">=</span> <span class="n">School</span><span class="p">(</span><span class="n">env</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span><span class="o">.</span><span class="n">run</span><span class="p">()</span> \<span class="n">o</span><span class="o">/</span> \<span class="n">o</span><span class="o">/</span> \<span class="n">o</span><span class="o">/</span> \<span class="n">o</span><span class="o">/</span> \<span class="n">o</span><span class="o">/</span> \<span class="n">o</span><span class="o">/</span> </pre></div> <p>This can also be used like the <em>passivate / reactivate</em> known from SimPy 2. The pupils <em>passivate</em> when class begins and are <em>reactivated</em> when the bell&nbsp;rings.</p> </div> <div class="section" id="let-time-pass-by-the-timeout"> <h2>Let time pass by: the <tt class="docutils literal">Timeout</tt></h2> <p>To actually let time pass in a simulation, there is the <em>timeout</em> event. A timeout has two parameters: a <em>delay</em> and an optional <em>value</em>: <tt class="docutils literal">Timeout(delay, value=None)</tt>. It triggers itself during its creation and schedules itself at <tt class="docutils literal">now + delay</tt>. Thus, the <tt class="docutils literal">succeed()</tt> and <tt class="docutils literal">fail()</tt> methods cannot be called again and you have to pass the event value to it when you create the&nbsp;timeout.</p> <p>The delay can be any kind of number, usually an <em>int</em> or <em>float</em> as long as it supports comparison and&nbsp;addition.</p> </div> <div class="section" id="processes-are-events-too"> <h2>Processes are events,&nbsp;too</h2> <p>SimPy processes (as created by <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.events.html#simpy.events.Process">Process</a> or <tt class="docutils literal">env.process()</tt>) have the nice property of being events,&nbsp;too.</p> <p>That means, that a process can yield another process. It will then be resumed when the other process ends. The event&#8217;s value will be the return value of that&nbsp;process:</p> <div class="highlight"><pre><span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">sub</span><span class="p">(</span><span class="n">env</span><span class="p">):</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="o">...</span> <span class="k">return</span> <span class="mi">23</span> <span class="o">...</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">parent</span><span class="p">(</span><span class="n">env</span><span class="p">):</span> <span class="o">...</span> <span class="n">ret</span> <span class="o">=</span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="n">sub</span><span class="p">(</span><span class="n">env</span><span class="p">))</span> <span class="o">...</span> <span class="k">return</span> <span class="n">ret</span> <span class="o">...</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="n">parent</span><span class="p">(</span><span class="n">env</span><span class="p">)))</span> <span class="mi">23</span> </pre></div> <p>The example above will only work in Python &gt;= 3.3. As a workaround for older Python versions, you can use <tt class="docutils literal">env.exit(23)</tt> with the same&nbsp;effect.</p> <p>When a process is created, it schedules an <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.events.html#simpy.events.Initialize">Initialize</a> event which will start the execution of the process when triggered. You usually won&#8217;t have to deal with this type of&nbsp;event.</p> <p>If you don&#8217;t want a process to start immediately but after a certain delay, you can use <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.util.html#simpy.util.start_delayed">simpy.util.start_delayed()</a>. This method returns a helper process that uses a <em>timeout</em> before actually starting a&nbsp;process.</p> <p>The example from above, but with a delayed start of <tt class="docutils literal">sub()</tt>:</p> <div class="highlight"><pre><span class="o">&gt;&gt;&gt;</span> <span class="kn">from</span> <span class="nn">simpy.util</span> <span class="kn">import</span> <span class="n">start_delayed</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">sub</span><span class="p">(</span><span class="n">env</span><span class="p">):</span> <span class="o">...</span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="o">...</span> <span class="k">return</span> <span class="mi">23</span> <span class="o">...</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">parent</span><span class="p">(</span><span class="n">env</span><span class="p">):</span> <span class="o">...</span> <span class="n">start</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span> <span class="o">...</span> <span class="n">sub_proc</span> <span class="o">=</span> <span class="k">yield</span> <span class="n">start_delayed</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">sub</span><span class="p">(</span><span class="n">env</span><span class="p">),</span> <span class="n">delay</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span> <span class="o">...</span> <span class="k">assert</span> <span class="n">env</span><span class="o">.</span><span class="n">now</span> <span class="o">-</span> <span class="n">start</span> <span class="o">==</span> <span class="mi">3</span> <span class="o">...</span> <span class="o">...</span> <span class="n">ret</span> <span class="o">=</span> <span class="k">yield</span> <span class="n">sub_proc</span> <span class="o">...</span> <span class="k">return</span> <span class="n">ret</span> <span class="o">...</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="n">parent</span><span class="p">(</span><span class="n">env</span><span class="p">)))</span> <span class="mi">23</span> </pre></div> </div> <div class="section" id="waiting-for-for-multiple-events-at-once"> <h2>Waiting for for multiple events at&nbsp;once</h2> <p>Sometimes, you want to wait for more than one event at the same time. For example, you may want to wait for a resource, but not for an unlimited amount of time. Or you may want to wait until all of a number of events have&nbsp;happened.</p> <p>SimPy therefore offers the <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.events.html#simpy.events.AnyOf">AnyOf</a> and <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.events.html#simpy.events.AllOf">AllOf</a> events which both are a <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.events.html#simpy.events.Condition">Condition</a>&nbsp;event.</p> <p>Both take a list of events as an argument and are triggered if at least one of them is triggered or all of&nbsp;them:</p> <div class="highlight"><pre><span class="o">&gt;&gt;&gt;</span> <span class="kn">from</span> <span class="nn">simpy.events</span> <span class="kn">import</span> <span class="n">AnyOf</span><span class="p">,</span> <span class="n">AllOf</span><span class="p">,</span> <span class="n">Event</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">events</span> <span class="o">=</span> <span class="p">[</span><span class="n">Event</span><span class="p">(</span><span class="n">env</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">3</span><span class="p">)]</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">a</span> <span class="o">=</span> <span class="n">AnyOf</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">events</span><span class="p">)</span> <span class="c"># Triggers if at least one of &quot;events&quot; is triggered.</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">b</span> <span class="o">=</span> <span class="n">AllOf</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">events</span><span class="p">)</span> <span class="c"># Triggers if all each of &quot;events&quot; is triggered.</span> </pre></div> <p>The value of a condition event is a dictionary with an entry for every triggered event. In the case of <tt class="docutils literal">AllOf</tt>, the size of that dictionary will always be the same as the length of the event list. The value dict of <tt class="docutils literal">AnyOf</tt> will have at least one entry. In both cases, the event instances are used as keys and the event values will be the&nbsp;values.</p> <p>As a shorthand for <tt class="docutils literal">AllOf</tt> and <tt class="docutils literal">AnyOf</tt>, you can also use the logical operators <tt class="docutils literal">&amp;</tt> (and) and <tt class="docutils literal">|</tt> (or):</p> <div class="highlight"><pre><span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">test_condition</span><span class="p">(</span><span class="n">env</span><span class="p">):</span> <span class="o">...</span> <span class="n">t1</span><span class="p">,</span> <span class="n">t2</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">value</span><span class="o">=</span><span class="s">&#39;spam&#39;</span><span class="p">),</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="n">value</span><span class="o">=</span><span class="s">&#39;eggs&#39;</span><span class="p">)</span> <span class="o">...</span> <span class="n">ret</span> <span class="o">=</span> <span class="k">yield</span> <span class="n">t1</span> <span class="o">|</span> <span class="n">t2</span> <span class="o">...</span> <span class="k">assert</span> <span class="n">ret</span> <span class="o">==</span> <span class="p">{</span><span class="n">t1</span><span class="p">:</span> <span class="s">&#39;spam&#39;</span><span class="p">}</span> <span class="o">...</span> <span class="o">...</span> <span class="n">t1</span><span class="p">,</span> <span class="n">t2</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">value</span><span class="o">=</span><span class="s">&#39;spam&#39;</span><span class="p">),</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="n">value</span><span class="o">=</span><span class="s">&#39;eggs&#39;</span><span class="p">)</span> <span class="o">...</span> <span class="n">ret</span> <span class="o">=</span> <span class="k">yield</span> <span class="n">t1</span> <span class="o">&amp;</span> <span class="n">t2</span> <span class="o">...</span> <span class="k">assert</span> <span class="n">ret</span> <span class="o">==</span> <span class="p">{</span><span class="n">t1</span><span class="p">:</span> <span class="s">&#39;spam&#39;</span><span class="p">,</span> <span class="n">t2</span><span class="p">:</span> <span class="s">&#39;eggs&#39;</span><span class="p">}</span> <span class="o">...</span> <span class="o">...</span> <span class="c"># You can also concatenate &amp; and |</span> <span class="o">...</span> <span class="n">e1</span><span class="p">,</span> <span class="n">e2</span><span class="p">,</span> <span class="n">e3</span> <span class="o">=</span> <span class="p">[</span><span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="n">i</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">3</span><span class="p">)]</span> <span class="o">...</span> <span class="k">yield</span> <span class="p">(</span><span class="n">e1</span> <span class="o">|</span> <span class="n">e2</span><span class="p">)</span> <span class="o">&amp;</span> <span class="n">e3</span> <span class="o">...</span> <span class="k">assert</span> <span class="nb">all</span><span class="p">(</span><span class="n">e</span><span class="o">.</span><span class="n">triggered</span> <span class="k">for</span> <span class="n">e</span> <span class="ow">in</span> <span class="p">[</span><span class="n">e1</span><span class="p">,</span> <span class="n">e2</span><span class="p">,</span> <span class="n">e3</span><span class="p">])</span> <span class="o">...</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">proc</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="n">test_condition</span><span class="p">(</span><span class="n">env</span><span class="p">))</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">env</span><span class="o">.</span><span class="n">run</span><span class="p">()</span> </pre></div> <p>That&#8217;s it. I&#8217;ve already started with the next guide – about process interaction – so hopefully it won&#8217;t take to long until publish it this&nbsp;time.</p> </div> mosaik – An open co-simulation framework for smart energy systems2014-03-27T13:37:00+01:00Stefan Scherfketag:stefan.sofa-rockers.org,2014-03-27:2014/03/27/mosaik/<p>Since nearly four years, we’ve been developing and working with <a class="reference external" href="http://mosaik.offis.de">mosaik</a> within our research group at <span class="caps">OFFIS</span>. Now, we finally had the opportunity to release it as Open Source&nbsp;Software.</p> <p>You can read the full announcement <a class="reference external" href="https://mosaik.offis.de/2014/03/27/mosaik-is-open-source/">here</a>.</p> SimPy: Environments2014-01-20T17:33:00+01:00Stefan Scherfketag:stefan.sofa-rockers.org,2014-01-20:2014/01/20/simpy-environments/<p>This is the second in a <a class="reference external" href="/2013/12/03/how-simpy-works/">series</a> of guides that describe how SimPy works and how to use it best. This time I’ll discuss <em>environments</em>.</p> <p>A simulation environment manages the simulation time as well as the scheduling and processing of events. It also provides means to step through or execute the&nbsp;simulation.</p> <p>The base class for all environments is <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.core.html#simpy.core.BaseEnvironment">BaseEnvironment</a>. “Normal” simulations usually use its subclass <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.core.html#simpy.core.Environment">Environment</a>. For real-time simulations, SimPy provides a <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.rt.html#simpy.rt.RealtimeEnvironment">RealtimeEnvironment</a> (more on that in another&nbsp;guide).</p> <div class="section" id="simulation-control"> <h2>Simulation&nbsp;control</h2> <p>SimPy is very flexible in terms of simulation execution. You can run your simulation until there is no more event, until a certain simulation time is reached, or until a certain event is triggered. You can also step through the simulation event by event. Furthermore, you can mix these things as you&nbsp;like.</p> <p>For example, you could run your simulation until an interesting event occurs. You could then step through the simulation event by event for a while; and finally run the simulation until there is no more event left and your processes all have&nbsp;terminated.</p> <p>The most important method here is <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.core.html#simpy.core.Environment.run">Environment.run()</a>:</p> <ul> <li><p class="first">If you call it without any argument (<tt class="docutils literal">env.run()</tt>), it steps through the simulation until there is no more event&nbsp;left.</p> <p><em><span class="caps">WARNING</span>:</em> If your processes run forever (<tt class="docutils literal">while True: yield env.timeout(1)</tt>), this method will never terminate (unless you kill your script by e.g., pressing <tt class="docutils literal"><span class="pre">Ctrl-C</span></tt>).</p> </li> <li><p class="first">In most cases it is more advisable to stop your simulation when it reaches a certain simulation time. Therefore, you can pass the desired time via the <em>until</em> parameter, e.g.: <tt class="docutils literal">env.run(until=10)</tt>.</p> <p>The simulation will then stop when the internal clock reaches 10 but will not process any events scheduled for time 10. This is similar to a new environment where the clock is 0 but (obviously) no events have yet been&nbsp;processed.</p> <p>If you want to integrate your simulation in a <span class="caps">GUI</span> and want to draw a process bar, you can repeatedly call this function with increasing <em>until</em> values and update your progress bar after each&nbsp;call:</p> <div class="highlight"><pre><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">100</span><span class="p">):</span> <span class="n">env</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">until</span><span class="o">=</span><span class="n">i</span><span class="p">)</span> <span class="n">progressbar</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">i</span><span class="p">)</span> </pre></div> </li> <li><p class="first">Instead of passing a number to <tt class="docutils literal">run()</tt>, you can also pass any event to it. <tt class="docutils literal">run()</tt> will then return when the event has been&nbsp;processed.</p> <p>Assuming that the current time is 0, <tt class="docutils literal">env.run(until=env.timeout(5))</tt> is equivalent to <tt class="docutils literal">env.run(until=5)</tt>.</p> <p>You can also pass other types of events (remember, that a <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.events.html#simpy.events.Process">Process</a> is an event,&nbsp;too):</p> <div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="kn">import</span> <span class="nn">simpy</span> <span class="go">&gt;&gt;&gt;</span> <span class="gp">&gt;&gt;&gt; </span><span class="k">def</span> <span class="nf">my_proc</span><span class="p">(</span><span class="n">env</span><span class="p">):</span> <span class="gp">... </span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="gp">... </span> <span class="k">return</span> <span class="s">&#39;Monty Python’s Flying Circus&#39;</span> <span class="go">&gt;&gt;&gt;</span> <span class="gp">&gt;&gt;&gt; </span><span class="n">env</span> <span class="o">=</span> <span class="n">simpy</span><span class="o">.</span><span class="n">Environment</span><span class="p">()</span> <span class="gp">&gt;&gt;&gt; </span><span class="n">proc</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="n">my_proc</span><span class="p">(</span><span class="n">env</span><span class="p">))</span> <span class="gp">&gt;&gt;&gt; </span><span class="n">env</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">until</span><span class="o">=</span><span class="n">proc</span><span class="p">)</span> <span class="go">&#39;Monty Python’s Flying Circus&#39;</span> </pre></div> </li> </ul> <p>To step through the simulation event by event, the environment offers <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.core.html#simpy.core.Environment.peek">peek()</a> and <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.core.html#simpy.core.Environment.step">step()</a>.</p> <p><tt class="docutils literal">peek()</tt> returns the time of the next scheduled event of <em>infinity</em> (<tt class="docutils literal"><span class="pre">float('inf')</span></tt>) of no more event is&nbsp;scheduled.</p> <p><tt class="docutils literal">step()</tt> processes the next scheduled event. It raises an <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.core.html#simpy.core.EmptySchedule">EmptySchedule</a> exception if no event is&nbsp;available.</p> <p>In a typical use case, you use these methods in a loop&nbsp;like:</p> <div class="highlight"><pre><span class="n">until</span> <span class="o">=</span> <span class="mi">10</span> <span class="k">while</span> <span class="n">env</span><span class="o">.</span><span class="n">peek</span><span class="p">()</span> <span class="o">&lt;</span> <span class="n">until</span><span class="p">:</span> <span class="n">env</span><span class="o">.</span><span class="n">step</span><span class="p">()</span> </pre></div> </div> <div class="section" id="state-access"> <h2>State&nbsp;access</h2> <p>The environment allows you to get the current simulation time via the <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.core.html#simpy.core.Environment.now">Environment.now</a> property. The simulation time is a number without unit and is increased via <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.events.html#simpy.events.Timeout">Timeout</a>&nbsp;events.</p> <p>By default, <tt class="docutils literal">now</tt> starts at 0, but you can pass an <tt class="docutils literal">initial_time</tt> to the Environment to use something&nbsp;else.</p> <p><em><span class="caps">NOTE</span>:</em> Although the simulation time is technically unitless, you can pretend that it is, for example, in seconds and use it like a timestamp returned by <a class="reference external" href="http://docs.python.org/3/library/time#time.time">time.time()</a> to calculate a date or the day of the&nbsp;week.</p> <p>The property <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.core.html#simpy.core.Environment.active_process">Environment.active_process</a> is comparable to <a class="reference external" href="http://docs.python.org/3/library/os#os.getpid">os.getpid()</a> and is either <tt class="docutils literal">None</tt> or pointing at the currently active Process. A process is <em>active</em> when its process function is being executed. It becomes <em>inactive</em> (or suspended) when it yields an&nbsp;event.</p> <p>Thus, it makes only sense to access this property from within a process function or a function that is called by your process&nbsp;function:</p> <pre class="literal-block"> &gt;&gt;&gt; def subfunc(env): ... print(env.active_process) # will print &quot;p1&quot; &gt;&gt;&gt; &gt;&gt;&gt; def my_proc(env): ... while True: ... print(env.active_process) # will print &quot;p1&quot; ... subfunc(env) ... yield env.timeout(1) &gt;&gt;&gt; &gt;&gt;&gt; env = simpy.Environment() &gt;&gt;&gt; p1 = env.process(my_proc(env)) &gt;&gt;&gt; env.active_process # None &gt;&gt;&gt; env.step() &lt;Process(my_proc) object at 0x...&gt; &lt;Process(my_proc) object at 0x...&gt; &gt;&gt;&gt; env.active_process # None </pre> <p>An exemplary use case for this is the resource system: If a process function calls <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.resources.resource.html#simpy.resources.resource.Resource.request">request()</a> to request a resource, the resource determines the requesting process via <tt class="docutils literal">env.active_process</tt>. Take a <a class="reference external" href="https://bitbucket.org/simpy/simpy/src/3.0.2/simpy/resources/base.py#cl-35">look at the code</a> to see how we do this&nbsp;:-).</p> </div> <div class="section" id="event-creation"> <h2>Event&nbsp;creation</h2> <p>To create events, you normally have to import <tt class="docutils literal">simpy.events</tt>, instantiate the event class and pass a reference to the environment to it. To reduce the amount of typing, the Environment provides some shortcuts for event creation. For example, <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.core.html#simpy.core.Environment.event">Environment.event()</a> is equivalent to <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.events.html#simpy.events.Event">simpy.events.Event(env)</a>.</p> <p>Other shortcuts&nbsp;are:</p> <ul class="simple"> <li><a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.core.html#simpy.core.Environment.process">Environment.process()</a></li> <li><a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.core.html#simpy.core.Environment.timeout">Environment.timeout()</a></li> <li><a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.core.html#simpy.core.Environment.all_of">Environment.all_of()</a></li> <li><a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.core.html#simpy.core.Environment.any_of">Environment.any_of()</a></li> </ul> <p>More details on what the events do can be found in the guide to events (not yet written&nbsp;:-)).</p> </div> <div class="section" id="miscellaneous"> <h2>Miscellaneous</h2> <p>Since Python 3.3, a generator function can have a return&nbsp;value:</p> <div class="highlight"><pre><span class="k">def</span> <span class="nf">my_proc</span><span class="p">(</span><span class="n">env</span><span class="p">):</span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="k">return</span> <span class="mi">42</span> </pre></div> <p>In SimPy, this can be used to provide return values for processes that can be used by other&nbsp;processes:</p> <div class="highlight"><pre><span class="k">def</span> <span class="nf">other_proc</span><span class="p">(</span><span class="n">env</span><span class="p">):</span> <span class="n">ret_val</span> <span class="o">=</span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="n">my_proc</span><span class="p">(</span><span class="n">env</span><span class="p">))</span> <span class="k">assert</span> <span class="n">ret_val</span> <span class="o">==</span> <span class="mi">42</span> </pre></div> <p>Internally, Python passes the return value as parameter to the <tt class="docutils literal">StopIteration</tt> exception that it raises when a generator is exhausted. So in Python 2.7 and 3.2 you could replace the <tt class="docutils literal">return 42</tt> with a <tt class="docutils literal">raise StopIteration(42)</tt> to achieve the same&nbsp;result.</p> <p>To keep your code more readable, the environment provides the method <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.core.html#simpy.core.Environment.exit">exit()</a> to do exactly&nbsp;this:</p> <div class="highlight"><pre><span class="k">def</span> <span class="nf">my_proc</span><span class="p">(</span><span class="n">env</span><span class="p">):</span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="n">env</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="mi">42</span><span class="p">)</span> <span class="c"># Py2 equivalent to &quot;return 42&quot;</span> </pre></div> <p>You can find the complete guide on <a class="reference external" href="http://simpy.readthedocs.org/en/latest/topical_guides/environments.html">Read the Docs</a>. The next one will be about events and the event types provided by&nbsp;SimPy.</p> </div> How SimPy works2013-12-03T09:48:00+01:00Stefan Scherfketag:stefan.sofa-rockers.org,2013-12-03:2013/12/03/how-simpy-works/<p>SimPy is a discrete event-simulation library for Python. Recently, the new version 3 has ben <a class="reference external" href="/2013/10/11/simpy-3-released/">released</a>. With that new release, we also changed the structure of the documentation. It now features a <a class="reference external" href="http://simpy.readthedocs.org/en/latest/simpy_intro/">Tutorial</a>, <a class="reference external" href="http://simpy.readthedocs.org/en/latest/topical_guides/">Topical Guides</a>, an <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/"><span class="caps">API</span> Reference</a> and a list of <a class="reference external" href="http://simpy.readthedocs.org/en/latest/examples/">examples</a>. Most of the topical guides have yet to written, though. So this is the first in a series of posts describing various concepts of&nbsp;SimPy.</p> <p>If you break SimPy down, it is just an asynchronous event dispatcher. You generate events and schedule them at a given simulation time. Events are sorted by priority, simulation time, and an increasing event id. An event also has a list of callbacks, which are executed when the event is triggered and processed by the event loop. Events may also have a return&nbsp;value.</p> <p>The components involved in this are the <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.core.html#simpy.core.Environment">Environment</a>, <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.events.html#module-simpy.events">events</a> and the process functions that you&nbsp;write.</p> <p>Process functions implement your simulation model, that is, they define the behavior of your simulation. They are plain Python generator functions that yield instances of <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.events.html#simpy.events.Event">Event</a>.</p> <p>The environment stores these events in its event list and keeps track of the current simulation&nbsp;time.</p> <p>If a process function yields and event, SimPy adds the process to the event&#8217;s callbacks and suspends the process until the event is triggered and processed. When a process waiting for an event is resumed, it will also receive the event&#8217;s&nbsp;value.</p> <p>Here is a very simple example that illustrates all this; the code is more verbose than it needs to be to make things extra clear. You find a compact version of it at the end of this&nbsp;section:</p> <div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="kn">import</span> <span class="nn">simpy</span> <span class="go">&gt;&gt;&gt;</span> <span class="gp">&gt;&gt;&gt; </span><span class="k">def</span> <span class="nf">example</span><span class="p">(</span><span class="n">env</span><span class="p">):</span> <span class="gp">... </span> <span class="n">event</span> <span class="o">=</span> <span class="n">simpy</span><span class="o">.</span><span class="n">events</span><span class="o">.</span><span class="n">Timeout</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">delay</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">value</span><span class="o">=</span><span class="mi">42</span><span class="p">)</span> <span class="gp">... </span> <span class="n">value</span> <span class="o">=</span> <span class="k">yield</span> <span class="n">event</span> <span class="gp">... </span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;now=</span><span class="si">%d</span><span class="s">, value=</span><span class="si">%d</span><span class="s">&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">,</span> <span class="n">value</span><span class="p">))</span> <span class="go">&gt;&gt;&gt;</span> <span class="gp">&gt;&gt;&gt; </span><span class="n">env</span> <span class="o">=</span> <span class="n">simpy</span><span class="o">.</span><span class="n">Environment</span><span class="p">()</span> <span class="gp">&gt;&gt;&gt; </span><span class="n">example_gen</span> <span class="o">=</span> <span class="n">example</span><span class="p">(</span><span class="n">env</span><span class="p">)</span> <span class="gp">&gt;&gt;&gt; </span><span class="n">p</span> <span class="o">=</span> <span class="n">simpy</span><span class="o">.</span><span class="n">events</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">example_gen</span><span class="p">)</span> <span class="go">&gt;&gt;&gt;</span> <span class="gp">&gt;&gt;&gt; </span><span class="n">env</span><span class="o">.</span><span class="n">run</span><span class="p">()</span> <span class="go">now=1, value=42</span> </pre></div> <p>The <tt class="docutils literal">example()</tt> process function above first creates a <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.events.html#simpy.events.Timeout">Timeout</a> event. It passes the environment, a delay, and a value to it. The Timeout schedules itself at <tt class="docutils literal">now + delay</tt> (that&#8217;s why the environment is required); other event types usually schedule themselves at the current simulation&nbsp;time.</p> <p>The process function then yields the event and thus gets suspended. It is resumed, when SimPy processes the Timeout event. The process function also receives the event&#8217;s value (42) &#8212; this is, however, optional, so <tt class="docutils literal">yield event</tt> would have been okay if the you were not interested in the value or if the event had no value at&nbsp;all.</p> <p>Finally, the process function prints the current simulation time (that is accessible via the environment&#8217;s <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.core.html#simpy.core.Environment.now">now</a> attribute) and the Timeout&#8217;s&nbsp;value.</p> <p>If all required process functions are defined, you can instantiate all objects for your simulation. In most cases, you start by creating an instance of <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.core.html#simpy.core.Environment">Environment</a>, because you&#8217;ll need to pass it around a lot when creating everything&nbsp;else.</p> <p>Starting a process function involves two&nbsp;things:</p> <ol class="arabic simple"> <li>You have to call the process function to create a generator object. (This will not execute any code of that function yet. Please read <a class="reference external" href="http://stackoverflow.com/questions/231767/the-python-yield-keyword-explained/231855#231855">The Python yield keyword explained</a>, to understand why this is the&nbsp;case.)</li> <li>You then create an instance of <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.events.html#simpy.events.Process">Process</a> and pass the environment and the generator object to it. This will schedule an <a class="reference external" href="http://simpy.readthedocs.org/en/latest/api_reference/simpy.events.html#simpy.events.Initialize">Initialize</a> event at the current simulation time which starts the execution of the process function. The process instance is also an event that is triggered when the process function returns. The guide to events <em>(not yet written)</em> explains why this is&nbsp;handy.</li> </ol> <p>Finally, you can start SimPy&#8217;s event loop. By default, it will run as long as there are events in the event list, but you can also let it stop earlier by providing an <tt class="docutils literal">until</tt> argument.</p> <p>The next guide/post will describe the environment and its interactions with events and process functions in more&nbsp;detail.</p> <div class="section" id="best-practice-version-of-the-example-above"> <h2>“Best practice” version of the example&nbsp;above</h2> <div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="kn">import</span> <span class="nn">simpy</span> <span class="go">&gt;&gt;&gt;</span> <span class="gp">&gt;&gt;&gt; </span><span class="k">def</span> <span class="nf">example</span><span class="p">(</span><span class="n">env</span><span class="p">):</span> <span class="gp">... </span> <span class="n">value</span> <span class="o">=</span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">value</span><span class="o">=</span><span class="mi">42</span><span class="p">)</span> <span class="gp">... </span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;now=</span><span class="si">%d</span><span class="s">, value=</span><span class="si">%d</span><span class="s">&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">,</span> <span class="n">value</span><span class="p">))</span> <span class="go">&gt;&gt;&gt;</span> <span class="gp">&gt;&gt;&gt; </span><span class="n">env</span> <span class="o">=</span> <span class="n">simpy</span><span class="o">.</span><span class="n">Environment</span><span class="p">()</span> <span class="gp">&gt;&gt;&gt; </span><span class="n">p</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="n">example</span><span class="p">(</span><span class="n">env</span><span class="p">))</span> <span class="gp">&gt;&gt;&gt; </span><span class="n">env</span><span class="o">.</span><span class="n">run</span><span class="p">()</span> <span class="go">now=1, value=42</span> </pre></div> </div> SimPy 3.0.2 released2013-10-24T22:37:00+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2013-10-24:2013/10/24/simpy-302-released/<p><a class="reference external" href="https://pypi.python.org/pypi/simpy/3.0.2">SimPy 3.0.2</a> has just been released. It fixes the default capacity for <tt class="docutils literal">Container</tt>, <tt class="docutils literal">Store</tt> and <tt class="docutils literal">FilterStore</tt>, which is now&nbsp;unlimited.</p> <div class="section" id="what-is-simpy"> <h2>What is&nbsp;SimPy?</h2> <p>SimPy is a process-based discrete-event simulation framework based on standard Python. Its event dispatcher is based on Python’s generators and can also be used for asynchronous networking or to implement multi-agent systems (with both, simulated and real&nbsp;communication).</p> <p>SimPy 3 has been rewritten from scratch. This is the <a class="reference external" href="http://stefan.sofa-rockers.org/2013/10/11/simpy-3-released/">original release</a>&nbsp;announcement:</p> <blockquote> <p>After one and a half year of development and many iterations, prototypes, and endless discussions, we now proudly release SimPy&nbsp;3.</p> <p>SimPy 3 has been completely rewritten from scratch. Our main goals were to simplify the <span class="caps">API</span> and code base as well as making SimPy more flexible and extensible. Some of the most important changes&nbsp;are:</p> <ul class="simple"> <li>Stronger focus on events. Processes yield event instances and are suspended until the event is triggered. An example for an event is a <em>timeout</em> (formerly known as <em>hold</em>), but even processes are now events, too (you can wait until a process&nbsp;terminates).</li> <li>Events can be combined with <tt class="docutils literal">&amp;</tt> (and) and <tt class="docutils literal">|</tt> (or) to create <em>condition events</em>.</li> <li>Process can now be defined by any generator function. You don&#8217;t have to subclass <tt class="docutils literal">Process</tt> anymore.</li> <li>No more global simulation state. Every simulation stores its state in an <em>environment</em> which is comparable to the old <tt class="docutils literal">Simulation</tt> class.</li> <li>Improved resource system with newly added resource&nbsp;types.</li> <li>Removed plotting and <span class="caps">GUI</span> capabilities. <a class="reference external" href="http://qt-project.org/wiki/PySide">Pyside</a> and <a class="reference external" href="http://matplotlib.org/">matplotlib</a> are much better with&nbsp;this.</li> <li>Greatly improved test suite. Its cleaner, and the tests are shorter and more&nbsp;numerous.</li> <li>Completely overhauled&nbsp;documentation.</li> </ul> <p>There is a <a class="reference external" href="https://simpy.readthedocs.org/en/latest/about/history.html">guide for porting from SimPy 2 to SimPy 3</a>. If you want to stick to SimPy 2 for a while, change your requirements to <tt class="docutils literal"><span class="pre">'SimPy&gt;=2.3,&lt;3'</span></tt>.</p> <p>All in all, SimPy has become a framework for asynchronous programming based on coroutines. It brings more than ten years of experience and scientific know-how in the field of event-discrete simulation to the world of asynchronous programming and should thus be a solid foundation for everything based on an event&nbsp;loop.</p> <p>SimPy 3 is dedicated to <em>Klaus G. Müller</em> and <em>Tony Vignaux</em> who originally created SimPy in 2002. Without their hard work and efforts to create and document SimPy and to build a friendly community around it, we wouldn’t be where we are today. Thank&nbsp;you!</p> </blockquote> </div> SimPy 3 released2013-10-11T22:35:00+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2013-10-11:2013/10/11/simpy-3-released/<p>After one and a half year of development and many iterations, prototypes, and endless discussions, we now proudly release SimPy&nbsp;3.</p> <p>SimPy 3 has been completely rewritten from scratch. Our main goals were to simplify the <span class="caps">API</span> and code base as well as making SimPy more flexible and extensible. Some of the most important changes&nbsp;are:</p> <ul class="simple"> <li>Stronger focus on events. Processes yield event instances and are suspended until the event is triggered. An example for an event is a <em>timeout</em> (formerly known as <em>hold</em>), but even processes are now events, too (you can wait until a process&nbsp;terminates).</li> <li>Events can be combined with <tt class="docutils literal">&amp;</tt> (and) and <tt class="docutils literal">|</tt> (or) to create <em>condition events</em>.</li> <li>Process can now be defined by any generator function. You don&#8217;t have to subclass <tt class="docutils literal">Process</tt> anymore.</li> <li>No more global simulation state. Every simulation stores its state in an <em>environment</em> which is comparable to the old <tt class="docutils literal">Simulation</tt> class.</li> <li>Improved resource system with newly added resource&nbsp;types.</li> <li>Removed plotting and <span class="caps">GUI</span> capabilities. <a class="reference external" href="http://qt-project.org/wiki/PySide">Pyside</a> and <a class="reference external" href="http://matplotlib.org/">matplotlib</a> are much better with&nbsp;this.</li> <li>Greatly improved test suite. Its cleaner, and the tests are shorter and more&nbsp;numerous.</li> <li>Completely overhauled&nbsp;documentation.</li> </ul> <p>There is a <a class="reference external" href="https://simpy.readthedocs.org/en/latest/about/history.html">guide for porting from SimPy 2 to SimPy 3</a>. If you want to stick to SimPy 2 for a while, change your requirements to <tt class="docutils literal"><span class="pre">'SimPy&gt;=2.3,&lt;3'</span></tt>.</p> <p>All in all, SimPy has become a framework for asynchronous programming based on coroutines. It brings more than ten years of experience and scientific know-how in the field of event-discrete simulation to the world of asynchronous programming and should thus be a solid foundation for everything based on an event&nbsp;loop.</p> <p>SimPy 3 is dedicated to <em>Klaus G. Müller</em> and <em>Tony Vignaux</em> who originally created SimPy in 2002. Without their hard work and efforts to create and document SimPy and to build a friendly community around it, we wouldn’t be where we are today. Thank&nbsp;you!</p> Tea Timer 1.8.12013-10-03T13:15:00+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2013-10-03:2013/10/03/tea-timer-181/<p>This is just a <a class="reference external" href="/downloads/Tea_Timer-1.8.1.zip">maintenance release</a> that fixes the following&nbsp;issues:</p> <ul class="simple"> <li>[<span class="caps">FIX</span>] Issue <a class="reference external" href="https://bitbucket.org/ssc/tea-timer/issue/33/">#33</a>: Trim whitespace around version&nbsp;number.</li> <li>[<span class="caps">FIX</span>]&nbsp;Issue <a class="reference external" href="https://bitbucket.org/ssc/tea-timer/issue/34/">#34</a>: Titles containing a «&#8217;» are now&nbsp;read.</li> </ul> <p>There are still some <a class="reference external" href="https://bitbucket.org/ssc/tea-timer/issues?status=new&amp;status=open">open issues</a> left. Unfortunately, I currently don’t have the time to fix them all, so any help is very&nbsp;welcome.</p> Switched from Django to Pelican2013-10-01T22:34:00+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2013-10-01:2013/10/01/switched-from-django-to-pelican/<p><a class="reference external" href="https://www.djangoproject.com/">Django</a> powered this website since 2009. I had much fun with it and learned a lot while using it. However, maintaining my Django setup became too time consuming for me, so I was starting to think about lightweight alternatives a few months&nbsp;ago.</p> <p>I’ve already been using <a class="reference external" href="http://docs.getpelican.com">Pelican</a> for two projects at work and it offers everything I&nbsp;need:</p> <ul class="simple"> <li>Generate static <span class="caps">HTML</span>&nbsp;files</li> <li>Write articles and pages using reStructured Text, including syntax&nbsp;highlighting</li> <li>Easy to&nbsp;deploy</li> <li>Enough&nbsp;flexibility</li> <li>Written in&nbsp;Python</li> </ul> <p>So I migrated everything from Django to Pelican and stripped a lot of clutter from my website. One of the bigger “features” that I removed are comments, because they are hard to to with static pages. I could have used <a class="reference external" href="http://disqus.com/">Disqus</a> but I was just too lazy. :-) You can leave comments on <a class="reference external" href="http://stefan.sofa-rockers.org/+">Google+</a> and <a class="reference external" href="https://twitter.com/sscherfke">Twitter</a>,&nbsp;though.</p> Handling sub-process hierarchies in Python on Linux, OS X and Windows2013-08-15T12:52:34+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2013-08-15:2013/08/15/handling-sub-process-hierarchies-python-linux-os-x/<blockquote id="top"> <span class="upperitalics"><span class="caps">TL</span>;<span class="caps">DR</span></span>—Windows doesn’t support Posix signals. I’ll show you how you can work around this to cleanly terminate process hierarchies. Take a look at the final <a class="reference internal" href="#example">example</a>.</blockquote> <p>In this article, I’m trying solve an—at a first glance—simple&nbsp;problem:</p> <p>You have a Python process that starts one ore more sub-processes, which again might start their own sub-processes. If you press <em>Ctrl-C</em> on the command line, your main process and all of its sub-process should stop, maybe performing some clean-up before they terminate. Also, one process may decide to stop one of its children (including all of its&nbsp;sub-processes).</p> <blockquote> <p><em>Example:</em> You have a server process that you start via the command line and that starts several sub-processes. You usually stop that server and all sub-processes by hitting <em>Ctrl-C</em>.</p> <p>You also want to start your server from within a test case to test its replies on certain requests. If the test is done, you have to send some kind of a message or signal to the server to stop it and all of its sub-processes. The message should obviously be nothing that normal users can send to the&nbsp;server.</p> </blockquote> <p>This should not only work on Linux and <span class="caps">OS</span> X (which would be trivial), but also on&nbsp;Windows.</p> <p>Before I’ll show you some code, I’ll briefly describe what happens when you press <em>Ctrl-C</em> in a shell and which types of inter-process communication exist to tell another process to stop. After that, I&#8217;ll introduce a simple example which I’m then going to extend until the requirements stated above are&nbsp;met.</p> <p>I’m using Python&nbsp;3.3 for all examples. The code is tested on Ubuntu&nbsp;13.04, <span class="caps">OS</span>&nbsp;X&nbsp;10.8 and&nbsp;Windows&nbsp;7.</p> <div class="section" id="signals-and-inter-process-communication"> <span id="signals"></span><h2>Signals and inter-process&nbsp;communication</h2> <p>Signals are a simple form of inter-process communication (<span class="upper"><span class="caps">IPC</span></span>) on Posix-compliant operating systems (like Linux or <span class="caps">OS</span> X). If one processes sends a signal to another one, the <span class="caps">OS</span> interrupts its execution to deliver the signal. The receiving process may now handle the&nbsp;signal.</p> <p>Some interesting signals for this article are (quoted from <a class="reference external" href="http://en.wikipedia.org/wiki/Unix_signal#POSIX_signals">Wikipedia</a>):</p> <dl class="docutils"> <dt><span class="upper"><span class="caps">SIGINT</span></span></dt> <dd>“The <span class="upper"><span class="caps">SIGINT</span></span> signal is sent to a process by its controlling terminal when a user wishes to interrupt the process. This is typically initiated by pressing <em>Ctrl-C</em>, but on some systems, the ‘delete’ character or ‘break’ key can be&nbsp;used.”</dd> <dt><span class="upper"><span class="caps">SIGTERM</span></span></dt> <dd>“The <span class="upper"><span class="caps">SIGTERM</span></span> signal is sent to a process to request its termination. Unlike the <span class="upper"><span class="caps">SIGKILL</span></span> signal, it can be caught and interpreted or ignored by the process. This allows the process to perform nice termination releasing resources and saving state if appropriate. It should be noted that <span class="upper"><span class="caps">SIGINT</span></span> is nearly identical to <span class="upper"><span class="caps">SIGTERM</span></span>.”</dd> <dt><span class="upper"><span class="caps">SIGKILL</span></span></dt> <dd>“The <span class="upper"><span class="caps">SIGKILL</span></span> signal is sent to a process to cause it to terminate immediately. In contrast to <span class="upper"><span class="caps">SIGTERM</span></span> and <span class="upper"><span class="caps">SIGINT</span></span>, this signal cannot be caught or ignored, and the receiving process cannot perform any clean-up upon receiving this&nbsp;signal.”</dd> </dl> <p>Though <span class="upper"><span class="caps">SIGINT</span></span> and <span class="upper"><span class="caps">SIGTERM</span></span> are quite similar, the difference between them is from my point of view that <span class="upper"><span class="caps">SIGINT</span></span> is something directly initiated by the user (you press <em>Ctrl-C</em> in your terminal) while <span class="upper"><span class="caps">SIGTERM</span></span> is rather used programmatically (e.g., processes receive a <span class="upper"><span class="caps">SIGTERM</span></span> when the <span class="upper"><span class="caps">OS</span></span> is shutting&nbsp;down).</p> <p>You can also group processes into <a class="reference external" href="http://en.wikipedia.org/wiki/Process_group">process groups</a> and send a signal to all processes in that group at once. That’s for example what your terminal does when you press <em>Ctrl-C</em>.</p> <p>If a Python program receives a <span class="upper"><span class="caps">SIGINT</span></span>, the default handler raises a <tt class="docutils literal">KeyboardInterrupt</tt> by default. You can catch that exception and handle the interrupt in anyway you want (e.g., terminate immediately, do some clean-up before or just ignore it). Martin Cracauer <a class="reference external" href="http://www.cons.org/cracauer/sigint.html">discusses</a> how a <span class="upper"><span class="caps">SIGINT</span></span> should be handled&nbsp;properly.</p> <p>To handle a <span class="upper"><span class="caps">SIGTERM</span></span> or suppress the <tt class="docutils literal">KeyboardInterrupt</tt> exception in Python, you can register your own handler via <tt class="docutils literal">signal.signal()</tt>.</p> <p>Since Windows is not Posix-compliant (what a shame), most of the signals won’t work there, but we’ll come to this later. Tim Golden also <a class="reference external" href="http://ramblings.timgolden.me.uk/2013/05/31/how-does-python-handle-signals-on-windows/">wrote</a> about this. Unix process groups also don’t work on&nbsp;Windows.</p> <p>Other, more complicated (and more powerful) ways for <span class="upper"><span class="caps">IPC</span></span> are, for example, files, pipes and sockets. Using these, you can just send a message like “plzdiekthxbye” to another process to tell it to&nbsp;stop.</p> </div> <div class="section" id="stopping-processes-via-ctrl-c"> <h2>Stopping processes via <em>Ctrl-C</em></h2> <p>For all examples, we’ll use a simple process hierarchy: Process <em>A</em> starts process <em>B</em> and process <em>B</em> starts process <em>C</em>. Every process will sleep for ten seconds and waits for a <tt class="docutils literal">KeyboardInterrupt</tt>:</p> <div class="highlight"><pre><span class="kn">import</span> <span class="nn">subprocess</span> <span class="kn">import</span> <span class="nn">sys</span> <span class="kn">import</span> <span class="nn">time</span> <span class="n">PYTHON</span> <span class="o">-</span> <span class="n">sys</span><span class="o">.</span><span class="n">executable</span> <span class="n">SCRIPT</span> <span class="o">-</span> <span class="n">__file__</span> <span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="n">name</span><span class="p">):</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;</span><span class="si">%s</span><span class="s"> started&#39;</span> <span class="o">%</span> <span class="n">name</span><span class="p">)</span> <span class="c"># A and B spawn a subprocess</span> <span class="k">if</span> <span class="n">name</span> <span class="o">--</span> <span class="s">&#39;A&#39;</span><span class="p">:</span> <span class="n">child</span> <span class="o">-</span> <span class="n">subproc</span><span class="p">(</span><span class="s">&#39;B&#39;</span><span class="p">)</span> <span class="k">elif</span> <span class="n">name</span> <span class="o">--</span> <span class="s">&#39;B&#39;</span><span class="p">:</span> <span class="n">child</span> <span class="o">-</span> <span class="n">subproc</span><span class="p">(</span><span class="s">&#39;C&#39;</span><span class="p">)</span> <span class="k">else</span><span class="p">:</span> <span class="n">child</span> <span class="o">-</span> <span class="bp">None</span> <span class="c"># Sleep and wait for a Ctrl-C</span> <span class="k">try</span><span class="p">:</span> <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;</span><span class="si">%s</span><span class="s"> done&#39;</span> <span class="o">%</span> <span class="n">name</span><span class="p">)</span> <span class="k">except</span> <span class="ne">KeyboardInterrupt</span><span class="p">:</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;</span><span class="si">%s</span><span class="s"> got KeyboardInterrupt&#39;</span> <span class="o">%</span> <span class="n">name</span><span class="p">)</span> <span class="k">finally</span><span class="p">:</span> <span class="k">if</span> <span class="n">child</span><span class="p">:</span> <span class="n">child</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span> <span class="k">def</span> <span class="nf">subproc</span><span class="p">(</span><span class="n">name</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;Create and return a new subprocess named *name*.&quot;&quot;&quot;</span> <span class="n">proc</span> <span class="o">-</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">Popen</span><span class="p">([</span><span class="n">PYTHON</span><span class="p">,</span> <span class="n">SCRIPT</span><span class="p">,</span> <span class="n">name</span><span class="p">])</span> <span class="k">return</span> <span class="n">proc</span> <span class="k">if</span> <span class="n">__name__</span> <span class="o">--</span> <span class="s">&#39;__main__&#39;</span><span class="p">:</span> <span class="n">name</span> <span class="o">-</span> <span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">)</span> <span class="o">&gt;</span> <span class="mi">1</span> <span class="k">else</span> <span class="s">&#39;A&#39;</span> <span class="n">main</span><span class="p">(</span><span class="n">name</span><span class="p">)</span> </pre></div> <p>Running this script and pressing <em>Ctrl-C</em> after about one or two seconds gives us the following&nbsp;output:</p> <div class="highlight"><pre><span class="nv">$ </span>python subprocs_1.py A started B started C started ^CA got KeyboardInterrupt B got KeyboardInterrupt C got KeyboardInterrupt </pre></div> <p>So far so good. It will get more complicated&nbsp;soon.</p> </div> <div class="section" id="stopping-a-sub-process-and-all-of-its-children"> <h2>Stopping a sub-process and all of its&nbsp;children</h2> <p>To request the termination of a subprocess, we’ll send a <span class="upper"><span class="caps">SIGTERM</span></span> to it (on Linux and <span class="caps">OS</span> X, this signal is sent if you call a Popen object’s <a class="reference external" href="http://docs.python.org/3.3/library/subprocess#subprocess.Popen.terminate">terminate()</a> method). For the process being terminated, this means the same as a <span class="upper"><span class="caps">SIGINT</span></span>—it is allowed to do some clean-up or even to ignore the&nbsp;signal.</p> <p>To catch the signal, you have to register a custom handler, though. To make things easier, you should also register the same handler for the <span class="upper"><span class="caps">SIGINT</span></span> signal, so that you don’t end up having the same code in the handler function and in an <tt class="docutils literal">except KeyboardInterrupt</tt> block:</p> <div class="highlight"><pre><span class="kn">import</span> <span class="nn">functools</span> <span class="kn">import</span> <span class="nn">signal</span> <span class="kn">import</span> <span class="nn">subprocess</span> <span class="kn">import</span> <span class="nn">sys</span> <span class="kn">import</span> <span class="nn">time</span> <span class="kn">import</span> <span class="nn">traceback</span> <span class="n">PYTHON</span> <span class="o">-</span> <span class="n">sys</span><span class="o">.</span><span class="n">executable</span> <span class="n">SCRIPT</span> <span class="o">-</span> <span class="n">__file__</span> <span class="n">SIGNALS</span> <span class="o">-</span> <span class="p">{</span> <span class="n">signal</span><span class="o">.</span><span class="n">SIGINT</span><span class="p">:</span> <span class="s">&#39;SIGINT&#39;</span><span class="p">,</span> <span class="n">signal</span><span class="o">.</span><span class="n">SIGTERM</span><span class="p">:</span> <span class="s">&#39;SIGTERM&#39;</span><span class="p">,</span> <span class="p">}</span> <span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">terminate</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;If *terminate* is ``True`` (should only be the case if *name* is ``A``),</span> <span class="sd"> A will try to terminate B.</span> <span class="sd"> B and C will always just sleep and wait for things to happen ...</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;</span><span class="si">%s</span><span class="s"> started&#39;</span> <span class="o">%</span> <span class="n">name</span><span class="p">)</span> <span class="c"># A and B spawn a subprocess</span> <span class="k">if</span> <span class="n">name</span> <span class="o">--</span> <span class="s">&#39;A&#39;</span><span class="p">:</span> <span class="n">child</span> <span class="o">-</span> <span class="n">subproc</span><span class="p">(</span><span class="s">&#39;B&#39;</span><span class="p">)</span> <span class="k">elif</span> <span class="n">name</span> <span class="o">--</span> <span class="s">&#39;B&#39;</span><span class="p">:</span> <span class="n">child</span> <span class="o">-</span> <span class="n">subproc</span><span class="p">(</span><span class="s">&#39;C&#39;</span><span class="p">)</span> <span class="k">else</span><span class="p">:</span> <span class="n">child</span> <span class="o">-</span> <span class="bp">None</span> <span class="c"># Curry our cleanup func and register it as handler for SIGINT and SIGTERM</span> <span class="n">handler</span> <span class="o">-</span> <span class="n">functools</span><span class="o">.</span><span class="n">partial</span><span class="p">(</span><span class="n">cleanup</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">child</span><span class="p">)</span> <span class="n">signal</span><span class="o">.</span><span class="n">signal</span><span class="p">(</span><span class="n">signal</span><span class="o">.</span><span class="n">SIGINT</span><span class="p">,</span> <span class="n">handler</span><span class="p">)</span> <span class="n">signal</span><span class="o">.</span><span class="n">signal</span><span class="p">(</span><span class="n">signal</span><span class="o">.</span><span class="n">SIGTERM</span><span class="p">,</span> <span class="n">handler</span><span class="p">)</span> <span class="k">if</span> <span class="n">terminate</span><span class="p">:</span> <span class="c"># A tries to terminate B</span> <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="n">term</span><span class="p">(</span><span class="n">child</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;</span><span class="si">%s</span><span class="s"> ended&#39;</span> <span class="o">%</span> <span class="n">name</span><span class="p">)</span> <span class="k">else</span><span class="p">:</span> <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;</span><span class="si">%s</span><span class="s"> done&#39;</span> <span class="o">%</span> <span class="n">name</span><span class="p">)</span> <span class="k">if</span> <span class="n">child</span><span class="p">:</span> <span class="n">child</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span> <span class="k">def</span> <span class="nf">subproc</span><span class="p">(</span><span class="n">name</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;Create and return a new subprocess named *name*.&quot;&quot;&quot;</span> <span class="n">proc</span> <span class="o">-</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">Popen</span><span class="p">([</span><span class="n">PYTHON</span><span class="p">,</span> <span class="n">SCRIPT</span><span class="p">,</span> <span class="n">name</span><span class="p">])</span> <span class="k">return</span> <span class="n">proc</span> <span class="k">def</span> <span class="nf">term</span><span class="p">(</span><span class="n">proc</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;Send a SIGTERM to *proc* and wait for it to terminate.&quot;&quot;&quot;</span> <span class="n">proc</span><span class="o">.</span><span class="n">terminate</span><span class="p">()</span> <span class="c"># Sends SIGTERM</span> <span class="n">proc</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span> <span class="k">def</span> <span class="nf">cleanup</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">child</span><span class="p">,</span> <span class="n">signum</span><span class="p">,</span> <span class="n">frame</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;Stop the sub-process *child* if *signum* is SIGTERM. Then terminate.&quot;&quot;&quot;</span> <span class="k">try</span><span class="p">:</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;</span><span class="si">%s</span><span class="s"> got a </span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">SIGNALS</span><span class="p">[</span><span class="n">signum</span><span class="p">]))</span> <span class="k">if</span> <span class="n">child</span> <span class="ow">and</span> <span class="n">signum</span> <span class="err">!</span><span class="o">-</span> <span class="n">signal</span><span class="o">.</span><span class="n">SIGINT</span><span class="p">:</span> <span class="n">term</span><span class="p">(</span><span class="n">child</span><span class="p">)</span> <span class="k">except</span><span class="p">:</span> <span class="n">traceback</span><span class="o">.</span><span class="n">print_exc</span><span class="p">()</span> <span class="k">finally</span><span class="p">:</span> <span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">()</span> <span class="k">if</span> <span class="n">__name__</span> <span class="o">--</span> <span class="s">&#39;__main__&#39;</span><span class="p">:</span> <span class="n">terminate</span> <span class="o">-</span> <span class="bp">False</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">)</span> <span class="o">--</span> <span class="mi">1</span><span class="p">:</span> <span class="n">name</span> <span class="o">-</span> <span class="s">&#39;A&#39;</span> <span class="k">elif</span> <span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">--</span> <span class="s">&#39;term&#39;</span><span class="p">:</span> <span class="n">terminate</span> <span class="o">-</span> <span class="bp">True</span> <span class="n">name</span> <span class="o">-</span> <span class="s">&#39;A&#39;</span> <span class="k">else</span><span class="p">:</span> <span class="n">name</span> <span class="o">-</span> <span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="c"># B or C</span> <span class="n">main</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">terminate</span><span class="p">)</span> </pre></div> <p>Our processes now handle <span class="upper"><span class="caps">SIGINT</span></span> and <span class="upper"><span class="caps">SIGTERM</span></span> signals via the <tt class="docutils literal">cleanup()</tt> handler. The curried version of this handler will have the name of the process and its child process set by default, leaving its signature as <tt class="docutils literal">handler(signum, frame)</tt> which is exactly what <tt class="docutils literal">signal.signal()</tt> expects. Note, that we don’t need to terminate our sub-processes when we get a <span class="upper"><span class="caps">SIGINT</span></span>, since the sub-processes will also receive&nbsp;one.</p> <p>We can now also pass a <em>terminate</em> argument to our script. <em>A</em> will then try to terminate <em>B</em> after a second. <em>B</em> will then stop <em>C</em>.</p> <p>If you run this script without arguments, you will get (nearly) the same output as in the last example. If you pass <tt class="docutils literal">term</tt>, you’ll get the following output from a Linux or <span class="caps">OS</span>&nbsp;X&nbsp;shell:</p> <div class="highlight"><pre><span class="nv">$ </span>python subprocs_2.py term A started B started C started B got a SIGTERM C got a SIGTERM A ended </pre></div> <p>If you run the same thing on Windows, you’ll&nbsp;get:</p> <div class="highlight"><pre><span class="p">&gt;</span><span class="n">python</span> subprocs_<span class="m">2</span>.py term A started B started C started A ended <span class="p">&gt;</span><span class="n">C</span> done </pre></div> <p>Our code now magically stopped working on Windows :-). <em>B</em> seems to receive something from <em>A</em>, but immediately terminates (no output is printed). It also does not forward the signal to <em>C</em>, so <em>C</em> just waits ten seconds and prints its message before exiting on its&nbsp;own.</p> <p>Clearly not the desired behavior, so this is where the fun begins&nbsp;…</p> </div> <div class="section" id="fixing-it-on-windows"> <h2>Fixing it on&nbsp;Windows</h2> <p>If you search for this problem on the Interwebs, you’ll find a lot of different answers—some more helpful, some less. This <a class="reference external" href="http://www.reddit.com/r/Python/comments/1dsblt/windows_command_line_automation_ctrlc_question/">Reddit post</a> helped me the most. According to the <span class="upper"><span class="caps">MSDN</span></span>, you can use <cite>GenerateConsoleCtrlEvent</cite> to send two types of signals to a process: <span class="upper">CTRL_C_EVENT</span> and <span class="upper">CTRL_BREAK_EVENT</span>. The former translates to a <span class="upper"><span class="caps">SIGINT</span></span>, the latter to a <span class="upper"><span class="caps">SIGBREAK</span></span>. Instead of <tt class="docutils literal">GenerateConsoleCtrlEvent</tt>, which is only available via the win32 <span class="upper"><span class="caps">API</span></span>, you can fortunately also use <tt class="docutils literal">os.kill</tt> or <tt class="docutils literal">Popen.send_signal()</tt> to send the&nbsp;signal.</p> <p>A <span class="upper">CTRL_C_EVENT</span> can <em>not</em> be directed to a single process and is always received by all processes that share the current console. <span class="upper">CTRL_BREAK_EVENT</span> on the other hand can be send to specific processes. However, in order to use it, we have to pass <tt class="docutils literal"><span class="pre">creationgflags-subprocess.CREATE_NEW_PROCESS_GROUP</span></tt> to the <a class="reference external" href="http://docs.python.org/3.3/library/subprocess#popen-constructor">Popen</a> constructor. This parameter is only available on Windows and has nothing to do with Unix process groups. When you set this flag, you can <a class="reference external" href="http://msdn.microsoft.com/en-us/library/windows/desktop/ms684863%28v-vs.85%29.aspx">no longer</a> send <span class="upper">CTRL_C_EVENT</span> and if you press <em>Ctrl-C</em> in your console, only the root process will receive a <span class="upper"><span class="caps">SIGINT</span></span> signal. Also note, that sending any other signal than <span class="upper">CTRL_C_EVENT</span> and <span class="upper">CTRL_BREAK_EVENT</span> will unconditionally <a class="reference external" href="http://docs.python.org/3/library/os#os.kill">kill the process</a>.</p> <p>We’ll end up with the following&nbsp;preconditions:</p> <p>If on Linux or <span class="caps">OS</span>&nbsp;X:</p> <ul class="simple"> <li>Your terminal sends a <span class="upper"><span class="caps">SIGINT</span></span> to all processes if you press <em>Ctrl-C</em>. There’s no need to forward it to&nbsp;sub-processes.</li> <li>If a process receives a <span class="upper"><span class="caps">SIGTERM</span></span>, it should forward it to its&nbsp;sub-processes.</li> </ul> <p>If on&nbsp;Windows:</p> <ul class="simple"> <li>You should start sub-processes with <tt class="docutils literal"><span class="pre">creationflags-CREATE_NEW_PROCESS_GROUP</span></tt></li> <li>This enables you to send a <span class="upper">CTRL_BREAK_EVENT</span> to a specific process. The process receives a <span class="upper"><span class="caps">SIGBREAK</span></span> and should forward it to its&nbsp;sub-processes.</li> <li>If you press <em>Ctrl-C</em> in your console, only the root process receives a <span class="upper"><span class="caps">SIGINT</span></span> and should send a <span class="upper">CTRL_BREAK_EVENT</span> to its&nbsp;sub-processes.</li> </ul> <p id="example">Incorporating what we know now, our example looks like&nbsp;this:</p> <div class="highlight"><pre><span class="kn">import</span> <span class="nn">functools</span> <span class="kn">import</span> <span class="nn">signal</span> <span class="kn">import</span> <span class="nn">subprocess</span> <span class="kn">import</span> <span class="nn">sys</span> <span class="kn">import</span> <span class="nn">time</span> <span class="kn">import</span> <span class="nn">traceback</span> <span class="n">PYTHON</span> <span class="o">-</span> <span class="n">sys</span><span class="o">.</span><span class="n">executable</span> <span class="n">SCRIPT</span> <span class="o">-</span> <span class="n">__file__</span> <span class="n">ON_WINDOWS</span> <span class="o">-</span> <span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">platform</span> <span class="o">--</span> <span class="s">&#39;win32&#39;</span><span class="p">)</span> <span class="n">SIGNALS</span> <span class="o">-</span> <span class="p">{</span> <span class="n">signal</span><span class="o">.</span><span class="n">SIGINT</span><span class="p">:</span> <span class="s">&#39;SIGINT&#39;</span><span class="p">,</span> <span class="n">signal</span><span class="o">.</span><span class="n">SIGTERM</span><span class="p">:</span> <span class="s">&#39;SIGTERM&#39;</span><span class="p">,</span> <span class="p">}</span> <span class="k">if</span> <span class="n">ON_WINDOWS</span><span class="p">:</span> <span class="n">SIGNALS</span><span class="p">[</span><span class="n">signal</span><span class="o">.</span><span class="n">SIGBREAK</span><span class="p">]</span> <span class="o">-</span> <span class="s">&#39;SIGBREAK&#39;</span> <span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">terminate</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;If *terminate* is ``True`` (should only be the case if *name* is ``A``),</span> <span class="sd"> A will try to terminate B.</span> <span class="sd"> B and C will always just sleep and wait for things to happen ...</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;</span><span class="si">%s</span><span class="s"> started&#39;</span> <span class="o">%</span> <span class="n">name</span><span class="p">)</span> <span class="c"># A and B spawn a subprocess</span> <span class="k">if</span> <span class="n">name</span> <span class="o">--</span> <span class="s">&#39;A&#39;</span><span class="p">:</span> <span class="n">child</span> <span class="o">-</span> <span class="n">subproc</span><span class="p">(</span><span class="s">&#39;B&#39;</span><span class="p">)</span> <span class="k">elif</span> <span class="n">name</span> <span class="o">--</span> <span class="s">&#39;B&#39;</span><span class="p">:</span> <span class="n">child</span> <span class="o">-</span> <span class="n">subproc</span><span class="p">(</span><span class="s">&#39;C&#39;</span><span class="p">)</span> <span class="k">else</span><span class="p">:</span> <span class="n">child</span> <span class="o">-</span> <span class="bp">None</span> <span class="c"># Curry our cleanup func and register it as handler for SIGINT and SIGTERM</span> <span class="n">handler</span> <span class="o">-</span> <span class="n">functools</span><span class="o">.</span><span class="n">partial</span><span class="p">(</span><span class="n">cleanup</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">child</span><span class="p">)</span> <span class="n">signal</span><span class="o">.</span><span class="n">signal</span><span class="p">(</span><span class="n">signal</span><span class="o">.</span><span class="n">SIGINT</span><span class="p">,</span> <span class="n">handler</span><span class="p">)</span> <span class="k">if</span> <span class="n">ON_WINDOWS</span><span class="p">:</span> <span class="n">signal</span><span class="o">.</span><span class="n">signal</span><span class="p">(</span><span class="n">signal</span><span class="o">.</span><span class="n">SIGBREAK</span><span class="p">,</span> <span class="n">handler</span><span class="p">)</span> <span class="k">else</span><span class="p">:</span> <span class="n">signal</span><span class="o">.</span><span class="n">signal</span><span class="p">(</span><span class="n">signal</span><span class="o">.</span><span class="n">SIGTERM</span><span class="p">,</span> <span class="n">handler</span><span class="p">)</span> <span class="k">if</span> <span class="n">terminate</span><span class="p">:</span> <span class="c"># A tries to terminate B</span> <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="n">term</span><span class="p">(</span><span class="n">child</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;</span><span class="si">%s</span><span class="s"> ended&#39;</span> <span class="o">%</span> <span class="n">name</span><span class="p">)</span> <span class="k">else</span><span class="p">:</span> <span class="c"># SIGBREAK cannot interrupt sleep(), so we sleep 10 * 1s instead</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">):</span> <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;</span><span class="si">%s</span><span class="s"> done&#39;</span> <span class="o">%</span> <span class="n">name</span><span class="p">)</span> <span class="k">if</span> <span class="n">child</span><span class="p">:</span> <span class="n">child</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span> <span class="k">def</span> <span class="nf">subproc</span><span class="p">(</span><span class="n">name</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;Create and return a new subprocess named *name*.&quot;&quot;&quot;</span> <span class="n">kwargs</span> <span class="o">-</span> <span class="p">{}</span> <span class="k">if</span> <span class="n">ON_WINDOWS</span><span class="p">:</span> <span class="n">kwargs</span><span class="p">[</span><span class="s">&#39;creationflags&#39;</span><span class="p">]</span> <span class="o">-</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">CREATE_NEW_PROCESS_GROUP</span> <span class="n">proc</span> <span class="o">-</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">Popen</span><span class="p">([</span><span class="n">PYTHON</span><span class="p">,</span> <span class="n">SCRIPT</span><span class="p">,</span> <span class="n">name</span><span class="p">],</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> <span class="k">return</span> <span class="n">proc</span> <span class="k">def</span> <span class="nf">term</span><span class="p">(</span><span class="n">proc</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;Send a SIGTERM/SIGBREAK to *proc* and wait for it to terminate.&quot;&quot;&quot;</span> <span class="k">if</span> <span class="n">ON_WINDOWS</span><span class="p">:</span> <span class="n">proc</span><span class="o">.</span><span class="n">send_signal</span><span class="p">(</span><span class="n">signal</span><span class="o">.</span><span class="n">CTRL_BREAK_EVENT</span><span class="p">)</span> <span class="k">else</span><span class="p">:</span> <span class="n">proc</span><span class="o">.</span><span class="n">terminate</span><span class="p">()</span> <span class="n">proc</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span> <span class="k">def</span> <span class="nf">cleanup</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">child</span><span class="p">,</span> <span class="n">signum</span><span class="p">,</span> <span class="n">frame</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;Stop the sub-process *child* if *signum* is SIGTERM. Then terminate.&quot;&quot;&quot;</span> <span class="k">try</span><span class="p">:</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;</span><span class="si">%s</span><span class="s"> got a </span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">SIGNALS</span><span class="p">[</span><span class="n">signum</span><span class="p">]))</span> <span class="k">if</span> <span class="n">child</span> <span class="ow">and</span> <span class="p">(</span><span class="n">ON_WINDOWS</span> <span class="ow">or</span> <span class="n">signum</span> <span class="err">!</span><span class="o">-</span> <span class="n">signal</span><span class="o">.</span><span class="n">SIGINT</span><span class="p">):</span> <span class="c"># Forward SIGTERM on Linux or any signal on Windows</span> <span class="n">term</span><span class="p">(</span><span class="n">child</span><span class="p">)</span> <span class="k">except</span><span class="p">:</span> <span class="n">traceback</span><span class="o">.</span><span class="n">print_exc</span><span class="p">()</span> <span class="k">finally</span><span class="p">:</span> <span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">()</span> <span class="k">if</span> <span class="n">__name__</span> <span class="o">--</span> <span class="s">&#39;__main__&#39;</span><span class="p">:</span> <span class="n">terminate</span> <span class="o">-</span> <span class="bp">False</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">)</span> <span class="o">--</span> <span class="mi">1</span><span class="p">:</span> <span class="n">name</span> <span class="o">-</span> <span class="s">&#39;A&#39;</span> <span class="k">elif</span> <span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">--</span> <span class="s">&#39;term&#39;</span><span class="p">:</span> <span class="n">terminate</span> <span class="o">-</span> <span class="bp">True</span> <span class="n">name</span> <span class="o">-</span> <span class="s">&#39;A&#39;</span> <span class="k">else</span><span class="p">:</span> <span class="n">name</span> <span class="o">-</span> <span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="c"># B or C</span> <span class="n">main</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">terminate</span><span class="p">)</span> </pre></div> <p>We can register a <span class="upper"><span class="caps">SIGINT</span></span> handler in any case. If on Windows we also register a <span class="upper"><span class="caps">SIGBREAK</span></span> handler. For all remaining cases we register a handler for <span class="upper"><span class="caps">SIGTERM</span></span>.</p> <p>Since <span class="upper"><span class="caps">SIGBREAK</span></span> cannot interrupt a <tt class="docutils literal">time.sleep()</tt> call, I changed it from <tt class="docutils literal">time.sleep(10)</tt> to <tt class="docutils literal">for i in range(10): time.sleep(1)</tt> so that the sub-processes terminate faster. I don’t know if this is intended behavior or a bug in&nbsp;Python.</p> <p>When terminating a process, you have to send a <span class="upper">CTRL_BREAK_EVENT</span> on Windows. Else, you can just use <tt class="docutils literal">Popen.terminate()</tt> which will send a <span class="upper"><span class="caps">SIGTERM</span></span> to the&nbsp;process.</p> <p>In our clean-up function, we only forward the signal if its not a <span class="upper"><span class="caps">SIGINT</span></span> or if we are on Windows. And if there is child process, of&nbsp;course.</p> <p>The output on Linux stays the same, but on Windows, we now&nbsp;get:</p> <div class="highlight"><pre><span class="p">&gt;</span><span class="n">python</span> subprocs_<span class="m">3</span>.py term A started B started C started B got a SIGBREAK C got a SIGBREAK A ended </pre></div> <p>It finally works on Linux, <span class="caps">OS</span> X <em>and</em> Windows!&nbsp;\o/</p> </div> <div class="section" id="conclusion"> <h2>Conclusion</h2> <p>If you don’t have to (or don’t want to) support Windows, handling sub-processes and sending signals to them is not that hard. But I hope I could demonstrate that it’s—despite of all problems—possible to get the job done on Windows, too, without too much&nbsp;overhead.</p> <p>An alternative to signals, if you are using some other kind of <span class="upper"><span class="caps">IPC</span></span> anyways, might be using it to send a termination requests to your processes instead of sending a signal. There would be no need for the <span class="upper">CREATE_NEW_PROCESS_GROUP</span> flag then and <em>Ctrl-C</em> would work nicely on all platforms. You may take care though, that clients are not able to send a termination message to your server, but that depends on your use&nbsp;case.</p> <p>In the end, I wish Windows was Posix-compliant. Would make life so much easier&nbsp;…</p> </div> django-sphinxdoc 1.22013-08-11T22:22:11+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2013-08-11:2013/08/11/django-sphinxdoc-12/<p>After more than a year, there is finally a new version of <a class="reference external" href="https://pypi.python.org/pypi/django-sphinxdoc">django-sphinxdoc</a>&nbsp;available.</p> <p>Django-sphinxdoc is a Django app that allows you to integrate documentation written with Sphinx into your site. You can also search the documentation using <a class="reference external" href="http://haystacksearch.org/">Haystack</a>.</p> <p>The new version adds internationalization support as well as Spanish and Basque localizations. It uses class-based views now and is compatible with Haystack &gt;= 2.1. Thanks to Mike Shantz, Josiah Klassen, Andres Riancho and Ales Zabala Alava for their&nbsp;contributions.</p> <p>Unfortunately, I currently don’t have very much time to keep improving this app. So any contribution that solves an <a class="reference external" href="https://bitbucket.org/ssc/django-sphinxdoc/issues?status=new&amp;status=open">issue</a> or adds a new feature is very&nbsp;welcome.</p> Bitbucket stopped rendering README.txt for Python projects [resolved]2013-01-11T09:34:29+01:00Stefan Scherfketag:stefan.sofa-rockers.org,2013-01-11:2013/01/11/bitbucket-stopped-rendering-readmetxt-python-proje/<p>Until recently, Bitbucket nicely rendered reStructuredText (reST) formatted <span class="caps">README</span>.txt files for Python projects. This made totally sense, because <a class="reference external" href="http://pypi.python.org/pypi">PyPI</a> requires the project’s description in reST and most people put it into <span class="caps">README</span>.txt&nbsp;files.</p> <p>However, for some reason the Bitbucket guys removed that feature and it seems that they don’t intend to bring it back. Instead, you are urged to rename your files from <em>*.txt</em> to <em>*.rst</em>.</p> <p>Maybe some <a class="reference external" href="https://bitbucket.org/site/master/issue/5617/readme-not-rendered-properly-on-the">feedback</a> would help to bring that feature back&nbsp;…</p> <p><strong>Update:</strong> Bitbucket fixed that issue.&nbsp;:-)</p> SimPy 3 Preview2012-12-12T15:15:22+01:00Stefan Scherfketag:stefan.sofa-rockers.org,2012-12-12:2012/12/12/simpy-3-preview/<p><a class="reference external" href="https://simpy.readthedocs.org/en/latest/">SimPy</a> is a process-based and event-driven simulation framework written in pure Python. It can also be used for multi-agent systems and other eventloop-based&nbsp;applications.</p> <p>After several months of work and various iterations of SimPy’s new <span class="caps">API</span>, we can finally present a preview with the most important features&nbsp;working.</p> <p>Here is a simple&nbsp;example:</p> <div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="kn">import</span> <span class="nn">simpy</span> <span class="go">&gt;&gt;&gt;</span> <span class="gp">&gt;&gt;&gt; </span><span class="k">def</span> <span class="nf">clock</span><span class="p">(</span><span class="n">env</span><span class="p">):</span> <span class="gp">... </span> <span class="k">while</span> <span class="bp">True</span><span class="p">:</span> <span class="gp">... </span> <span class="k">print</span><span class="p">(</span><span class="n">env</span><span class="o">.</span><span class="n">now</span><span class="p">)</span> <span class="gp">... </span> <span class="k">yield</span> <span class="n">env</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="gp">...</span> <span class="gp">&gt;&gt;&gt; </span><span class="n">env</span> <span class="o">=</span> <span class="n">simpy</span><span class="o">.</span><span class="n">Environment</span><span class="p">()</span> <span class="gp">&gt;&gt;&gt; </span><span class="n">env</span><span class="o">.</span><span class="n">start</span><span class="p">(</span><span class="n">clock</span><span class="p">(</span><span class="n">env</span><span class="p">))</span> <span class="go">Process(clock)</span> <span class="gp">&gt;&gt;&gt; </span><span class="n">simpy</span><span class="o">.</span><span class="n">simulate</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">until</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span> <span class="go">0</span> <span class="go">1</span> <span class="go">2</span> </pre></div> <p>You can find the full announcement at <a class="reference external" href="https://plus.google.com/u/0/101634625602509193865/posts/196xrYcKsuL">Google+</a> and on our <a class="reference external" href="http://sourceforge.net/mailarchive/message.php?msg_id=30218376">mailing list</a>.</p> <p>The code is at <a class="reference external" href="https://bitbucket.org/simpy/simpy">Bitbucket</a>, the documentation at <a class="reference external" href="https://simpy.readthedocs.org/en/latest/">Read the Docs</a>.</p> django-sphinxdoc 1.12012-04-19T17:20:25+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2012-04-19:2012/04/19/django-sphinxdoc-11/<p>Most Python projects use <a class="reference external" href="http://sphinx.pocoo.org/">Sphinx</a> for their documentation. And many (most?) Python powered websites use <a class="reference external" href="http://www.djangoproject.com/">Django</a> as&nbsp;framework.</p> <p>So there might be some people who use both Sphinx and Django. If you belong to this group and want to integrate the documentation of your projects into your Django powered website, <em>django-sphinxdoc</em> might be the app you’re searching&nbsp;for.</p> <p><em>Django-sphinxdoc</em> can build and import your Sphinxdocumentation and provides views for browsing and searching it. You can see <em>django-sphinxdoc</em> in action be reading <a class="reference external" href="/docs/django-sphinxdoc/">its documentation</a>.</p> <p>What’s new in this&nbsp;version?</p> <ul class="simple"> <li>[<span class="caps">NEW</span>] Support static and download&nbsp;files.</li> <li>[<span class="caps">NEW</span>] Additional context to search view so that project information is available in the&nbsp;template.</li> <li>[<span class="caps">CHANGE</span>] Updated some <a class="reference external" href="https://bitbucket.org/ssc/django-sphinxdoc/changeset/e876d5e72b34">templates</a></li> <li>[<span class="caps">FIX</span>] Fixed a bug with the updatedoc command and <tt class="docutils literal">~</tt> in&nbsp;paths.</li> <li>[<span class="caps">FIX</span>] Include all module index&nbsp;files.</li> <li>[<span class="caps">FIX</span>] Improved indexing&nbsp;behaviour</li> <li>[<span class="caps">FIX</span>] Improved behaviour when building the&nbsp;docs.</li> </ul> <p>You can find <em>django-sphinxdoc</em> in the <a class="reference external" href="http://pypi.python.org/pypi/django-sphinxdoc">Cheese Shop</a> or at <a class="reference external" href="http://bitbucket.org/ssc/django-sphinxdoc">Bitbucket</a>.</p> Check Python site-packages for Updates2012-04-14T11:50:50+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2012-04-14:2012/04/14/check-python-site-packages-updates/<p>A while ago, I found a nice little script called <a class="reference external" href="https://gist.github.com/910447">check_for_updates.py</a> which uses <span class="caps">PIP</span> to check your installed Python packages for updates. However, it didn’t work under Python 3, so I ported it&nbsp;myself:</p> <div class="highlight"><pre><span class="sd">&quot;&quot;&quot;</span> <span class="sd">Use pip to get a list of local packages to check against one or more package</span> <span class="sd">indexes for updated versions.</span> <span class="sd">&quot;&quot;&quot;</span> <span class="k">try</span><span class="p">:</span> <span class="kn">from</span> <span class="nn">cStringIO</span> <span class="kn">import</span> <span class="n">StringIO</span> <span class="kn">import</span> <span class="nn">xmlrpclib</span> <span class="k">except</span> <span class="ne">ImportError</span><span class="p">:</span> <span class="kn">from</span> <span class="nn">io</span> <span class="kn">import</span> <span class="n">StringIO</span> <span class="kn">import</span> <span class="nn">xmlrpc.client</span> <span class="kn">as</span> <span class="nn">xmlrpclib</span> <span class="kn">from</span> <span class="nn">distutils.version</span> <span class="kn">import</span> <span class="n">StrictVersion</span><span class="p">,</span> <span class="n">LooseVersion</span> <span class="kn">import</span> <span class="nn">sys</span> <span class="kn">import</span> <span class="nn">pip</span> <span class="k">def</span> <span class="nf">get_local_packages</span><span class="p">():</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Call pip&#39;s freeze -l</span> <span class="sd"> returns a list of package_name, version tuples</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="n">sys</span><span class="o">.</span><span class="n">stdout</span> <span class="o">=</span> <span class="n">mystdout</span> <span class="o">=</span> <span class="n">StringIO</span><span class="p">()</span> <span class="n">pip</span><span class="o">.</span><span class="n">main</span><span class="p">([</span><span class="s">&#39;freeze&#39;</span><span class="p">,</span> <span class="s">&#39;-l&#39;</span><span class="p">])</span> <span class="n">sys</span><span class="o">.</span><span class="n">stdout</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">__stdout__</span> <span class="n">pkgs</span> <span class="o">=</span> <span class="n">mystdout</span><span class="o">.</span><span class="n">getvalue</span><span class="p">()</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s">&#39;</span><span class="se">\n</span><span class="s">&#39;</span><span class="p">)</span> <span class="k">return</span> <span class="p">[</span><span class="n">p</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s">&#39;==&#39;</span><span class="p">)</span> <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">pkgs</span><span class="p">]</span> <span class="k">def</span> <span class="nf">find_current_version</span><span class="p">(</span><span class="n">package</span><span class="p">,</span> <span class="n">index_urls</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Using the XMLRPC method available for PyPI, get the most recent version</span> <span class="sd"> of &lt;package&gt; from each of the index_urls and figure out which one (if any)</span> <span class="sd"> is higher</span> <span class="sd"> Returns a tuple of the index with the higher version and the version it has</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="k">if</span> <span class="n">index_urls</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span> <span class="n">index_urls</span> <span class="o">=</span> <span class="p">[</span><span class="s">&#39;http://pypi.python.org/pypi&#39;</span><span class="p">]</span> <span class="n">cur_version</span> <span class="o">=</span> <span class="s">&#39;0&#39;</span> <span class="n">cur_index</span> <span class="o">=</span> <span class="s">&#39;&#39;</span> <span class="k">for</span> <span class="n">index_url</span> <span class="ow">in</span> <span class="n">index_urls</span><span class="p">:</span> <span class="n">pypi</span> <span class="o">=</span> <span class="n">xmlrpclib</span><span class="o">.</span><span class="n">ServerProxy</span><span class="p">(</span><span class="n">index_url</span><span class="p">,</span> <span class="n">xmlrpclib</span><span class="o">.</span><span class="n">Transport</span><span class="p">())</span> <span class="n">pypi_hits</span> <span class="o">=</span> <span class="n">pypi</span><span class="o">.</span><span class="n">package_releases</span><span class="p">(</span><span class="n">package</span><span class="p">)</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">pypi_hits</span><span class="p">)</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">:</span> <span class="k">if</span> <span class="n">compare_versions</span><span class="p">(</span><span class="n">pypi_hits</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">cur_version</span><span class="p">):</span> <span class="n">cur_version</span> <span class="o">=</span> <span class="n">pypi_hits</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="n">cur_index</span> <span class="o">=</span> <span class="n">index_url</span> <span class="k">return</span> <span class="n">cur_index</span><span class="p">,</span> <span class="n">cur_version</span> <span class="k">def</span> <span class="nf">compare_versions</span><span class="p">(</span><span class="n">version1</span><span class="p">,</span> <span class="n">version2</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Compare 2 versions, starting with StrictVersion, and falling back on</span> <span class="sd"> LooseVersion. Returns ``True`` if *version1* is greater than *version2*.</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="k">try</span><span class="p">:</span> <span class="k">return</span> <span class="n">StrictVersion</span><span class="p">(</span><span class="n">version1</span><span class="p">)</span> <span class="o">&gt;</span> <span class="n">StrictVersion</span><span class="p">(</span><span class="n">version2</span><span class="p">)</span> <span class="c"># in case of abnormal version number, fall back to LooseVersion</span> <span class="k">except</span> <span class="ne">ValueError</span><span class="p">:</span> <span class="k">return</span> <span class="n">LooseVersion</span><span class="p">(</span><span class="n">version1</span><span class="p">)</span> <span class="o">&gt;</span> <span class="n">LooseVersion</span><span class="p">(</span><span class="n">version2</span><span class="p">)</span> <span class="k">def</span> <span class="nf">output_line</span><span class="p">(</span><span class="n">pkg_name</span><span class="p">,</span> <span class="n">new_version</span><span class="p">,</span> <span class="n">old_version</span><span class="p">,</span> <span class="n">index_url</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;Output the line showing the formatted information.&quot;&quot;&quot;</span> <span class="n">msg</span> <span class="o">=</span> <span class="s">&#39;</span><span class="si">%(bd)s%(pkg_name)s%(nm)s</span><span class="s"> (</span><span class="si">%(new)s</span><span class="s">) via </span><span class="si">%(index)s</span><span class="s">. Currently </span><span class="si">%(old)s</span><span class="s">.&#39;</span> <span class="n">params</span> <span class="o">=</span> <span class="p">{</span> <span class="s">&#39;bd&#39;</span><span class="p">:</span> <span class="n">BOLD</span><span class="p">,</span> <span class="s">&#39;nm&#39;</span><span class="p">:</span> <span class="n">NORMAL</span><span class="p">,</span> <span class="s">&#39;pkg_name&#39;</span><span class="p">:</span> <span class="n">pkg_name</span><span class="p">,</span> <span class="s">&#39;new&#39;</span><span class="p">:</span> <span class="n">new_version</span><span class="p">,</span> <span class="s">&#39;old&#39;</span><span class="p">:</span> <span class="n">old_version</span><span class="p">,</span> <span class="s">&#39;index&#39;</span><span class="p">:</span> <span class="n">index_url</span><span class="p">,</span> <span class="p">}</span> <span class="k">print</span><span class="p">(</span><span class="n">msg</span> <span class="o">%</span> <span class="n">params</span><span class="p">)</span> <span class="n">NEWER</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">:</span> <span class="n">compare_versions</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">x</span><span class="p">),</span> <span class="n">y</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span> <span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">&#39;__main__&#39;</span><span class="p">:</span> <span class="kn">import</span> <span class="nn">curses</span> <span class="n">curses</span><span class="o">.</span><span class="n">setupterm</span><span class="p">()</span> <span class="n">CLEAR_SCREEN</span> <span class="o">=</span> <span class="n">curses</span><span class="o">.</span><span class="n">tigetstr</span><span class="p">(</span><span class="s">&#39;clear&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s">&#39;utf-8&#39;</span><span class="p">)</span> <span class="n">BOLD</span> <span class="o">=</span> <span class="n">curses</span><span class="o">.</span><span class="n">tigetstr</span><span class="p">(</span><span class="s">&#39;bold&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s">&#39;utf-8&#39;</span><span class="p">)</span> <span class="n">NORMAL</span> <span class="o">=</span> <span class="n">curses</span><span class="o">.</span><span class="n">tigetstr</span><span class="p">(</span><span class="s">&#39;sgr0&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s">&#39;utf-8&#39;</span><span class="p">)</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">)</span> <span class="o">&gt;</span> <span class="mi">1</span><span class="p">:</span> <span class="n">indexes</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">:]</span> <span class="k">else</span><span class="p">:</span> <span class="n">indexes</span> <span class="o">=</span> <span class="p">[</span><span class="s">&#39;http://pypi.python.org/pypi&#39;</span><span class="p">]</span> <span class="k">print</span><span class="p">(</span><span class="n">CLEAR_SCREEN</span> <span class="o">+</span> <span class="n">BOLD</span> <span class="o">+</span> <span class="s">&#39;Packages with newer versions:&#39;</span> <span class="o">+</span> <span class="n">NORMAL</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;&#39;</span><span class="p">)</span> <span class="k">for</span> <span class="n">pkg</span> <span class="ow">in</span> <span class="n">get_local_packages</span><span class="p">():</span> <span class="c"># pip outputs a single 0 at the end of the list. Ignore it.</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">pkg</span><span class="p">)</span> <span class="o">&lt;</span> <span class="mi">2</span><span class="p">:</span> <span class="k">continue</span> <span class="n">index</span><span class="p">,</span> <span class="n">current_version</span> <span class="o">=</span> <span class="n">find_current_version</span><span class="p">(</span><span class="n">pkg</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">index_urls</span><span class="o">=</span><span class="n">indexes</span><span class="p">)</span> <span class="k">if</span> <span class="n">current_version</span> <span class="ow">and</span> <span class="n">NEWER</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">current_version</span><span class="p">),</span> <span class="n">pkg</span><span class="p">[</span><span class="mi">1</span><span class="p">]):</span> <span class="n">output_line</span><span class="p">(</span><span class="n">pkg</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">current_version</span><span class="p">,</span> <span class="n">pkg</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">index</span><span class="p">)</span> </pre></div> <p>Just execute it from anywhere using a Python interpreter whose packages you&#8217;d like to check (e.g., <tt class="docutils literal">python3 check_for_updates.py</tt> or from within a virtualenv). You can then uses <tt class="docutils literal">pip install <span class="pre">-U</span> &lt;packagename&gt;</tt> (or <tt class="docutils literal"><span class="pre">pip-3.2</span></tt> for Python 3) to update the&nbsp;packages.</p> <p><strong>Update:</strong> The <a class="reference external" href="http://pypi.python.org/pypi/yolk">yolk</a> package does the same via <tt class="docutils literal">yolk <span class="pre">-U</span></tt>. It doesn’t support Python 3, though. I wonder when <span class="caps">PIP</span> will get that&nbsp;functionality.</p> A Simple Web Bot with Requests and BeautifulSoup2012-02-21T12:59:51+01:00Stefan Scherfketag:stefan.sofa-rockers.org,2012-02-21:2012/02/21/simple-web-bot-requests-and-beautifulsoup/<p>Today I helped a colleague debugging a web bot written in Java. Since I did’t really work with Java since a few years, I thought it would be easier for me to reproduce (and solve) the problem with <a class="reference external" href="http://docs.python-requests.org/en/latest/">Requests</a> and <a class="reference external" href="http://www.crummy.com/software/BeautifulSoup/">BeautifulSoup</a>. (I’ve actually been looking for an opportunity to try Requests out for a while, since I’ve heard so much good about&nbsp;it.)</p> <p>And what can I say—I was blown away by how easy it was to implement a simple web bot that filled out a form and grabbed a huge table of data for&nbsp;me.</p> <p>I cannot show you exactly that web bot, but here’s how you could search my website for “Python” and get the headings of the resulting&nbsp;posts:</p> <div class="highlight"><pre><span class="kn">from</span> <span class="nn">BeautifulSoup</span> <span class="kn">import</span> <span class="n">BeautifulSoup</span> <span class="kn">import</span> <span class="nn">requests</span> <span class="n">url</span> <span class="o">=</span> <span class="s">&#39;http://stefan.sofa-rockers.org/search/?q=</span><span class="si">%(q)s</span><span class="s">&#39;</span> <span class="n">payload</span> <span class="o">=</span> <span class="p">{</span> <span class="s">&#39;q&#39;</span><span class="p">:</span> <span class="s">&#39;Python&#39;</span><span class="p">,</span> <span class="p">}</span> <span class="n">r</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">url</span> <span class="o">%</span> <span class="n">payload</span><span class="p">)</span> <span class="n">soup</span> <span class="o">=</span> <span class="n">BeautifulSoup</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">text</span><span class="p">)</span> <span class="n">titles</span> <span class="o">=</span> <span class="p">[</span><span class="n">h2</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">h2</span> <span class="ow">in</span> <span class="n">soup</span><span class="o">.</span><span class="n">findAll</span><span class="p">(</span><span class="s">&#39;h2&#39;</span><span class="p">,</span> <span class="n">attrs</span><span class="o">=</span><span class="p">{</span><span class="s">&#39;class&#39;</span><span class="p">:</span> <span class="s">&#39;post_title&#39;</span><span class="p">})]</span> <span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="n">titles</span><span class="p">:</span> <span class="k">print</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> </pre></div> Designing and Testing PyZMQ Applications – Part 32012-02-15T18:20:22+01:00Stefan Scherfketag:stefan.sofa-rockers.org,2012-02-15:2012/02/15/designing-and-testing-pyzmq-applications-part-3/<p>The third and last part of this <a class="reference external" href="/2012/02/01/designing-and-testing-pyzmq-applications-part-1/">series</a> is again just about testing. While the <a class="reference external" href="/2012/02/07/designing-and-testing-pyzmq-applications-part-2/">previous article</a> focused on unit testing, this one will be about testing complete PyZMQ processes. This even involves some <a class="reference external" href="http://bit.ly/g3wTBC">magic</a>!</p> <p>Once you’ve made sure that your message dispatching and application logic works fine, you can actually start sending real messages to your process and checking real replies. This can be done for single processes—I call this <em>process testing</em>—and for your complete application (<em>system testing</em>).</p> <p>When you test a single process, you create sockets that mimic all processes the tested process communicates with. When you do a system test, you only mimic a client or just invoke your program from the command line and check its output (e.g., what it prints to <em>stdtout</em> and <em>stderr</em> or results written to a&nbsp;database).</p> <p>I’ll start with process testing, which is a bit more generalizable than system&nbsp;testing.</p> <div class="section" id="process-testing"> <h2>Process&nbsp;Testing</h2> <p>The biggest problem I ran into when I started testing processes was that I often made blocking calls to <em>recv</em> methods and these halted my tests and gave me no output about what actually went wrong. Though you can make them non- blocking by passing <tt class="docutils literal">zmq.<span class="caps">NOBLOCK</span></tt> as an extra argument, this doesn’t solve your problems. You will now need a very precise timing and many <tt class="docutils literal">time.sleep(x)</tt> calls, because <em>recv</em> will instantly raise an error if there is nothing to be&nbsp;received.</p> <p>My solution for this was to wrap PyZMQ sockets and add a timeout to its <em>send</em> and <em>recv</em> methods. The following wrapper will try to receive something for one second and raise an exception if that failed. There’s also <a class="reference external" href="https://bitbucket.org/ssc/pyzmq-article/src/tip/example_app/test/support.py#cl-27">a simple wrapper</a> for methods like <em>connect</em> or <em>bind</em>, but it’s really not that interesting, so I’ll omit it&nbsp;here.</p> <div class="highlight"><pre><span class="c"># test/support.py</span> <span class="k">def</span> <span class="nf">get_wrapped_fwd</span><span class="p">(</span><span class="n">func</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Returns a wrapper, that tries to call *func* multiple time in non-blocking</span> <span class="sd"> mode before rasing an :class:`zmq.ZMQError`.</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="k">def</span> <span class="nf">forwarder</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> <span class="c"># 100 tries * 0.01 second == 1 second</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">100</span><span class="p">):</span> <span class="k">try</span><span class="p">:</span> <span class="n">rep</span> <span class="o">=</span> <span class="n">func</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="n">flags</span><span class="o">=</span><span class="n">zmq</span><span class="o">.</span><span class="n">NOBLOCK</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> <span class="k">return</span> <span class="n">rep</span> <span class="k">except</span> <span class="n">zmq</span><span class="o">.</span><span class="n">ZMQError</span><span class="p">:</span> <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.01</span><span class="p">)</span> <span class="c"># We should not get here, so raise an error.</span> <span class="n">msg</span> <span class="o">=</span> <span class="s">&#39;Could not </span><span class="si">%s</span><span class="s"> message.&#39;</span> <span class="o">%</span> <span class="n">func</span><span class="o">.</span><span class="n">__name__</span><span class="p">[:</span><span class="mi">4</span><span class="p">]</span> <span class="k">raise</span> <span class="n">zmq</span><span class="o">.</span><span class="n">ZMQError</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span> <span class="k">return</span> <span class="n">forwarder</span> </pre></div> <p>This wrapper is now used to create a <em>TestSocket</em> class with the desired&nbsp;behavior:</p> <div class="highlight"><pre><span class="c"># test/support.py</span> <span class="k">class</span> <span class="nc">TestSocket</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Wraps ZMQ :class:`~zmq.core.socket.Socket`. All *recv* and *send* methods</span> <span class="sd"> will be called multiple times in non-blocking mode before a</span> <span class="sd"> :class:`zmq.ZMQError` is raised.</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">,</span> <span class="n">sock_type</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">_context</span> <span class="o">=</span> <span class="n">context</span> <span class="n">sock</span> <span class="o">=</span> <span class="n">context</span><span class="o">.</span><span class="n">socket</span><span class="p">(</span><span class="n">sock_type</span><span class="p">)</span> <span class="bp">self</span><span class="o">.</span><span class="n">_sock</span> <span class="o">=</span> <span class="n">sock</span> <span class="n">forwards</span> <span class="o">=</span> <span class="p">[</span> <span class="c"># These methods can simply be forwarded</span> <span class="n">sock</span><span class="o">.</span><span class="n">bind</span><span class="p">,</span> <span class="n">sock</span><span class="o">.</span><span class="n">bind_to_random_port</span><span class="p">,</span> <span class="n">sock</span><span class="o">.</span><span class="n">connect</span><span class="p">,</span> <span class="n">sock</span><span class="o">.</span><span class="n">close</span><span class="p">,</span> <span class="n">sock</span><span class="o">.</span><span class="n">setsockopt</span><span class="p">,</span> <span class="p">]</span> <span class="n">wrapped_fwd</span> <span class="o">=</span> <span class="p">[</span> <span class="c"># These methods are wrapped with a for loop</span> <span class="n">sock</span><span class="o">.</span><span class="n">recv</span><span class="p">,</span> <span class="n">sock</span><span class="o">.</span><span class="n">recv_json</span><span class="p">,</span> <span class="n">sock</span><span class="o">.</span><span class="n">recv_multipart</span><span class="p">,</span> <span class="n">sock</span><span class="o">.</span><span class="n">recv_unicode</span><span class="p">,</span> <span class="n">sock</span><span class="o">.</span><span class="n">send</span><span class="p">,</span> <span class="n">sock</span><span class="o">.</span><span class="n">send_json</span><span class="p">,</span> <span class="n">sock</span><span class="o">.</span><span class="n">send_multipart</span><span class="p">,</span> <span class="n">sock</span><span class="o">.</span><span class="n">send_unicode</span><span class="p">,</span> <span class="p">]</span> <span class="k">for</span> <span class="n">func</span> <span class="ow">in</span> <span class="n">forwards</span><span class="p">:</span> <span class="nb">setattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">func</span><span class="o">.</span><span class="n">__name__</span><span class="p">,</span> <span class="n">get_forwarder</span><span class="p">(</span><span class="n">func</span><span class="p">))</span> <span class="k">for</span> <span class="n">func</span> <span class="ow">in</span> <span class="n">wrapped_fwd</span><span class="p">:</span> <span class="nb">setattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">func</span><span class="o">.</span><span class="n">__name__</span><span class="p">,</span> <span class="n">get_wrapped_fwd</span><span class="p">(</span><span class="n">func</span><span class="p">))</span> </pre></div> <p>In order to reuse the same ports for all test methods, you need to cleanly close all sockets after each test. To handle method level setup/teardown in pytest, you need to implement a <em>setup_method</em> and a <em>teardown_method</em>. In the setup method, you create one or more <em>TestSocket</em> instances that mimic other processes and you also start the process to be&nbsp;tested:</p> <div class="highlight"><pre><span class="c"># test/process/test_pongproc.py</span> <span class="kn">import</span> <span class="nn">pytest</span> <span class="kn">import</span> <span class="nn">zmq</span> <span class="kn">from</span> <span class="nn">test.support</span> <span class="kn">import</span> <span class="n">ProcessTest</span><span class="p">,</span> <span class="n">make_sock</span> <span class="kn">import</span> <span class="nn">pongproc</span> <span class="n">host</span> <span class="o">=</span> <span class="s">&#39;127.0.0.1&#39;</span> <span class="n">port</span> <span class="o">=</span> <span class="mi">5678</span> <span class="k">class</span> <span class="nc">TestProngProc</span><span class="p">(</span><span class="n">ProcessTest</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;Communication test for the Platform Manager process.&quot;&quot;&quot;</span> <span class="k">def</span> <span class="nf">setup_method</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">method</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Creates and starts a PongProc process and sets up sockets to</span> <span class="sd"> communicate with it.</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="bp">self</span><span class="o">.</span><span class="n">context</span> <span class="o">=</span> <span class="n">zmq</span><span class="o">.</span><span class="n">Context</span><span class="p">()</span> <span class="c"># make_sock creates and connects a TestSocket that we will use to</span> <span class="c"># mimic the Ping process</span> <span class="bp">self</span><span class="o">.</span><span class="n">req_sock</span> <span class="o">=</span> <span class="n">make_sock</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">context</span><span class="p">,</span> <span class="n">zmq</span><span class="o">.</span><span class="n">REQ</span><span class="p">,</span> <span class="n">connect</span><span class="o">=</span><span class="p">(</span><span class="n">host</span><span class="p">,</span> <span class="n">port</span><span class="p">))</span> <span class="bp">self</span><span class="o">.</span><span class="n">pp</span> <span class="o">=</span> <span class="n">pongproc</span><span class="o">.</span><span class="n">PongProc</span><span class="p">((</span><span class="n">host</span><span class="p">,</span> <span class="n">port</span><span class="p">))</span> <span class="bp">self</span><span class="o">.</span><span class="n">pp</span><span class="o">.</span><span class="n">start</span><span class="p">()</span> <span class="k">def</span> <span class="nf">teardown_method</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">method</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Sends a kill message to the pp and waits for the process to terminate.</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="c"># Send a stop message to the prong process and wait until it joins</span> <span class="bp">self</span><span class="o">.</span><span class="n">req_sock</span><span class="o">.</span><span class="n">send_multipart</span><span class="p">([</span><span class="n">b</span><span class="s">&#39;[&quot;plzdiekthxbye&quot;, null]&#39;</span><span class="p">])</span> <span class="bp">self</span><span class="o">.</span><span class="n">pp</span><span class="o">.</span><span class="n">join</span><span class="p">()</span> <span class="bp">self</span><span class="o">.</span><span class="n">req_sock</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> </pre></div> <p>You may have noticed that our test class inherits <em>ProcessTests</em>. This class and some helpers in a <a class="reference external" href="http://pytest.org/latest/plugins.html">conftest.py</a> allow us to use some magic that improves the readability of the actual&nbsp;test:</p> <div class="highlight"><pre><span class="c"># test/process/test_pongproc.py</span> <span class="k">def</span> <span class="nf">test_ping</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;Tests a ping-pong sequence.&quot;&quot;&quot;</span> <span class="k">yield</span> <span class="p">(</span><span class="s">&#39;send&#39;</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">req_sock</span><span class="p">,</span> <span class="p">[],</span> <span class="p">[</span><span class="s">&#39;ping&#39;</span><span class="p">,</span> <span class="mi">1</span><span class="p">])</span> <span class="n">reply</span> <span class="o">=</span> <span class="k">yield</span> <span class="p">(</span><span class="s">&#39;recv&#39;</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">req_sock</span><span class="p">)</span> <span class="k">assert</span> <span class="n">reply</span> <span class="o">==</span> <span class="p">[[</span><span class="s">&#39;pong&#39;</span><span class="p">,</span> <span class="mi">1</span><span class="p">]]</span> </pre></div> <p>You can just yield <em>send</em> or <em>recv</em> events from your test case! When you yield a <em>send</em>, the test machinery tries to send a message via the specified socket. When you yield a receive, <em>ProcessTest</em> tries to receive something from the socket and sends its result back to your test function, so that you can easily compare the reply with the expected&nbsp;result.</p> <p>The example above is roughly equivalent to the following&nbsp;code:</p> <div class="highlight"><pre><span class="bp">self</span><span class="o">.</span><span class="n">req_sock</span><span class="o">.</span><span class="n">send_multipart</span><span class="p">([]</span> <span class="o">+</span> <span class="p">[</span><span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">([</span><span class="s">&#39;ping&#39;</span><span class="p">,</span> <span class="mi">1</span><span class="p">])])</span> <span class="n">reply</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">req_sock</span><span class="o">.</span><span class="n">recv_multipart</span><span class="p">()</span> <span class="n">reply</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">[</span><span class="n">reply</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]]</span> <span class="k">assert</span> <span class="n">reply</span> <span class="o">==</span> <span class="p">[[</span><span class="s">&#39;pong&#39;</span><span class="p">,</span> <span class="mi">1</span><span class="p">]]</span> </pre></div> <p>So how does this work? By default, if pytests finds a test function that is a generator, it assumes that it generates further test functions. Hence, our first step is to override this behavior. We can do this in a <tt class="docutils literal">conftest.py</tt> file in the <tt class="docutils literal">test/process/</tt> directory by implementing a <em>pytest_pycollect_makeitem</em> function. In this case, we collect generator functions like normal&nbsp;functions:</p> <div class="highlight"><pre><span class="c"># test/process/conftest.py</span> <span class="kn">from</span> <span class="nn">inspect</span> <span class="kn">import</span> <span class="n">isfunction</span><span class="p">,</span> <span class="n">isgeneratorfunction</span> <span class="k">def</span> <span class="nf">pytest_pycollect_makeitem</span><span class="p">(</span><span class="n">collector</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">obj</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Collects all instance methods that are generators and returns them as</span> <span class="sd"> normal function items.</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="k">if</span> <span class="n">collector</span><span class="o">.</span><span class="n">funcnamefilter</span><span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="ow">and</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">obj</span><span class="p">,</span> <span class="s">&#39;__call__&#39;</span><span class="p">):</span> <span class="k">if</span> <span class="n">isfunction</span><span class="p">(</span><span class="n">obj</span><span class="p">)</span> <span class="ow">or</span> <span class="n">isgeneratorfunction</span><span class="p">(</span><span class="n">obj</span><span class="p">):</span> <span class="k">return</span> <span class="n">collector</span><span class="o">.</span><span class="n">_genfunctions</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">obj</span><span class="p">)</span> </pre></div> <p>Now, we need to tell pytest how to run a test on the collected generator functions. This can be done by implementing <em>pytest_runtest_call</em>. If the object we are going to test (<tt class="docutils literal">item.obj</tt>) is a generator function, we call the <em>run</em> method of the object’s instance (<tt class="docutils literal">item.obj.__self__.run</tt>) and pass the generator function to it. If the test item contains a normal function, we run the default&nbsp;test.</p> <div class="highlight"><pre><span class="c"># test/process/conftest.py</span> <span class="k">def</span> <span class="nf">pytest_runtest_call</span><span class="p">(</span><span class="n">item</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Passes the test generator (``item.obj``) to the ``run()`` method of the</span> <span class="sd"> generator&#39;s instance. This method should be inherited from</span> <span class="sd"> :class:`test.support.ProcessTest`.</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="k">if</span> <span class="n">isgeneratorfunction</span><span class="p">(</span><span class="n">item</span><span class="o">.</span><span class="n">obj</span><span class="p">):</span> <span class="n">item</span><span class="o">.</span><span class="n">obj</span><span class="o">.</span><span class="n">__self__</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">item</span><span class="o">.</span><span class="n">obj</span><span class="p">)</span> <span class="k">else</span><span class="p">:</span> <span class="c"># Normal test execution for normal instance methods</span> <span class="n">item</span><span class="o">.</span><span class="n">runtest</span><span class="p">()</span> </pre></div> <p>But wait—we didn’t implement a <em>run</em> method in our test case! So it must be inherited from <em>ProcessTest</em>. Let’s take a look at&nbsp;it:</p> <div class="highlight"><pre><span class="c"># test/support.py</span> <span class="k">class</span> <span class="nc">ProcessTest</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Base class for process tests. It offers basic actions for sending and</span> <span class="sd"> receiving messages and implements the *run* methods that handles the</span> <span class="sd"> actual test generators.</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">testfunc</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Iterates over the *testfunc* generator and executes all actions it</span> <span class="sd"> yields. Results will be sent back into the generator.</span> <span class="sd"> :param testfunc: A generator function that yields tuples containing</span> <span class="sd"> an action keyword, which should be a function of this or</span> <span class="sd"> the inheriting class (like ``send`` or ``recv``) and additional</span> <span class="sd"> parameters that will be passed to that function, e.g.:</span> <span class="sd"> ``(&#39;send&#39;, socket_obj, [&#39;header&#39;], &#39;body&#39;)``</span> <span class="sd"> :type testfunc: generatorfunction</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="n">item_gen</span> <span class="o">=</span> <span class="n">testfunc</span><span class="p">()</span> <span class="n">item</span> <span class="o">=</span> <span class="nb">next</span><span class="p">(</span><span class="n">item_gen</span><span class="p">)</span> <span class="k">def</span> <span class="nf">throw_err</span><span class="p">(</span><span class="n">skip_levels</span><span class="o">=</span><span class="mi">0</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Throws the last error to *item_gen* and skips *skip_levels* in</span> <span class="sd"> the traceback to point to the line that yielded the last event.</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="n">etype</span><span class="p">,</span> <span class="n">evalue</span><span class="p">,</span> <span class="n">tb</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">exc_info</span><span class="p">()</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">skip_levels</span><span class="p">):</span> <span class="n">tb</span> <span class="o">=</span> <span class="n">tb</span><span class="o">.</span><span class="n">tb_next</span> <span class="n">item_gen</span><span class="o">.</span><span class="n">throw</span><span class="p">(</span><span class="n">etype</span><span class="p">,</span> <span class="n">evalue</span><span class="p">,</span> <span class="n">tb</span><span class="p">)</span> <span class="k">try</span><span class="p">:</span> <span class="k">while</span> <span class="bp">True</span><span class="p">:</span> <span class="k">try</span><span class="p">:</span> <span class="c"># Call the event handler and pass the args,</span> <span class="c"># e.g., self.send(socket_obj, header, body)</span> <span class="n">ret</span> <span class="o">=</span> <span class="nb">getattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">item</span><span class="p">[</span><span class="mi">0</span><span class="p">])(</span><span class="o">*</span><span class="n">item</span><span class="p">[</span><span class="mi">1</span><span class="p">:])</span> <span class="c"># Send the results back to the test and get the next item</span> <span class="n">item</span> <span class="o">=</span> <span class="n">item_gen</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">ret</span><span class="p">)</span> <span class="k">except</span> <span class="n">zmq</span><span class="o">.</span><span class="n">ZMQError</span><span class="p">:</span> <span class="n">throw_err</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="c"># PyZMQ could not send/recv</span> <span class="k">except</span> <span class="ne">AssertionError</span><span class="p">:</span> <span class="n">throw_err</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="c"># Error in the test</span> <span class="k">except</span> <span class="ne">StopIteration</span><span class="p">:</span> <span class="k">pass</span> </pre></div> <p>The <em>run</em> method simply iterates over all events our <em>testfunc</em> generates and calls a method with the name of the event (e.g., <em>send</em> or <em>recv</em>). Their return value is sent back into the generator. If an error occurs, the exception’s traceback is modified to point to the line of code that yielded the according event and not to the <em>run</em> method&nbsp;itself.</p> <p>The methods <em>send</em> and <em>recv</em> roughly do the same as the snippet I showed you&nbsp;above:</p> <div class="highlight"><pre><span class="c"># test/support.py</span> <span class="k">def</span> <span class="nf">send</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">socket</span><span class="p">,</span> <span class="n">header</span><span class="p">,</span> <span class="n">body</span><span class="p">,</span> <span class="n">extra_data</span><span class="o">=</span><span class="p">[]):</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> JSON-encodes *body*, concatenates it with *header*, appends</span> <span class="sd"> *extra_data* and sends it as multipart message over *socket*.</span> <span class="sd"> *header* and *extra_data* should be lists containg byte objects or</span> <span class="sd"> objects implementing the buffer interface (like NumPy arrays).</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="n">socket</span><span class="o">.</span><span class="n">send_multipart</span><span class="p">(</span><span class="n">header</span> <span class="o">+</span> <span class="p">[</span><span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">body</span><span class="p">)]</span> <span class="o">+</span> <span class="n">extra_data</span><span class="p">)</span> <span class="k">def</span> <span class="nf">recv</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">socket</span><span class="p">,</span> <span class="n">json_load_index</span><span class="o">=-</span><span class="mi">1</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Receives and returns a multipart message from *socket* and tries to</span> <span class="sd"> JSON-decode the item at position *json_load_index* (defaults to ``-1``;</span> <span class="sd"> the last element in the list). The original byte string will be</span> <span class="sd"> replaced by the loaded object. Set *json_load_index* to ``None`` to get</span> <span class="sd"> the original, unchanged message.</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="n">msg</span> <span class="o">=</span> <span class="n">socket</span><span class="o">.</span><span class="n">recv_multipart</span><span class="p">()</span> <span class="k">if</span> <span class="n">json_load_index</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span><span class="p">:</span> <span class="n">msg</span><span class="p">[</span><span class="n">json_load_index</span><span class="p">]</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">msg</span><span class="p">[</span><span class="n">json_load_index</span><span class="p">])</span> <span class="k">return</span> <span class="n">msg</span> </pre></div> <p>You can even add your own event handler to your test class. I used this, for example, to add a <em>log</em> event that checks if a PyZMQ log handler sent the expected log&nbsp;messages:</p> <div class="highlight"><pre><span class="k">def</span> <span class="nf">log</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">substr</span><span class="o">=</span><span class="s">&#39;&#39;</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Receives a message and asserts, that it is a log message and that</span> <span class="sd"> *substr* is in that message.</span> <span class="sd"> Usage:</span> <span class="sd"> yield (&#39;log&#39;, &#39;Ai iz in ur log mesage&#39;)</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="n">msg</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">log_sock</span><span class="o">.</span><span class="n">recv_json</span><span class="p">()</span> <span class="k">assert</span> <span class="n">msg</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">==</span> <span class="s">&#39;log_message&#39;</span> <span class="k">assert</span> <span class="n">substr</span> <span class="ow">in</span> <span class="n">msg</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> </pre></div> <div class="section" id="what-if-your-process-starts-further-subprocesses"> <h3>What if your process starts further&nbsp;subprocesses?</h3> <p>In some cases, the process you are about to test starts additional subprocesses that you don’t want to test. Even worse, these processes might communicate via sockets bound to random ports. And <span class="caps">EVEN</span> <span class="caps">WORSE</span>, the process you are testing might depend on excepting a <em>KeyboardInterrupt</em> to send stop messages to child processes or to clean something&nbsp;up!</p> <p>The last problem is quite easy to solve: You just a send a <em><span class="caps">SIGINT</span></em> to your process from the&nbsp;test:</p> <div class="highlight"><pre><span class="kn">import</span> <span class="nn">os</span><span class="o">,</span> <span class="nn">signal</span> <span class="k">def</span> <span class="nf">teardown_method</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">method</span><span class="p">):</span> <span class="n">os</span><span class="o">.</span><span class="n">kill</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">my_proc</span><span class="o">.</span><span class="n">pid</span><span class="p">,</span> <span class="n">signal</span><span class="o">.</span><span class="n">SIGINT</span><span class="p">)</span> <span class="bp">self</span><span class="o">.</span><span class="n">my_proc</span><span class="o">.</span><span class="n">join</span><span class="p">()</span> <span class="c"># Now you can close the test sockets</span> </pre></div> <p>If you don’t want to start a certain subprocess, you can just mock it. Imagine, you have two processes <tt class="docutils literal">a.A</tt> and <tt class="docutils literal">b.B</tt>, where <em>A</em> starts <em>B</em>, then you just mock <em>B</em> before starting <em>A</em>:</p> <div class="highlight"><pre><span class="k">with</span> <span class="n">mock</span><span class="o">.</span><span class="n">patch</span><span class="p">(</span><span class="s">&#39;b.B&#39;</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">a</span> <span class="o">=</span> <span class="n">A</span><span class="p">()</span> <span class="bp">self</span><span class="o">.</span><span class="n">a</span><span class="o">.</span><span class="n">start</span><span class="p">()</span> </pre></div> <p>Imagine now, that <em>A</em> binds a socket to a random port and uses that socket to communicate with <em>B</em>. If you want to mock <em>B</em> in your tests, you need that port number in order to connect to it and send messages to <em>A</em>.</p> <p>But how can you get that number? When <em>A</em> creates <em>B</em>, it already runs in its own process, so a simple attribute access won’t work. Setting a random seed would only work if you did that directly in <em>A</em> when it’s already running. But doing that just for the tests is not such a good idea. It also may not work reliably on all systems and Python&nbsp;versions.</p> <p>However, <em>A</em> must pass the socket number to <em>B</em>, so that <em>B</em> can connect to <em>A</em>. Thus, we can create a mock for <em>B</em> that will send us its port number via a <cite>queue &lt;http://docs.python.org/py3k/library/multiprocessing#exchanging-objects- between-processes&gt;</cite>:</p> <div class="highlight"><pre><span class="k">class</span> <span class="nc">ProcMock</span><span class="p">(</span><span class="n">mock</span><span class="o">.</span><span class="n">Mock</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> This mock returns itself when called, so it acts like both, the</span> <span class="sd"> process’ class and instance object.</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">__init__</span><span class="p">()</span> <span class="bp">self</span><span class="o">.</span><span class="n">queue</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Queue</span><span class="p">()</span> <span class="k">def</span> <span class="nf">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">port</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;Will be called when A instantiates B and passes its port number.&quot;&quot;&quot;</span> <span class="bp">self</span><span class="o">.</span><span class="n">queue</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">port</span><span class="p">)</span> <span class="k">return</span> <span class="bp">self</span> <span class="k">def</span> <span class="nf">start</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">return</span> <span class="c"># Just make sure the methods exists and returns nothing</span> <span class="k">def</span> <span class="nf">join</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">return</span> <span class="c"># Just make sure the methods exists and returns nothing</span> <span class="k">class</span> <span class="nc">TestA</span><span class="p">(</span><span class="n">ProcessTest</span><span class="p">):</span> <span class="k">def</span> <span class="nf">setup_method</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="n">b_mock</span> <span class="o">=</span> <span class="n">ProcMock</span><span class="p">()</span> <span class="k">with</span> <span class="n">mock</span><span class="o">.</span><span class="n">patch</span><span class="p">(</span><span class="s">&#39;b.B&#39;</span><span class="p">,</span> <span class="n">new</span><span class="o">=</span><span class="n">b_mock</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">a</span> <span class="o">=</span> <span class="n">A</span><span class="p">()</span> <span class="bp">self</span><span class="o">.</span><span class="n">a</span><span class="o">.</span><span class="n">start</span><span class="p">()</span> <span class="c"># Get the port A is listening on</span> <span class="n">port</span> <span class="o">=</span> <span class="n">b_mock</span><span class="o">.</span><span class="n">queue</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="c"># ...</span> </pre></div> <p>As you’ve seen, process testing is really not as simple as unit testing. But I always found bugs with it that my unit tests coudn’t detect. If you cover all communication sequences for a process in a process test, you can be pretty sure, that it will also work flawlessly in the final&nbsp;application.</p> </div> </div> <div class="section" id="system-testing"> <h2>System&nbsp;Testing</h2> <p>If your application consists of more than one process, you still need to test whether all processes work nicely together or not. This is something you cannot simulate reliably with a process tests, as much as unit tests can’t replace the process&nbsp;test.</p> <p>Writing a good system test is very application-specific and can, depending on the complexity of your application, be very hard or very easy. Fortunately, the latter is the case for our ping-pong app. We just start it and copy its output to a file. If the output is not what we expected, we modify the file accordingly. In our test, we can now simply invoke our programm again, capture its output and compare it to the contents of the file we created&nbsp;before:</p> <div class="highlight"><pre><span class="c"># test/system/test_pongproc.py</span> <span class="kn">import</span> <span class="nn">os.path</span> <span class="kn">import</span> <span class="nn">subprocess</span> <span class="kn">import</span> <span class="nn">pytest</span> <span class="k">def</span> <span class="nf">test_pongproc</span><span class="p">():</span> <span class="n">filename</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="s">&#39;test&#39;</span><span class="p">,</span> <span class="s">&#39;data&#39;</span><span class="p">,</span> <span class="s">&#39;pongproc.out&#39;</span><span class="p">)</span> <span class="n">expected</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">()</span> <span class="n">output</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">check_output</span><span class="p">([</span><span class="s">&#39;python&#39;</span><span class="p">,</span> <span class="s">&#39;pongproc.py&#39;</span><span class="p">],</span> <span class="n">universal_newlines</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> <span class="k">assert</span> <span class="n">output</span> <span class="o">==</span> <span class="n">expected</span> </pre></div> <p>If your application was a server, another way of doing the system test would be to emulate a client that speaks with it. Your system test would then be very similar to your process tests, except that you only mimic the client and not all processes your main process communicates&nbsp;with.</p> <p>Other applications (like, for instance, simulations) might create a database containing collected data. Here, you might check if these results match your&nbsp;expectations.</p> <p>Of course you can also combine these possibilities or do something completely different&nbsp;…</p> </div> <div class="section" id="my-test-are-now-running-sooo-slow"> <h2>“My Test Are now Running sooo&nbsp;Slow!”</h2> <p>System and process tests often run much slower than simple unit tests, so you may want to skip them most of the time. Pytest allows you to <a class="reference external" href="http://pytest.org/latest/example/markers.html#mark-examples">mark</a> a test with a given name. You can then (de)select tests based on their mark when you invoke&nbsp;pytest.</p> <p>To mark a module e.g. as <tt class="docutils literal">process</tt> test, just put a line <tt class="docutils literal">pytestmark = pytest.mark.process</tt> somewhere in it. Likewise, you can add a <tt class="docutils literal">pytestmark = pytest.mark.system</tt> to mark a module as system&nbsp;test.</p> <p>You can now deselect process and system&nbsp;tests:</p> <div class="highlight"><pre><span class="nv">$ </span>py.test -m <span class="s2">&quot;not (process or system)&quot;</span> </pre></div> <p>You can put this into a <em>pytest.ini</em> as a default setting. To override this again, use <tt class="docutils literal">1</tt> or <tt class="docutils literal">True</tt> as selection&nbsp;expression:</p> <div class="highlight"><pre><span class="nv">$ </span>py.test -m 1 </pre></div> </div> <div class="section" id="summary"> <h2>Summary</h2> <p>Process and system testing were the last two topics I wanted to cover in this series. Compared to simple unit tests, they require a bit more effort. I think they are definitely worth the extra work since they give you a lot more confidence that your program actually works, because they are much more realistic than unit tests can&nbsp;be.</p> <p>In the end, these articles became much longer and more elaborate then I originally planned them to be. However, I hope they provided a good overview about how to design and test applications with PyZMQ and I hope that you now run into much less problems than I did when I first started working with&nbsp;PyZMQ.</p> </div> Designing and Testing PyZMQ Applications – Part 22012-02-07T18:54:47+01:00Stefan Scherfketag:stefan.sofa-rockers.org,2012-02-07:2012/02/07/designing-and-testing-pyzmq-applications-part-2/<p>This is the second part of the series <em>Designing and Testing PyZMQ Applications</em>. In <a class="reference external" href="/2012/02/01/designing-and-testing-pyzmq-applications-part-1/">the first part</a>, I wrote about designing a PyZMQ application, so this time it’s all about (unit) testing (remember, if it’s not tested, it’s broken). I also updated the <a class="reference external" href="https://bitbucket.org/ssc/pyzmq-article">repository for this article</a> with the new code&nbsp;examples.</p> <p>My favorite testing tools are <a class="reference external" href="http://pytest.org/latest/">pytest</a> by Holger Krekel and <a class="reference external" href="http://www.voidspace.org.uk/python/mock/">Mock</a> by Michael Ford. Pytest is particularly awesome because of its re-evaluation of <tt class="docutils literal">assert</tt> statements. If your test contains an <tt class="docutils literal">assert spam == 'eggs'</tt> and the assert fails, pytest re-evaluates it and prints the value of <tt class="docutils literal">spam</tt>. Really helpful and you don’t need any boilerplate code for that. Mock is really nice for mocking external dependencies and asserting that your code called them in the correct&nbsp;way.</p> <p>If you cloned the <a class="reference external" href="https://bitbucket.org/ssc/pyzmq-article">repository for this article</a>, just run <tt class="docutils literal">py.test</tt> from its root&nbsp;directory:</p> <div class="highlight"><pre><span class="nv">$ </span>pip install pytest mock ... Successfully installed pytest mock Cleaning up... <span class="nv">$ </span>py.test <span class="o">===================</span> <span class="nb">test </span>session <span class="nv">starts</span> <span class="o">====================</span> platform darwin -- Python 3.2.2 -- pytest-2.2.3 collected 11 items example_app/test/test_base.py .... example_app/test/test_pongproc.py ....... <span class="o">================</span> 11 passed in 0.12 <span class="nv">seconds</span> <span class="o">=================</span> </pre></div> <div class="section" id="unit-testing"> <h2>Unit&nbsp;Testing</h2> <p>The probability that PyZMQ works correctly is very high. The probability that your code will call a PyZMQ function in such a way that it blocks forever and halts your test runner is also very high. Therefore, it’s a good idea to mock everything PyZMQ-related for your unit tests. And since your application logic might also not be implemented when you start testing your process, you should mock that,&nbsp;too.</p> <p>What you’ll actually end up testing is the&nbsp;following:</p> <ul class="simple"> <li>Does your message handler call your application logic in the right way given a certain input&nbsp;message?</li> <li>Does your message handler create and send the correct reply based on the return value of your application&nbsp;logic?</li> </ul> <div class="section" id="zmqprocess"> <h3>ZmqProcess</h3> <p>Let’s start with <tt class="docutils literal">ZmqProcess</tt> again. After all, everything else depends on it. Testing its <em>setup</em> method is easy. We just check that it creates a <em>context</em> and a <em>loop</em>:</p> <div class="highlight"><pre><span class="c"># example_app/test/test_zmqproc.py</span> <span class="kn">from</span> <span class="nn">zmq.eventloop</span> <span class="kn">import</span> <span class="n">ioloop</span> <span class="kn">import</span> <span class="nn">mock</span> <span class="kn">import</span> <span class="nn">pytest</span> <span class="kn">import</span> <span class="nn">zmq</span> <span class="kn">import</span> <span class="nn">zmqproc</span> <span class="k">class</span> <span class="nc">TestZmqProcess</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;Tests for :class:`base.ZmqProcess`.&quot;&quot;&quot;</span> <span class="k">def</span> <span class="nf">test_setup</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="n">zp</span> <span class="o">=</span> <span class="n">base</span><span class="o">.</span><span class="n">ZmqProcess</span><span class="p">()</span> <span class="n">zp</span><span class="o">.</span><span class="n">setup</span><span class="p">()</span> <span class="k">assert</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">zp</span><span class="o">.</span><span class="n">context</span><span class="p">,</span> <span class="n">zmq</span><span class="o">.</span><span class="n">Context</span><span class="p">)</span> <span class="k">assert</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">zp</span><span class="o">.</span><span class="n">loop</span><span class="p">,</span> <span class="n">ioloop</span><span class="o">.</span><span class="n">IOLoop</span><span class="p">)</span> </pre></div> <p>Testing <em>stream</em> is more complicated. We need to test if it can handle various address formats, if it creates or binds correctly and if it performs a default subscription for <em><span class="caps">SUB</span></em>&nbsp;sockets.</p> <p>Pytest 2.2 introduced a <em>parametrize</em> decorator, that helps calling a test multiple times with varying inputs. You just define one or more arguments for your test function and a list of values for these arguments. For <em>test_stream</em>, I only need a <em>kwargs</em> parameter containing the parameters for the <em>stream</em>&nbsp;call:</p> <div class="highlight"><pre><span class="c"># example_app/test/test_zmqproc.py</span> <span class="nd">@pytest.mark.parametrize</span><span class="p">(</span><span class="s">&#39;kwargs&#39;</span><span class="p">,</span> <span class="p">[</span> <span class="nb">dict</span><span class="p">(</span><span class="n">sock_type</span><span class="o">=</span><span class="mi">23</span><span class="p">,</span> <span class="n">addr</span><span class="o">=</span><span class="s">&#39;127.0.0.1:1234&#39;</span><span class="p">,</span> <span class="n">bind</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">callback</span><span class="o">=</span><span class="n">mock</span><span class="o">.</span><span class="n">Mock</span><span class="p">()),</span> <span class="nb">dict</span><span class="p">(</span><span class="n">sock_type</span><span class="o">=</span><span class="mi">23</span><span class="p">,</span> <span class="n">addr</span><span class="o">=</span><span class="s">&#39;127.0.0.1&#39;</span><span class="p">,</span> <span class="n">bind</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">callback</span><span class="o">=</span><span class="n">mock</span><span class="o">.</span><span class="n">Mock</span><span class="p">()),</span> <span class="nb">dict</span><span class="p">(</span><span class="n">sock_type</span><span class="o">=</span><span class="n">zmq</span><span class="o">.</span><span class="n">SUB</span><span class="p">,</span> <span class="n">addr</span><span class="o">=</span><span class="p">(</span><span class="s">&#39;localhost&#39;</span><span class="p">,</span> <span class="mi">1234</span><span class="p">),</span> <span class="n">bind</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">callback</span><span class="o">=</span><span class="n">mock</span><span class="o">.</span><span class="n">Mock</span><span class="p">(),</span> <span class="n">subscribe</span><span class="o">=</span><span class="n">b</span><span class="s">&#39;ohai&#39;</span><span class="p">),</span> <span class="p">])</span> <span class="k">def</span> <span class="nf">test_stream</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">kwargs</span><span class="p">):</span> </pre></div> <p>The next step is to create an instance of <em>ZmqProcess</em> and patch some of its attributes. We also need to set a defined return value for the socket’s <em>bind_to_random_port</em>&nbsp;method:</p> <div class="highlight"><pre><span class="c"># example_app/test/test_zmqproc.py</span> <span class="n">zp</span> <span class="o">=</span> <span class="n">base</span><span class="o">.</span><span class="n">ZmqProcess</span><span class="p">()</span> <span class="c"># Patch the ZmqProcess instance</span> <span class="n">zp</span><span class="o">.</span><span class="n">context</span> <span class="o">=</span> <span class="n">mock</span><span class="o">.</span><span class="n">Mock</span><span class="p">(</span><span class="n">spec_set</span><span class="o">=</span><span class="n">zmq</span><span class="o">.</span><span class="n">Context</span><span class="p">)</span> <span class="n">zp</span><span class="o">.</span><span class="n">loop</span> <span class="o">=</span> <span class="n">mock</span><span class="o">.</span><span class="n">Mock</span><span class="p">(</span><span class="n">spec_set</span><span class="o">=</span><span class="n">ioloop</span><span class="o">.</span><span class="n">IOLoop</span><span class="p">)</span> <span class="n">sock_mock</span> <span class="o">=</span> <span class="n">zp</span><span class="o">.</span><span class="n">context</span><span class="o">.</span><span class="n">socket</span><span class="o">.</span><span class="n">return_value</span> <span class="n">sock_mock</span><span class="o">.</span><span class="n">bind_to_random_port</span><span class="o">.</span><span class="n">return_value</span> <span class="o">=</span> <span class="mi">42</span> </pre></div> <p>For the actual test, we also need to patch <em>ZMQStream</em>. Although <em>mock.patch</em> could work as a function decorator, we need to use it as context processor if we also uses pytest funcargs (e.g., via the <em>parametrize</em> decorator—I don’t know if it’s even possible to uses both, <em>mock.patch</em> as decorator and pytest funcargs in one&nbsp;test).</p> <div class="highlight"><pre><span class="c"># example_app/test/test_zmqproc.py</span> <span class="c"># Patch ZMQStream and start testing</span> <span class="k">with</span> <span class="n">mock</span><span class="o">.</span><span class="n">patch</span><span class="p">(</span><span class="s">&#39;zmq.eventloop.zmqstream.ZMQStream&#39;</span><span class="p">)</span> <span class="k">as</span> <span class="n">zmqstream_mock</span><span class="p">:</span> <span class="n">stream</span><span class="p">,</span> <span class="n">port</span> <span class="o">=</span> <span class="n">zp</span><span class="o">.</span><span class="n">stream</span><span class="p">(</span><span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> </pre></div> <p>Finally, we can check the return values of our <em>stream</em> method and it made the correct calls to create the&nbsp;stream:</p> <div class="highlight"><pre><span class="c"># example_app/test/test_zmqproc.py</span> <span class="c"># Assert that the return values are correct</span> <span class="k">assert</span> <span class="n">stream</span> <span class="ow">is</span> <span class="n">zmqstream_mock</span><span class="o">.</span><span class="n">return_value</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">kwargs</span><span class="p">[</span><span class="s">&#39;addr&#39;</span><span class="p">],</span> <span class="nb">tuple</span><span class="p">):</span> <span class="k">assert</span> <span class="n">port</span> <span class="o">==</span> <span class="n">kwargs</span><span class="p">[</span><span class="s">&#39;addr&#39;</span><span class="p">][</span><span class="mi">1</span><span class="p">]</span> <span class="k">elif</span> <span class="s">&#39;:&#39;</span> <span class="ow">in</span> <span class="n">kwargs</span><span class="p">[</span><span class="s">&#39;addr&#39;</span><span class="p">]:</span> <span class="k">assert</span> <span class="n">port</span> <span class="o">==</span> <span class="nb">int</span><span class="p">(</span><span class="n">kwargs</span><span class="p">[</span><span class="s">&#39;addr&#39;</span><span class="p">][</span><span class="o">-</span><span class="mi">4</span><span class="p">:])</span> <span class="k">else</span><span class="p">:</span> <span class="k">assert</span> <span class="n">port</span> <span class="o">==</span> <span class="n">sock_mock</span><span class="o">.</span><span class="n">bind_to_random_port</span><span class="o">.</span><span class="n">return_value</span> <span class="c"># Check that the socket was crated correctly</span> <span class="k">assert</span> <span class="n">zp</span><span class="o">.</span><span class="n">context</span><span class="o">.</span><span class="n">socket</span><span class="o">.</span><span class="n">call_args</span> <span class="o">==</span> <span class="p">((</span><span class="n">kwargs</span><span class="p">[</span><span class="s">&#39;sock_type&#39;</span><span class="p">],),</span> <span class="p">{})</span> <span class="k">if</span> <span class="n">kwargs</span><span class="p">[</span><span class="s">&#39;bind&#39;</span><span class="p">]</span> <span class="ow">and</span> <span class="s">&#39;:&#39;</span> <span class="ow">in</span> <span class="n">kwargs</span><span class="p">[</span><span class="s">&#39;addr&#39;</span><span class="p">]:</span> <span class="k">assert</span> <span class="n">sock_mock</span><span class="o">.</span><span class="n">bind</span><span class="o">.</span><span class="n">call_args</span> <span class="o">==</span> <span class="p">(</span> <span class="p">(</span><span class="s">&#39;tcp://</span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="n">kwargs</span><span class="p">[</span><span class="s">&#39;addr&#39;</span><span class="p">],),</span> <span class="p">{})</span> <span class="k">elif</span> <span class="n">kwargs</span><span class="p">[</span><span class="s">&#39;bind&#39;</span><span class="p">]:</span> <span class="k">assert</span> <span class="n">sock_mock</span><span class="o">.</span><span class="n">bind_to_random_port</span><span class="o">.</span><span class="n">call_args</span> <span class="o">==</span> <span class="p">(</span> <span class="p">(</span><span class="s">&#39;tcp://</span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="n">kwargs</span><span class="p">[</span><span class="s">&#39;addr&#39;</span><span class="p">],),</span> <span class="p">{})</span> <span class="k">else</span><span class="p">:</span> <span class="k">assert</span> <span class="n">sock_mock</span><span class="o">.</span><span class="n">connect</span><span class="o">.</span><span class="n">call_args</span> <span class="o">==</span> <span class="p">(</span> <span class="p">(</span><span class="s">&#39;tcp://</span><span class="si">%s</span><span class="s">:</span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="n">kwargs</span><span class="p">[</span><span class="s">&#39;addr&#39;</span><span class="p">],),</span> <span class="p">{})</span> <span class="c"># Check creation of the stream</span> <span class="k">assert</span> <span class="n">zmqstream_mock</span><span class="o">.</span><span class="n">call_args</span> <span class="o">==</span> <span class="p">((</span><span class="n">sock_mock</span><span class="p">,</span> <span class="n">zp</span><span class="o">.</span><span class="n">loop</span><span class="p">),</span> <span class="p">{})</span> <span class="k">assert</span> <span class="n">zmqstream_mock</span><span class="o">.</span><span class="n">return_value</span><span class="o">.</span><span class="n">on_recv</span><span class="o">.</span><span class="n">call_args</span> <span class="o">==</span> <span class="p">(</span> <span class="p">(</span><span class="n">kwargs</span><span class="p">[</span><span class="s">&#39;callback&#39;</span><span class="p">],),</span> <span class="p">{})</span> <span class="c"># Check default subscribtion</span> <span class="k">if</span> <span class="s">&#39;subscribe&#39;</span> <span class="ow">in</span> <span class="n">kwargs</span><span class="p">:</span> <span class="k">assert</span> <span class="n">sock_mock</span><span class="o">.</span><span class="n">setsockopt</span><span class="o">.</span><span class="n">call_args</span> <span class="o">==</span> <span class="p">(</span> <span class="p">(</span><span class="n">zmq</span><span class="o">.</span><span class="n">SUBSCRIBE</span><span class="p">,</span> <span class="n">kwargs</span><span class="p">[</span><span class="s">&#39;subscribe&#39;</span><span class="p">]),</span> <span class="p">{})</span> </pre></div> <p><em>Note: You may have noticed that I use</em> <tt class="docutils literal">assert my_mock.call_args == ...</tt> <em>rather than</em> <tt class="docutils literal"><span class="pre">my_mock.assert_called_with(...)</span></tt>. <em>The reason for that is simply, that assert statements are highlighted but ordinary function calls are not. This makes it easier for me to find all assertions in a&nbsp;test.</em></p> </div> <div class="section" id="messagehandler"> <h3>MessageHandler</h3> <p>The <em>MessageHandler</em> base class has only one methd, <em>__call__</em>, but I split the test for it into two methods—one that tests the <span class="caps">JSON</span>-loading functionality and one that checks if the correct handler method is&nbsp;called:</p> <div class="highlight"><pre><span class="c"># example_app/test/test_base.py</span> <span class="k">class</span> <span class="nc">TestMessageHandler</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;Tests for :class:`base.TestMessageHandler`.&quot;&quot;&quot;</span> <span class="nd">@pytest.mark.parametrize</span><span class="p">((</span><span class="s">&#39;idx&#39;</span><span class="p">,</span> <span class="s">&#39;msg&#39;</span><span class="p">),</span> <span class="p">[</span> <span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="p">[</span><span class="mi">23</span><span class="p">,</span> <span class="n">b</span><span class="s">&#39;[&quot;test&quot;, null]&#39;</span><span class="p">]),</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="p">[</span><span class="mi">23</span><span class="p">,</span> <span class="n">b</span><span class="s">&#39;[&quot;test&quot;, &quot;spam&quot;]&#39;</span><span class="p">,</span> <span class="mi">42</span><span class="p">]),</span> <span class="p">(</span><span class="ne">TypeError</span><span class="p">,</span> <span class="p">[</span><span class="mi">23</span><span class="p">,</span> <span class="mi">42</span><span class="p">]),</span> <span class="p">(</span><span class="ne">ValueError</span><span class="p">,</span> <span class="p">[</span><span class="mi">23</span><span class="p">,</span> <span class="n">b</span><span class="s">&#39;[&quot;test&quot;]23spam&#39;</span><span class="p">]),</span> <span class="p">])</span> <span class="k">def</span> <span class="nf">test_call_json_load</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">idx</span><span class="p">,</span> <span class="n">msg</span><span class="p">):</span> <span class="n">handler</span> <span class="o">=</span> <span class="n">mock</span><span class="o">.</span><span class="n">Mock</span><span class="p">()</span> <span class="n">mh</span> <span class="o">=</span> <span class="n">base</span><span class="o">.</span><span class="n">MessageHandler</span><span class="p">(</span><span class="n">idx</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">idx</span><span class="p">,</span> <span class="nb">int</span><span class="p">)</span> <span class="k">else</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="n">mh</span><span class="o">.</span><span class="n">test</span> <span class="o">=</span> <span class="n">handler</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">idx</span><span class="p">,</span> <span class="nb">int</span><span class="p">):</span> <span class="n">mh</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span> <span class="k">assert</span> <span class="n">handler</span><span class="o">.</span><span class="n">call_count</span> <span class="o">==</span> <span class="mi">1</span> <span class="k">else</span><span class="p">:</span> <span class="n">pytest</span><span class="o">.</span><span class="n">raises</span><span class="p">(</span><span class="n">idx</span><span class="p">,</span> <span class="n">mh</span><span class="p">,</span> <span class="n">msg</span><span class="p">)</span> <span class="nd">@pytest.mark.parametrize</span><span class="p">((</span><span class="s">&#39;ok&#39;</span><span class="p">,</span> <span class="s">&#39;msg&#39;</span><span class="p">),</span> <span class="p">[</span> <span class="p">(</span><span class="bp">True</span><span class="p">,</span> <span class="p">[</span><span class="mi">23</span><span class="p">,</span> <span class="n">b</span><span class="s">&#39;[&quot;test&quot;, &quot;spam&quot;]&#39;</span><span class="p">,</span> <span class="mi">42</span><span class="p">]),</span> <span class="p">(</span><span class="ne">AttributeError</span><span class="p">,</span> <span class="p">[</span><span class="mi">23</span><span class="p">,</span> <span class="n">b</span><span class="s">&#39;[&quot;_test&quot;, &quot;spam&quot;]&#39;</span><span class="p">,</span> <span class="mi">42</span><span class="p">]),</span> <span class="p">(</span><span class="ne">TypeError</span><span class="p">,</span> <span class="p">[</span><span class="mi">23</span><span class="p">,</span> <span class="n">b</span><span class="s">&#39;[&quot;spam&quot;, &quot;spam&quot;]&#39;</span><span class="p">,</span> <span class="mi">42</span><span class="p">]),</span> <span class="p">(</span><span class="ne">AttributeError</span><span class="p">,</span> <span class="p">[</span><span class="mi">23</span><span class="p">,</span> <span class="n">b</span><span class="s">&#39;[&quot;eggs&quot;, &quot;spam&quot;]&#39;</span><span class="p">,</span> <span class="mi">42</span><span class="p">]),</span> <span class="p">])</span> <span class="k">def</span> <span class="nf">test_call_get_handler</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">ok</span><span class="p">,</span> <span class="n">msg</span><span class="p">):</span> <span class="n">handler</span> <span class="o">=</span> <span class="n">mock</span><span class="o">.</span><span class="n">Mock</span><span class="p">()</span> <span class="n">mh</span> <span class="o">=</span> <span class="n">base</span><span class="o">.</span><span class="n">MessageHandler</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="n">mh</span><span class="o">.</span><span class="n">test</span> <span class="o">=</span> <span class="n">handler</span> <span class="n">mh</span><span class="o">.</span><span class="n">spam</span> <span class="o">=</span> <span class="s">&#39;spam&#39;</span> <span class="k">if</span> <span class="n">ok</span> <span class="ow">is</span> <span class="bp">True</span><span class="p">:</span> <span class="n">mh</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span> <span class="k">assert</span> <span class="n">handler</span><span class="o">.</span><span class="n">call_args</span> <span class="o">==</span> <span class="p">(</span> <span class="p">(</span><span class="n">msg</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="s">&#39;spam&#39;</span><span class="p">,</span> <span class="n">msg</span><span class="p">[</span><span class="mi">2</span><span class="p">]),</span> <span class="p">{})</span> <span class="k">else</span><span class="p">:</span> <span class="n">pytest</span><span class="o">.</span><span class="n">raises</span><span class="p">(</span><span class="n">ok</span><span class="p">,</span> <span class="n">mh</span><span class="p">,</span> <span class="n">msg</span><span class="p">)</span> </pre></div> </div> <div class="section" id="pongproc"> <h3>PongProc</h3> <p>Testing the <em>PongProc</em> is not much different from testing its base class. <em>pytest_funcarg__pp</em> will instantiate a <em>PongProc</em> instance for each test that has a <tt class="docutils literal">pp</tt> argument. The tests for <em>setup</em>, <em>run</em> and <em>stop</em> are easy to do. We create a few mocks and then ask them if the tested function called them&nbsp;correctly:</p> <div class="highlight"><pre><span class="c"># example_app/test/test_pongproc.py</span> <span class="kn">from</span> <span class="nn">zmq.utils</span> <span class="kn">import</span> <span class="n">jsonapi</span> <span class="k">as</span> <span class="n">json</span> <span class="kn">import</span> <span class="nn">mock</span><span class="o">,</span> <span class="nn">pytest</span><span class="o">,</span> <span class="nn">zmq</span> <span class="kn">import</span> <span class="nn">pongproc</span> <span class="n">host</span><span class="p">,</span> <span class="n">port</span> <span class="o">=</span> <span class="s">&#39;127.0.0.1&#39;</span><span class="p">,</span> <span class="mi">5678</span> <span class="k">def</span> <span class="nf">pytest_funcarg__pp</span><span class="p">(</span><span class="n">request</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;Creates a PongProc instance.&quot;&quot;&quot;</span> <span class="k">return</span> <span class="n">pongproc</span><span class="o">.</span><span class="n">PongProc</span><span class="p">((</span><span class="n">host</span><span class="p">,</span> <span class="n">port</span><span class="p">))</span> <span class="k">class</span> <span class="nc">TestPongProc</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;Tests :class:`pongproc.PongProc`.&quot;&quot;&quot;</span> <span class="k">def</span> <span class="nf">test_setup</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">pp</span><span class="p">):</span> <span class="k">def</span> <span class="nf">make_stream</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> <span class="n">stream</span> <span class="o">=</span> <span class="n">mock</span><span class="o">.</span><span class="n">Mock</span><span class="p">()</span> <span class="n">stream</span><span class="o">.</span><span class="n">type</span> <span class="o">=</span> <span class="n">args</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">return</span> <span class="n">stream</span><span class="p">,</span> <span class="n">mock</span><span class="o">.</span><span class="n">Mock</span><span class="p">()</span> <span class="n">pp</span><span class="o">.</span><span class="n">stream</span> <span class="o">=</span> <span class="n">mock</span><span class="o">.</span><span class="n">Mock</span><span class="p">(</span><span class="n">side_effect</span><span class="o">=</span><span class="n">make_stream</span><span class="p">)</span> <span class="k">with</span> <span class="n">mock</span><span class="o">.</span><span class="n">patch</span><span class="p">(</span><span class="s">&#39;base.ZmqProcess.setup&#39;</span><span class="p">)</span> <span class="k">as</span> <span class="n">setup_mock</span><span class="p">:</span> <span class="n">pp</span><span class="o">.</span><span class="n">setup</span><span class="p">()</span> <span class="k">assert</span> <span class="n">setup_mock</span><span class="o">.</span><span class="n">call_count</span> <span class="o">==</span> <span class="mi">1</span> <span class="k">assert</span> <span class="n">pp</span><span class="o">.</span><span class="n">stream</span><span class="o">.</span><span class="n">call_args_list</span> <span class="o">==</span> <span class="p">[</span> <span class="p">((</span><span class="n">zmq</span><span class="o">.</span><span class="n">REP</span><span class="p">,</span> <span class="p">(</span><span class="n">host</span><span class="p">,</span> <span class="n">port</span><span class="p">)),</span> <span class="nb">dict</span><span class="p">(</span><span class="n">bind</span><span class="o">=</span><span class="bp">True</span><span class="p">)),</span> <span class="p">]</span> <span class="k">assert</span> <span class="n">pp</span><span class="o">.</span><span class="n">rep_stream</span><span class="o">.</span><span class="n">type</span> <span class="o">==</span> <span class="n">zmq</span><span class="o">.</span><span class="n">REP</span> <span class="c"># Test if the message handler was configured correctly</span> <span class="n">rsh</span> <span class="o">=</span> <span class="n">pp</span><span class="o">.</span><span class="n">rep_stream</span><span class="o">.</span><span class="n">on_recv</span><span class="o">.</span><span class="n">call_args</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span> <span class="c"># Get the msg handler</span> <span class="k">assert</span> <span class="n">rsh</span><span class="o">.</span><span class="n">_rep_stream</span> <span class="o">==</span> <span class="n">pp</span><span class="o">.</span><span class="n">rep_stream</span> <span class="k">assert</span> <span class="n">rsh</span><span class="o">.</span><span class="n">_stop</span> <span class="o">==</span> <span class="n">pp</span><span class="o">.</span><span class="n">stop</span> <span class="k">def</span> <span class="nf">test_run</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">pp</span><span class="p">):</span> <span class="n">pp</span><span class="o">.</span><span class="n">setup</span> <span class="o">=</span> <span class="n">mock</span><span class="o">.</span><span class="n">Mock</span><span class="p">()</span> <span class="n">pp</span><span class="o">.</span><span class="n">loop</span> <span class="o">=</span> <span class="n">mock</span><span class="o">.</span><span class="n">Mock</span><span class="p">()</span> <span class="n">pp</span><span class="o">.</span><span class="n">run</span><span class="p">()</span> <span class="k">assert</span> <span class="n">pp</span><span class="o">.</span><span class="n">setup</span><span class="o">.</span><span class="n">call_count</span> <span class="o">==</span> <span class="mi">1</span> <span class="k">assert</span> <span class="n">pp</span><span class="o">.</span><span class="n">loop</span><span class="o">.</span><span class="n">start</span><span class="o">.</span><span class="n">call_count</span> <span class="o">==</span> <span class="mi">1</span> <span class="k">def</span> <span class="nf">test_stop</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">pp</span><span class="p">):</span> <span class="n">pp</span><span class="o">.</span><span class="n">loop</span> <span class="o">=</span> <span class="n">mock</span><span class="o">.</span><span class="n">Mock</span><span class="p">()</span> <span class="n">pp</span><span class="o">.</span><span class="n">stop</span><span class="p">()</span> <span class="k">assert</span> <span class="n">pp</span><span class="o">.</span><span class="n">loop</span><span class="o">.</span><span class="n">stop</span><span class="o">.</span><span class="n">call_count</span> <span class="o">==</span> <span class="mi">1</span> </pre></div> </div> <div class="section" id="repstreamhandler"> <h3>RepStreamHandler</h3> <p>Testing the actual message handler requires some mocks, but is apart from that straight forward. A <em>funcarg</em> method creates an instance of the message handler for each test case which we feed with a message. We than check if the application logic was called correctly and/or if a correct reply is&nbsp;sent:</p> <div class="highlight"><pre><span class="c"># example_app/test/test_pongproc.py</span> <span class="k">def</span> <span class="nf">pytest_funcarg__rsh</span><span class="p">(</span><span class="n">request</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;Creates a RepStreamHandler instance.&quot;&quot;&quot;</span> <span class="k">return</span> <span class="n">pongproc</span><span class="o">.</span><span class="n">RepStreamHandler</span><span class="p">(</span> <span class="n">rep_stream</span><span class="o">=</span><span class="n">mock</span><span class="o">.</span><span class="n">Mock</span><span class="p">(),</span> <span class="n">stop</span><span class="o">=</span><span class="n">mock</span><span class="o">.</span><span class="n">Mock</span><span class="p">(),</span> <span class="n">ping_handler</span><span class="o">=</span><span class="n">mock</span><span class="o">.</span><span class="n">Mock</span><span class="p">(</span><span class="n">spec_set</span><span class="o">=</span><span class="n">pongproc</span><span class="o">.</span><span class="n">PingHandler</span><span class="p">()))</span> <span class="k">class</span> <span class="nc">TestRepStreamHandler</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span> <span class="k">def</span> <span class="nf">test_ping</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">rsh</span><span class="p">):</span> <span class="n">msg</span> <span class="o">=</span> <span class="p">[</span><span class="s">&#39;ping&#39;</span><span class="p">,</span> <span class="mi">1</span><span class="p">]</span> <span class="n">retval</span> <span class="o">=</span> <span class="s">&#39;spam&#39;</span> <span class="n">rsh</span><span class="o">.</span><span class="n">_ping_handler</span> <span class="o">=</span> <span class="n">mock</span><span class="o">.</span><span class="n">Mock</span><span class="p">(</span><span class="n">spec_set</span><span class="o">=</span><span class="n">pongproc</span><span class="o">.</span><span class="n">PingHandler</span><span class="p">)</span> <span class="n">rsh</span><span class="o">.</span><span class="n">_ping_handler</span><span class="o">.</span><span class="n">make_pong</span><span class="o">.</span><span class="n">return_value</span> <span class="o">=</span> <span class="n">retval</span> <span class="n">rsh</span><span class="p">([</span><span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">msg</span><span class="p">)])</span> <span class="k">assert</span> <span class="n">rsh</span><span class="o">.</span><span class="n">_ping_handler</span><span class="o">.</span><span class="n">make_pong</span><span class="o">.</span><span class="n">call_args</span> <span class="o">==</span> <span class="p">((</span><span class="n">msg</span><span class="p">[</span><span class="mi">1</span><span class="p">],),</span> <span class="p">{})</span> <span class="k">assert</span> <span class="n">rsh</span><span class="o">.</span><span class="n">_rep_stream</span><span class="o">.</span><span class="n">send_json</span><span class="o">.</span><span class="n">call_args</span> <span class="o">==</span> <span class="p">((</span><span class="n">retval</span><span class="p">,),</span> <span class="p">{})</span> <span class="k">def</span> <span class="nf">test_plzdiekthybye</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">rsh</span><span class="p">):</span> <span class="n">rsh</span><span class="p">([</span><span class="n">b</span><span class="s">&#39;[&quot;plzdiekthxbye&quot;, null]&#39;</span><span class="p">])</span> <span class="k">assert</span> <span class="n">rsh</span><span class="o">.</span><span class="n">_stop</span><span class="o">.</span><span class="n">call_count</span> <span class="o">==</span> <span class="mi">1</span> </pre></div> </div> <div class="section" id="pinghandler"> <h3>PingHandler</h3> <p>When we are done with all that network stuff, we can finally test the application logic. Easy-peasy in our&nbsp;case:</p> <div class="highlight"><pre><span class="c"># example_app/test/test_pongproc.py</span> <span class="k">def</span> <span class="nf">pytest_funcarg__ph</span><span class="p">(</span><span class="n">request</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;Creates a PingHandler instance.&quot;&quot;&quot;</span> <span class="k">return</span> <span class="n">pongproc</span><span class="o">.</span><span class="n">PingHandler</span><span class="p">()</span> <span class="k">class</span> <span class="nc">TestPingHandler</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span> <span class="k">def</span> <span class="nf">test_make_pong</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">ph</span><span class="p">):</span> <span class="n">ping_num</span> <span class="o">=</span> <span class="mi">23</span> <span class="n">ret</span> <span class="o">=</span> <span class="n">ph</span><span class="o">.</span><span class="n">make_pong</span><span class="p">(</span><span class="n">ping_num</span><span class="p">)</span> <span class="k">assert</span> <span class="n">ret</span> <span class="o">==</span> <span class="p">[</span><span class="s">&#39;pong&#39;</span><span class="p">,</span> <span class="n">ping_num</span><span class="p">]</span> </pre></div> </div> </div> <div class="section" id="summary"> <h2>Summary</h2> <p>Thanks to the <a class="reference external" href="http://www.voidspace.org.uk/python/mock/">Mock</a> library, unit testing PyZMQ apps is really not that hard and not much different from normal unit testing. However, what we know now is only, that our process should work <em>in theory</em>. We haven’t yet started it and sent real messages to&nbsp;it.</p> <p>The <a class="reference external" href="/2012/02/15/designing-and-testing-pyzmq-applications-part-3/">next and final part</a> of this series will show you how you can automate testing complete processes. Until then, you should get your <a class="reference external" href="http://pypi.python.org/pypi/pytest-cov">test coverage</a> up to 100% to protect yourself from nasty surprises when you start with process&nbsp;testing.</p> </div> Designing and Testing PyZMQ Applications – Part 12012-02-01T08:05:33+01:00Stefan Scherfketag:stefan.sofa-rockers.org,2012-02-01:2012/02/01/designing-and-testing-pyzmq-applications-part-1/<p><a class="reference external" href="http://www.zeromq.org/">ZeroMQ</a> (or ØMQ or <span class="caps">ZMQ</span>) is an intelligent messaging framework and <a class="reference external" href="http://zguide.zeromq.org/page:all#Fixing-the-World">described</a> as “sockets on steroids”. That is, they look like normal <span class="caps">TCP</span> sockets but actually work as you’d expect sockets to work. <a class="reference external" href="http://www.zeromq.org/bindings:python">PyZMQ</a> adds even more convenience to them, which makes it a really a good choice if you want to implement a distributed application. Another big plus for ØMQ is that you can integrate sub-systems written in C, Java or any other language ØMQ supports (which are a&nbsp;lot).</p> <p>If you’ve never heard of ØMQ before, I recommend to read <a class="reference external" href="http://nichol.as/zeromq-an-introduction">ZeroMQ an Introduction</a> by Nicholas Piël, before you go on with this&nbsp;article.</p> <p>The <a class="reference external" href="http://zguide.zeromq.org/page:all">ØMQ Guide</a> and <a class="reference external" href="http://zeromq.github.com/pyzmq/">PyZMQ’s documentation</a> are really good, so you can easily get started. However, when we began to implement a larger application with it (a distributed simulation framework), several questions arose which were not covered by the&nbsp;documentation:</p> <ul class="simple"> <li>What’s the best way do design our&nbsp;application?</li> <li>How can we keep it readable, flexible and&nbsp;maintainable?</li> <li>How do we test&nbsp;it?</li> </ul> <p>I didn’t find something like a best practice article that answered my questions. So in this series of articles, I’m going to talk about what I’ve learned during the last months. I’m not a PyZMQ expert (yet ;-)), but what I’ve done so far works quite well and I never had more tests in a project than I do have&nbsp;now.</p> <p>You’ll find the source for the examples at <a class="reference external" href="https://bitbucket.org/ssc/pyzmq-article">bitbucket</a>. They are written in Python 3.2 and tested under Mac <span class="caps">OS</span> X Lion, Ubuntu 11.10 and Windows 7, 64 bit in each case. If you have any suggestions or improvements, please fork me or just leave a <a class="reference external" href="#comments">comment</a>.</p> <p>In this first article, I’m going to talk a bit about how you could generally design your application to be flexible, maintainable and testable. The second part will be about unit testing and the finally, I’ll cover process and system&nbsp;testing.</p> <div class="section" id="comparison-of-different-approaches"> <h2>Comparison of Different&nbsp;Approaches</h2> <p>There are basically three possible ways to implement a PyZMQ application. One, that’s easy, but limited in practical use, one that’s more flexible, but not really pythonic and one, that needs a bit more setup, but is flexible and&nbsp;pythonic.</p> <p>All three examples feature a simple ping process and a pong process with varying complexity. I use multiprocessing to run the pong process, because that’s what you should usually do in real PyZMQ applications (you don’t want to use threads and if both processes are running on the same machine, there’s no need to invoke both of them&nbsp;separately).</p> <p>All of the examples will have the following&nbsp;output:</p> <div class="highlight"><pre><span class="o">(</span>zmq<span class="o">)</span><span class="nv">$ </span>python blocking_recv.py Pong got request: ping 0 Ping got reply: pong 0 ... Pong got request: ping 4 Ping got reply: pong 4 </pre></div> <p>Let’s start with the easy one first. You just use on of the socket’s recv methods in a&nbsp;loop:</p> <div class="highlight"><pre><span class="c"># blocking_recv.py</span> <span class="kn">import</span> <span class="nn">multiprocessing</span> <span class="kn">import</span> <span class="nn">zmq</span> <span class="n">addr</span> <span class="o">=</span> <span class="s">&#39;tcp://127.0.0.1:5678&#39;</span> <span class="k">def</span> <span class="nf">ping</span><span class="p">():</span> <span class="sd">&quot;&quot;&quot;Sends ping requests and waits for replies.&quot;&quot;&quot;</span> <span class="n">context</span> <span class="o">=</span> <span class="n">zmq</span><span class="o">.</span><span class="n">Context</span><span class="p">()</span> <span class="n">sock</span> <span class="o">=</span> <span class="n">context</span><span class="o">.</span><span class="n">socket</span><span class="p">(</span><span class="n">zmq</span><span class="o">.</span><span class="n">REQ</span><span class="p">)</span> <span class="n">sock</span><span class="o">.</span><span class="n">bind</span><span class="p">(</span><span class="n">addr</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">):</span> <span class="n">sock</span><span class="o">.</span><span class="n">send_unicode</span><span class="p">(</span><span class="s">&#39;ping </span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="n">i</span><span class="p">)</span> <span class="n">rep</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">recv_unicode</span><span class="p">()</span> <span class="c"># This blocks until we get something</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;Ping got reply:&#39;</span><span class="p">,</span> <span class="n">rep</span><span class="p">)</span> <span class="k">def</span> <span class="nf">pong</span><span class="p">():</span> <span class="sd">&quot;&quot;&quot;Waits for ping requests and replies with a pong.&quot;&quot;&quot;</span> <span class="n">context</span> <span class="o">=</span> <span class="n">zmq</span><span class="o">.</span><span class="n">Context</span><span class="p">()</span> <span class="n">sock</span> <span class="o">=</span> <span class="n">context</span><span class="o">.</span><span class="n">socket</span><span class="p">(</span><span class="n">zmq</span><span class="o">.</span><span class="n">REP</span><span class="p">)</span> <span class="n">sock</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">addr</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">):</span> <span class="n">req</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">recv_unicode</span><span class="p">()</span> <span class="c"># This also blocks</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;Pong got request:&#39;</span><span class="p">,</span> <span class="n">req</span><span class="p">)</span> <span class="n">sock</span><span class="o">.</span><span class="n">send_unicode</span><span class="p">(</span><span class="s">&#39;pong </span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="n">i</span><span class="p">)</span> <span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">&#39;__main__&#39;</span><span class="p">:</span> <span class="n">pong_proc</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">pong</span><span class="p">)</span> <span class="n">pong_proc</span><span class="o">.</span><span class="n">start</span><span class="p">()</span> <span class="n">ping</span><span class="p">()</span> <span class="n">pong_proc</span><span class="o">.</span><span class="n">join</span><span class="p">()</span> </pre></div> <p>So this is very easy and no that much code. The problem with this is, that it only works well if your process only uses one socket. Unfortunately, in larger applications that is rather rarely the&nbsp;case.</p> <p>A way to handle multiple sockets per process is polling. In addition to your context and socket(s), you need a <em>poller</em>. You also have to tell it which events on which socket you are going to&nbsp;poll:</p> <div class="highlight"><pre><span class="c"># polling.py</span> <span class="k">def</span> <span class="nf">pong</span><span class="p">():</span> <span class="sd">&quot;&quot;&quot;Waits for ping requests and replies with a pong.&quot;&quot;&quot;</span> <span class="n">context</span> <span class="o">=</span> <span class="n">zmq</span><span class="o">.</span><span class="n">Context</span><span class="p">()</span> <span class="n">sock</span> <span class="o">=</span> <span class="n">context</span><span class="o">.</span><span class="n">socket</span><span class="p">(</span><span class="n">zmq</span><span class="o">.</span><span class="n">REP</span><span class="p">)</span> <span class="n">sock</span><span class="o">.</span><span class="n">bind</span><span class="p">(</span><span class="n">addr</span><span class="p">)</span> <span class="c"># Create a poller and register the events we want to poll</span> <span class="n">poller</span> <span class="o">=</span> <span class="n">zmq</span><span class="o">.</span><span class="n">Poller</span><span class="p">()</span> <span class="n">poller</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">sock</span><span class="p">,</span> <span class="n">zmq</span><span class="o">.</span><span class="n">POLLIN</span><span class="o">|</span><span class="n">zmq</span><span class="o">.</span><span class="n">POLLOUT</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">):</span> <span class="c"># Get all sockets that can do something</span> <span class="n">socks</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span><span class="n">poller</span><span class="o">.</span><span class="n">poll</span><span class="p">())</span> <span class="c"># Check if we can receive something</span> <span class="k">if</span> <span class="n">sock</span> <span class="ow">in</span> <span class="n">socks</span> <span class="ow">and</span> <span class="n">socks</span><span class="p">[</span><span class="n">sock</span><span class="p">]</span> <span class="o">==</span> <span class="n">zmq</span><span class="o">.</span><span class="n">POLLIN</span><span class="p">:</span> <span class="n">req</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">recv_unicode</span><span class="p">()</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;Pong got request:&#39;</span><span class="p">,</span> <span class="n">req</span><span class="p">)</span> <span class="c"># Check if we cann send something</span> <span class="k">if</span> <span class="n">sock</span> <span class="ow">in</span> <span class="n">socks</span> <span class="ow">and</span> <span class="n">socks</span><span class="p">[</span><span class="n">sock</span><span class="p">]</span> <span class="o">==</span> <span class="n">zmq</span><span class="o">.</span><span class="n">POLLOUT</span><span class="p">:</span> <span class="n">sock</span><span class="o">.</span><span class="n">send_unicode</span><span class="p">(</span><span class="s">&#39;pong </span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">i</span> <span class="o">//</span> <span class="mi">2</span><span class="p">))</span> <span class="n">poller</span><span class="o">.</span><span class="n">unregister</span><span class="p">(</span><span class="n">sock</span><span class="p">)</span> </pre></div> <p>You see, that our <em>pong</em> function got pretty ugly. You need 10 iterations to do five ping-pongs, because in each iteration you can either send or reply. And each socket you add to your process adds two more if-statements. You could improve that design if you created a base class wrapping the polling loop and just register sockets and callbacks in an inheriting&nbsp;class.</p> <p>That brings us to our final example. PyZMQ comes with with an <a class="reference external" href="http://zeromq.github.com/pyzmq/eventloop.html">adapted Tornado eventloop</a> that handles the polling and works with <a class="reference external" href="http://zeromq.github.com/pyzmq/api/generated/zmq.eventloop.zmqstream.html#zmq.eventloop.zmqstream.ZMQStream">ZMQStreams</a>, that wrap sockets and add some&nbsp;functionality:</p> <div class="highlight"><pre><span class="c"># eventloop.py</span> <span class="kn">from</span> <span class="nn">zmq.eventloop</span> <span class="kn">import</span> <span class="n">ioloop</span><span class="p">,</span> <span class="n">zmqstream</span> <span class="k">class</span> <span class="nc">Pong</span><span class="p">(</span><span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;Waits for ping requests and replies with a pong.&quot;&quot;&quot;</span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">__init__</span><span class="p">()</span> <span class="bp">self</span><span class="o">.</span><span class="n">loop</span> <span class="o">=</span> <span class="bp">None</span> <span class="bp">self</span><span class="o">.</span><span class="n">stream</span> <span class="o">=</span> <span class="bp">None</span> <span class="bp">self</span><span class="o">.</span><span class="n">i</span> <span class="o">=</span> <span class="mi">0</span> <span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Initializes the event loop, creates the sockets/streams and</span> <span class="sd"> starts the (blocking) loop.</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="n">context</span> <span class="o">=</span> <span class="n">zmq</span><span class="o">.</span><span class="n">Context</span><span class="p">()</span> <span class="bp">self</span><span class="o">.</span><span class="n">loop</span> <span class="o">=</span> <span class="n">ioloop</span><span class="o">.</span><span class="n">IOLoop</span><span class="o">.</span><span class="n">instance</span><span class="p">()</span> <span class="c"># This is the event loop</span> <span class="n">sock</span> <span class="o">=</span> <span class="n">context</span><span class="o">.</span><span class="n">socket</span><span class="p">(</span><span class="n">zmq</span><span class="o">.</span><span class="n">REP</span><span class="p">)</span> <span class="n">sock</span><span class="o">.</span><span class="n">bind</span><span class="p">(</span><span class="n">addr</span><span class="p">)</span> <span class="c"># We need to create a stream from our socket and</span> <span class="c"># register a callback for recv events.</span> <span class="bp">self</span><span class="o">.</span><span class="n">stream</span> <span class="o">=</span> <span class="n">zmqstream</span><span class="o">.</span><span class="n">ZMQStream</span><span class="p">(</span><span class="n">sock</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">loop</span><span class="p">)</span> <span class="bp">self</span><span class="o">.</span><span class="n">stream</span><span class="o">.</span><span class="n">on_recv</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">handle_ping</span><span class="p">)</span> <span class="c"># Start the loop. It runs until we stop it.</span> <span class="bp">self</span><span class="o">.</span><span class="n">loop</span><span class="o">.</span><span class="n">start</span><span class="p">()</span> <span class="k">def</span> <span class="nf">handle_ping</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">msg</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;Handles ping requests and sends back a pong.&quot;&quot;&quot;</span> <span class="c"># req is a list of byte objects</span> <span class="n">req</span> <span class="o">=</span> <span class="n">msg</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">decode</span><span class="p">()</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;Pong got request:&#39;</span><span class="p">,</span> <span class="n">req</span><span class="p">)</span> <span class="bp">self</span><span class="o">.</span><span class="n">stream</span><span class="o">.</span><span class="n">send_unicode</span><span class="p">(</span><span class="s">&#39;pong </span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="bp">self</span><span class="o">.</span><span class="n">i</span><span class="p">)</span> <span class="c"># We’ll stop the loop after 5 pings</span> <span class="bp">self</span><span class="o">.</span><span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span> <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">i</span> <span class="o">==</span> <span class="mi">5</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">stream</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span> <span class="bp">self</span><span class="o">.</span><span class="n">loop</span><span class="o">.</span><span class="n">stop</span><span class="p">()</span> </pre></div> <p>This even adds more boilerplate code, but it will pay of if you use more sockets and most of that stuff in <em>run()</em> can be put into a base class. Another drawback is, that the IOLoop only uses <a class="reference external" href="http://zeromq.github.com/pyzmq/api/generated/zmq.core.socket.html#zmq.core.socket.Socket.recv_multipart">recv_multipart()</a>. So you always get a lists of byte strings which you have to decode or deserialize on your own. However, you can use all the <em>send</em> methods socket offers (like <em>send_unicode()</em> or <em>send_json()</em>). You can also stop the loop from within a message&nbsp;handler.</p> <p>In the next sections, I’ll discuss how you could implement a PyZMQ process that uses the event&nbsp;loop.</p> </div> <div class="section" id="communication-design"> <h2>Communication&nbsp;Design</h2> <p>Before you start to implement anything, you should think about what kind of processes you need in your application and which messages they exchange. You should also decide what kind of message format and serialization you want to&nbsp;use.</p> <p>PyZMQ has built-in support for Unicode (<em>send</em> sends plain C strings which map to Python <em>byte</em> objects, so there’s a separate method to send Unicode strings), <span class="caps">JSON</span> and&nbsp;Pickle.</p> <p><span class="caps">JSON</span> is nice, because it’s fast and lets you integrate processes written in other languages into you application. It’s also a bit safer, because you cannot receive arbitrary objects as with <a class="reference external" href="http://docs.python.org/py3k/library/pickle#restricting-globals">pickle</a>. The most straightforward syntax for <span class="caps">JSON</span> messages is to let them be triples <em>[msg_type, args, kwargs]</em>, where <em>msg_type</em> maps to a method name and <em>args</em> and <em>kwargs</em> get passed as positional and keyword&nbsp;arguments.</p> <p>I strongly recommend you to document each chain of messages your application sends to perform a certain task. I do this with fancy PowerPoint graphics and with even fancier <span class="caps">ASCII</span> art in <a class="reference external" href="http://sphinx.pocoo.org">Sphinx</a>. Here is how I would document our&nbsp;ping-pong:</p> <div class="highlight"><pre><span class="gh">Sending pings</span> <span class="gh">-------------</span> <span class="m">*</span> If the ping process sends a <span class="ge">*ping*</span>, the pong processes responds with a <span class="ge">*pong*</span>. <span class="m">*</span> The number of pings (and pongs) is counted. The current ping count is sent with each message. <span class="se">::</span> <span class="s"> PingProc PongProc</span> <span class="s"> [REQ] ---1--&gt; [REP]</span> <span class="s"> &lt;--2---</span> <span class="s"> 1 IN : [&#39;ping, count&#39;]</span> <span class="s"> 1 OUT: [&#39;ping, count&#39;]</span> <span class="s"> 2 IN : [&#39;pong, count&#39;]</span> <span class="s"> 2 OUT: [&#39;pong, count&#39;]</span> </pre></div> <p>First, I write some bullet points that explain how the processes behave and why they behave this way. This is followed by some kind of sequence diagram that shows when which process sents which message using which socket type. Finally, I write down how the messages are looking. <tt class="docutils literal"># <span class="caps">IN</span></tt> is what you would pass to <em>send_multipart</em> and <tt class="docutils literal"># <span class="caps">OUT</span></tt> is, what is received on the other side by <em>recv_multipart</em>. If one of the participating sockets is a <em><span class="caps">ROUTER</span></em> or <em><span class="caps">DEALER</span></em>, <em><span class="caps">IN</span></em> and <em><span class="caps">OUT</span></em> will differ (though that’s not the case in this example). Everything in single quotation marks (<tt class="docutils literal">'</tt>) represents a <span class="caps">JSON</span> serialized&nbsp;list.</p> <p>If our pong process used a <em><span class="caps">ROUTER</span></em> socket instead of the <em><span class="caps">REP</span></em> socket, it would look like&nbsp;this:</p> <pre class="literal-block"> 1 IN : ['ping, count'] 1 OUT: [ping_uuid, '', 'ping, count'] 2 IN : [ping_uuid, '', 'pong, count'] 2 OUT: ['pong, count'] </pre> <p>This seems like a lot of tedious work, but trust me, it really helps a lot when you need to change something a few weeks&nbsp;later!</p> </div> <div class="section" id="application-design"> <h2>Application&nbsp;Design</h2> <p>In the examples above, the <em>Pong</em> process was responsible for setting everything up, for receiving/sending messages and for the actual application logic (counting incoming pings and creating a&nbsp;pong).</p> <p>Obviously, this is not a very good design (at least if your application gets more complex than our little ping-pong example). What we can do about this is to put most of that nasty setup stuff into a base class which all your processes can inherit from, separate message handling and (de)serialization from it and finally put all the actual application logic into a separate (PyZMQ-independent) class. This will result in a three-level&nbsp;architecture:</p> <ol class="arabic simple"> <li>The lowest tier will contain the entry point of the process, set-up everything and start the event loop. A common base class provides utilities for creating sockets/streams and setting everything&nbsp;up.</li> <li>The second level is message handling and (de) serialization. A base class performs the (de)serialization and error handling. A message handler inherits this class and implements a method for each message type that should be&nbsp;handled.</li> <li>The third level will be the application logic and completely&nbsp;PyZMQ-agnostic.</li> </ol> <p>Base classes should be defined for the first two tiers two reduce redundant code in multiple processes or message handlers. The following figure shows the five classes our process is going to consist&nbsp;of:</p> <div class="figure align-center"> <img alt="The architecture of our pong process." src="/images/pongproc_architecture.png" style="width: 450px;" /> <p class="caption">The refactored PongProc now consists of three layers. The main class <em>PongProc</em> inherits <em>ZMQProcess</em>. Every stream gets a <em>MessageHandler</em>. In our example it’s just <em>RepStreamHandler</em>. Finally, you can have one ore more classes containing the (PyZMQ-agnostic) application logic. In our example, it’s called <em>PingHandler</em>, because it handles incoming&nbsp;pings.</p> </div> <div class="section" id="zmqpocess-the-base-class-for-all-processes"> <h3>ZmqPocess – The Base Class for all&nbsp;Processes</h3> <p>The base class basically implements two&nbsp;things:</p> <ul class="simple"> <li>a <em>setup</em> method that creates a context an a&nbsp;loop</li> <li>a <em>stream</em> factory method for streams with a <em>on_recv</em> callback. It creates a socket and can connect/bind it to a given address or bind it to a random port (that’s why it returns the port number in addition to the stream&nbsp;itself).</li> </ul> <p>It also inherits <a class="reference external" href="http://docs.python.org/py3k/library/multiprocessing#process-and-exceptions">multiprocessing.Process</a> so that it is easier to spawn it as sub-process. Of course, you can also just call its <em>run()</em> method from you <em>main()</em>.</p> <div class="highlight"><pre><span class="c"># example_app/base.py</span> <span class="kn">import</span> <span class="nn">multiprocessing</span> <span class="kn">from</span> <span class="nn">zmq.eventloop</span> <span class="kn">import</span> <span class="n">ioloop</span><span class="p">,</span> <span class="n">zmqstream</span> <span class="kn">import</span> <span class="nn">zmq</span> <span class="k">class</span> <span class="nc">ZmqProcess</span><span class="p">(</span><span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> This is the base for all processes and offers utility functions</span> <span class="sd"> for setup and creating new streams.</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">__init__</span><span class="p">()</span> <span class="bp">self</span><span class="o">.</span><span class="n">context</span> <span class="o">=</span> <span class="bp">None</span> <span class="sd">&quot;&quot;&quot;The ØMQ :class:`~zmq.Context` instance.&quot;&quot;&quot;</span> <span class="bp">self</span><span class="o">.</span><span class="n">loop</span> <span class="o">=</span> <span class="bp">None</span> <span class="sd">&quot;&quot;&quot;PyZMQ&#39;s event loop (:class:`~zmq.eventloop.ioloop.IOLoop`).&quot;&quot;&quot;</span> <span class="k">def</span> <span class="nf">setup</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Creates a :attr:`context` and an event :attr:`loop` for the process.</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="bp">self</span><span class="o">.</span><span class="n">context</span> <span class="o">=</span> <span class="n">zmq</span><span class="o">.</span><span class="n">Context</span><span class="p">()</span> <span class="bp">self</span><span class="o">.</span><span class="n">loop</span> <span class="o">=</span> <span class="n">ioloop</span><span class="o">.</span><span class="n">IOLoop</span><span class="o">.</span><span class="n">instance</span><span class="p">()</span> <span class="k">def</span> <span class="nf">stream</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">sock_type</span><span class="p">,</span> <span class="n">addr</span><span class="p">,</span> <span class="n">bind</span><span class="p">,</span> <span class="n">callback</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">subscribe</span><span class="o">=</span><span class="n">b</span><span class="s">&#39;&#39;</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Creates a :class:`~zmq.eventloop.zmqstream.ZMQStream`.</span> <span class="sd"> :param sock_type: The ØMQ socket type (e.g. ``zmq.REQ``)</span> <span class="sd"> :param addr: Address to bind or connect to formatted as *host:port*,</span> <span class="sd"> *(host, port)* or *host* (bind to random port).</span> <span class="sd"> If *bind* is ``True``, *host* may be:</span> <span class="sd"> - the wild-card ``*``, meaning all available interfaces,</span> <span class="sd"> - the primary IPv4 address assigned to the interface, in its</span> <span class="sd"> numeric representation or</span> <span class="sd"> - the interface name as defined by the operating system.</span> <span class="sd"> If *bind* is ``False``, *host* may be:</span> <span class="sd"> - the DNS name of the peer or</span> <span class="sd"> - the IPv4 address of the peer, in its numeric representation.</span> <span class="sd"> If *addr* is just a host name without a port and *bind* is</span> <span class="sd"> ``True``, the socket will be bound to a random port.</span> <span class="sd"> :param bind: Binds to *addr* if ``True`` or tries to connect to it</span> <span class="sd"> otherwise.</span> <span class="sd"> :param callback: A callback for</span> <span class="sd"> :meth:`~zmq.eventloop.zmqstream.ZMQStream.on_recv`, optional</span> <span class="sd"> :param subscribe: Subscription pattern for *SUB* sockets, optional,</span> <span class="sd"> defaults to ``b&#39;&#39;``.</span> <span class="sd"> :returns: A tuple containg the stream and the port number.</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="n">sock</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">context</span><span class="o">.</span><span class="n">socket</span><span class="p">(</span><span class="n">sock_type</span><span class="p">)</span> <span class="c"># addr may be &#39;host:port&#39; or (&#39;host&#39;, port)</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">addr</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span> <span class="n">addr</span> <span class="o">=</span> <span class="n">addr</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s">&#39;:&#39;</span><span class="p">)</span> <span class="n">host</span><span class="p">,</span> <span class="n">port</span> <span class="o">=</span> <span class="n">addr</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">addr</span><span class="p">)</span> <span class="o">==</span> <span class="mi">2</span> <span class="k">else</span> <span class="p">(</span><span class="n">addr</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="bp">None</span><span class="p">)</span> <span class="c"># Bind/connect the socket</span> <span class="k">if</span> <span class="n">bind</span><span class="p">:</span> <span class="k">if</span> <span class="n">port</span><span class="p">:</span> <span class="n">sock</span><span class="o">.</span><span class="n">bind</span><span class="p">(</span><span class="s">&#39;tcp://</span><span class="si">%s</span><span class="s">:</span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">host</span><span class="p">,</span> <span class="n">port</span><span class="p">))</span> <span class="k">else</span><span class="p">:</span> <span class="n">port</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">bind_to_random_port</span><span class="p">(</span><span class="s">&#39;tcp://</span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="n">host</span><span class="p">)</span> <span class="k">else</span><span class="p">:</span> <span class="n">sock</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="s">&#39;tcp://</span><span class="si">%s</span><span class="s">:</span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">host</span><span class="p">,</span> <span class="n">port</span><span class="p">))</span> <span class="c"># Add a default subscription for SUB sockets</span> <span class="k">if</span> <span class="n">sock_type</span> <span class="o">==</span> <span class="n">zmq</span><span class="o">.</span><span class="n">SUB</span><span class="p">:</span> <span class="n">sock</span><span class="o">.</span><span class="n">setsockopt</span><span class="p">(</span><span class="n">zmq</span><span class="o">.</span><span class="n">SUBSCRIBE</span><span class="p">,</span> <span class="n">subscribe</span><span class="p">)</span> <span class="c"># Create the stream and add the callback</span> <span class="n">stream</span> <span class="o">=</span> <span class="n">zmqstream</span><span class="o">.</span><span class="n">ZMQStream</span><span class="p">(</span><span class="n">sock</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">loop</span><span class="p">)</span> <span class="k">if</span> <span class="n">callback</span><span class="p">:</span> <span class="n">stream</span><span class="o">.</span><span class="n">on_recv</span><span class="p">(</span><span class="n">callback</span><span class="p">)</span> <span class="k">return</span> <span class="n">stream</span><span class="p">,</span> <span class="nb">int</span><span class="p">(</span><span class="n">port</span><span class="p">)</span> </pre></div> </div> <div class="section" id="pongproc-the-actual-process"> <h3>PongProc – The Actual&nbsp;Process</h3> <p>The <em>PongProc</em> inherits <em>ZmqProcess</em> and is the main class for our process. It creates the streams, starts the event loop and dispatches all messages to the appropriate&nbsp;handlers:</p> <div class="highlight"><pre><span class="c"># example_app/pongproc.py</span> <span class="kn">import</span> <span class="nn">zmq</span> <span class="kn">import</span> <span class="nn">base</span> <span class="n">host</span> <span class="o">=</span> <span class="s">&#39;127.0.0.1&#39;</span> <span class="n">port</span> <span class="o">=</span> <span class="mi">5678</span> <span class="k">class</span> <span class="nc">PongProc</span><span class="p">(</span><span class="n">base</span><span class="o">.</span><span class="n">ZmqProcess</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Main processes for the Ponger. It handles ping requests and sends back</span> <span class="sd"> a pong.</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bind_addr</span><span class="p">):</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">__init__</span><span class="p">()</span> <span class="bp">self</span><span class="o">.</span><span class="n">bind_addr</span> <span class="o">=</span> <span class="n">bind_addr</span> <span class="bp">self</span><span class="o">.</span><span class="n">rep_stream</span> <span class="o">=</span> <span class="bp">None</span> <span class="bp">self</span><span class="o">.</span><span class="n">ping_handler</span> <span class="o">=</span> <span class="n">PingHandler</span><span class="p">()</span> <span class="k">def</span> <span class="nf">setup</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;Sets up PyZMQ and creates all streams.&quot;&quot;&quot;</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">setup</span><span class="p">()</span> <span class="c"># Create the stream and add the message handler</span> <span class="bp">self</span><span class="o">.</span><span class="n">rep_stream</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">stream</span><span class="p">(</span><span class="n">zmq</span><span class="o">.</span><span class="n">REP</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">bind_addr</span><span class="p">,</span> <span class="n">bind</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> <span class="bp">self</span><span class="o">.</span><span class="n">rep_stream</span><span class="o">.</span><span class="n">on_recv</span><span class="p">(</span><span class="n">RepStreamHandler</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">rep_stream</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">stop</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">ping_handler</span><span class="p">))</span> <span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;Sets up everything and starts the event loop.&quot;&quot;&quot;</span> <span class="bp">self</span><span class="o">.</span><span class="n">setup</span><span class="p">()</span> <span class="bp">self</span><span class="o">.</span><span class="n">loop</span><span class="o">.</span><span class="n">start</span><span class="p">()</span> <span class="k">def</span> <span class="nf">stop</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;Stops the event loop.&quot;&quot;&quot;</span> <span class="bp">self</span><span class="o">.</span><span class="n">loop</span><span class="o">.</span><span class="n">stop</span><span class="p">()</span> </pre></div> <p>If you are going to start this process as a sub-process via <em>start</em>, make sure everything you instantiate in <em>__init__</em> is pickle-able or it won’t work on Windows (Linux and Mac <span class="caps">OS</span> X use <em>fork</em> to create a sub-process and <em>fork</em> just makes a copy of the main process and gives it a new process <span class="caps">ID</span>. <a class="reference external" href="http://docs.python.org/py3k/library/multiprocessing#windows">On Windows</a>, there is no <em>fork</em> and the context of your main process is pickled and sent to the&nbsp;sub-process).</p> <p>In <em>setup</em>, call <tt class="docutils literal"><span class="pre">super().setup()</span></tt> before you create a stream or you won’t have a <em>loop</em> instance for them. We call <em>setup</em> from <em>run</em>, because the context must be created within the new system process, which wouldn’t be the case if we called <em>setup</em> from <em>__init__</em>.</p> <p>The <em>stop</em> method is not really necessary in this example, but it can be used to send stop messages to sub-processes when the main process terminates and to do other kinds of clean-up. You can also execute it if you except a <tt class="docutils literal">KeyboardInterrupt</tt> after calling <em>run</em>.</p> </div> <div class="section" id="messagehandler-the-base-class-for-message-handlers"> <h3>MessageHandler — The Base Class for Message&nbsp;Handlers</h3> <p>A PyZMQ message handler can be any callable that accepts one argument—the list of message parts as byte objects. Hence, our <em>MessageHandler</em> class needs to implement <em>__call__</em>:</p> <div class="highlight"><pre><span class="c"># exmaple_app/base.py</span> <span class="kn">from</span> <span class="nn">zmq.utils</span> <span class="kn">import</span> <span class="n">jsonapi</span> <span class="k">as</span> <span class="n">json</span> <span class="k">class</span> <span class="nc">MessageHandler</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Base class for message handlers for a :class:`ZMQProcess`.</span> <span class="sd"> Inheriting classes only need to implement a handler function for each</span> <span class="sd"> message type.</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">json_load</span><span class="o">=-</span><span class="mi">1</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">_json_load</span> <span class="o">=</span> <span class="n">json_load</span> <span class="k">def</span> <span class="nf">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">msg</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Gets called when a messages is received by the stream this handlers is</span> <span class="sd"> registered at. *msg* is a list as return by</span> <span class="sd"> :meth:`zmq.core.socket.Socket.recv_multipart`.</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="c"># Try to JSON-decode the index &quot;self._json_load&quot; of the message</span> <span class="n">i</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_json_load</span> <span class="n">msg_type</span><span class="p">,</span> <span class="n">data</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">msg</span><span class="p">[</span><span class="n">i</span><span class="p">])</span> <span class="n">msg</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="n">data</span> <span class="c"># Get the actual message handler and call it</span> <span class="k">if</span> <span class="n">msg_type</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s">&#39;_&#39;</span><span class="p">):</span> <span class="k">raise</span> <span class="ne">AttributeError</span><span class="p">(</span><span class="s">&#39;</span><span class="si">%s</span><span class="s"> starts with an &quot;_&quot;&#39;</span> <span class="o">%</span> <span class="n">msg_type</span><span class="p">)</span> <span class="nb">getattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">msg_type</span><span class="p">)(</span><span class="o">*</span><span class="n">msg</span><span class="p">)</span> </pre></div> <p>As you can see, it’s quite simle. It just tries to <span class="caps">JSON</span>-load the index defined by <tt class="docutils literal">self._json_load</tt>. We earlier defined, that the first element of the <span class="caps">JSON</span>-encoded message defines the message type (e.g., <em>ping</em>). If an attribute of the same name exists in the inheriting class, it is called with the remainer of the&nbsp;message.</p> <p>You can also add logging or additional security measures here, but that is not necessary&nbsp;here.</p> </div> <div class="section" id="repstreamhandler-the-concrete-message-handler"> <h3>RepStreamHandler —&nbsp;The Concrete Message&nbsp;Handler</h3> <p>This class inherits the <em>MessageHandler</em> I just showed you and is used in <em>PongProc.setup</em>. It defines a handler method for <em>ping</em> messages and the <em>plzdiekthxbye</em> stop message. In its <em>__init__</em> it receives references to the <em>rep_stream</em>, PongProcs <em>stop</em> method and to the <em>ping_handler</em>, our actual application&nbsp;logic:</p> <div class="highlight"><pre><span class="c"># example_app/pongproc.py</span> <span class="k">class</span> <span class="nc">RepStreamHandler</span><span class="p">(</span><span class="n">base</span><span class="o">.</span><span class="n">MessageHandler</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;Handels messages arrvinge at the PongProc’s REP stream.&quot;&quot;&quot;</span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">rep_stream</span><span class="p">,</span> <span class="n">stop</span><span class="p">,</span> <span class="n">ping_handler</span><span class="p">):</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">__init__</span><span class="p">()</span> <span class="bp">self</span><span class="o">.</span><span class="n">_rep_stream</span> <span class="o">=</span> <span class="n">rep_stream</span> <span class="bp">self</span><span class="o">.</span><span class="n">_stop</span> <span class="o">=</span> <span class="n">stop</span> <span class="bp">self</span><span class="o">.</span><span class="n">_ping_handler</span> <span class="o">=</span> <span class="n">ping_handler</span> <span class="k">def</span> <span class="nf">ping</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">data</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;Send back a pong.&quot;&quot;&quot;</span> <span class="n">rep</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_ping_handler</span><span class="o">.</span><span class="n">make_pong</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> <span class="bp">self</span><span class="o">.</span><span class="n">_rep_stream</span><span class="o">.</span><span class="n">send_json</span><span class="p">(</span><span class="n">rep</span><span class="p">)</span> <span class="k">def</span> <span class="nf">plzdiekthxbye</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">data</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;Just calls :meth:`PongProc.stop`.&quot;&quot;&quot;</span> <span class="bp">self</span><span class="o">.</span><span class="n">_stop</span><span class="p">()</span> </pre></div> </div> <div class="section" id="pinghandler-the-application-logic"> <h3>PingHandler – The Application&nbsp;Logic</h3> <p>The <em>PingHandler</em> contains the actual application logic (which is not much, in this example). The <em>make_pong</em> method just gets the number of pings sent with the <em>ping</em> message and creates a new <em>pong</em> message. The serialization is done by <em>PongProc</em>, so our Handler does not depend on&nbsp;PyZMQ:</p> <div class="highlight"><pre><span class="c"># example_app/pongproc.py</span> <span class="k">class</span> <span class="nc">PingHandler</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span> <span class="k">def</span> <span class="nf">make_pong</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">num_pings</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;Creates and returns a pong message.&quot;&quot;&quot;</span> <span class="k">print</span><span class="p">(</span><span class="s">&#39;Pong got request number </span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="n">num_pings</span><span class="p">)</span> <span class="k">return</span> <span class="p">[</span><span class="s">&#39;pong&#39;</span><span class="p">,</span> <span class="n">num_pings</span><span class="p">]</span> </pre></div> </div> </div> <div class="section" id="summary"> <h2>Summary</h2> <p>Okay, that’s it for now. I showed you three ways to use PyZMQ. If you have a very simple process with only one socket, you can easily use its blocking <em>recv</em> methods. If you need more than one socket, I recommend using the event loop. And polling … you don’t want to use&nbsp;that.</p> <p>If you decide to use PyZMQ’s event loop, you should separate the application logic from all the PyZMQ stuff (like creating streams, sending/receiving messages and dispatching them). If your application consists of more then one process (which is usually the case), you should also create a base class with shared functionality for&nbsp;them.</p> <p>In the <a class="reference external" href="/2012/02/07/designing-and-testing-pyzmq-applications-part-2/">next part</a>, I’m going to talk about how you can test your&nbsp;application.</p> </div> Book review: NumPy 1.5 Beginner’s Guide2011-11-21T18:55:00+01:00Stefan Scherfketag:stefan.sofa-rockers.org,2011-11-21:2011/11/21/book-review-numpy-15-beginners-guide/<p>I recently got the chance to review the book <a class="reference external" href="http://www.packtpub.com/numpy-1-5-using-real-world-examples-beginners-guide/book">NumPy 1.5 Beginner’s Guide</a> by <em>Ivan Idris</em> and published by <a class="reference external" href="http://www.packtpub.com">Packt Publishing</a>.</p> <p>It covers many aspects of <a class="reference external" href="http://numpy.scipy.org/">NumPy</a> and also introduces <a class="reference external" href="http://www.scipy.org/">SciPy</a> as well as <a class="reference external" href="http://matplotlib.sourceforge.net/">Matplotlib</a>. The author includes a lot of examples and exercises and also shows the effects of some not-so-easy-to-understand functions using matplotlib&nbsp;graphs.</p> <p>The book is easy to read, so you should make fast progress in learning NumPy. Overall, it’s a good read for NumPy beginners. Advanced NumPy users, who just want to look up how specific things work, are better of with <a class="reference external" href="http://docs.scipy.org/doc/numpy/">NumPy’s documentation</a>,&nbsp;though.</p> Tea Timer 1.82011-11-20T17:55:49+01:00Stefan Scherfketag:stefan.sofa-rockers.org,2011-11-20:2011/11/20/tea-timer-18/<p>Since everything seems to work nicely, I’ve just uploaded the final <a class="reference external" href="/downloads/Tea_Timer-1.8.zip">Tea Timer 1.8 release</a>.</p> <p>Unfortunately, I wasn’t able to submit the new version to <a class="reference external" href="http://apple.com">apple.com</a>. Seems like they don’t accept new widgets anymore&nbsp;…</p> <p>However, here is the list of changes for this&nbsp;release:</p> <ul class="simple"> <li>[<span class="caps">NEW</span>] Growl 1.3&nbsp;support</li> <li>[<span class="caps">NEW</span>] Pink&nbsp;background</li> <li>[<span class="caps">CHANGE</span>] Alarms are now called asynchronously. The voice and Growl message appear at the same time now (instead of after the&nbsp;other).</li> <li>[<span class="caps">CHANGE</span>] The list of alarm sounds and voices is now created dynamically and thus contain Lion’s new&nbsp;voices.</li> <li>[<span class="caps">CHANGE</span>] Editing the fields for the timer target (e.g. “Tea”) or “ready in” during a countdown will now longer reset the countdown. However, editing the countdown time during an active countdown will reset the countdown&nbsp;accordingly.</li> <li>[<span class="caps">FIX</span>] Fixed a bug with keep alarming and sticky Growl messages. You won’t get flooded by them anymore. Instead, there will be only one sticky Growl message when keep alarming is&nbsp;activated.</li> </ul> Tea Timer 1.8 beta22011-11-09T20:03:14+01:00Stefan Scherfketag:stefan.sofa-rockers.org,2011-11-09:2011/11/09/tea-timer-18-beta2/<p>The new beta of Tea Timer 1.8 supports Growl 1.3 as well as 1.2 (if I didn’t break anything&nbsp;;-))</p> <p>If no further problems arise, I’ll release 1.8 final within this&nbsp;month.</p> <p><a class="reference external" href="https://bitbucket.org/ssc/tea-timer/get/1.8b2.zip">Download Tea Timer&nbsp;1.8b2</a></p> SimPy 2.22011-09-27T21:42:25+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2011-09-27:2011/09/27/simpy-22/<p>SimPy is a process-based discrete-event simulation library written in pure&nbsp;Python.</p> <p>Ontje Lünsdorf and I have already contributed to prior version of SimPy and now have become members of the SimPy&nbsp;team.</p> <p>In SimPy 2.2, we have restructured the package layout to be conform to the <a class="reference external" href="http://guide.python-distribute.org/">Hitchhiker’s Guide to packaging</a>. We ported the the unit tests (524 test cases) to <a class="reference external" href="http://pytest.org/">pytest</a>. We improved and cleaned up parts of the documentation and finally, we fixed the behavior of <em>Store._put</em>—thanks to Johannes Koomer for the heads-up and the&nbsp;fix.</p> <p>You can download SimPy from <a class="reference external" href="http://pypi.python.org/pypi/SimPy/2.2">PyPI</a>.</p> Tea Timer 1.8 beta12011-09-26T15:25:36+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2011-09-26:2011/09/26/tea-timer-18-beta1/<p>It’s been a while since I released a new version of Tea Timer. But finally, here it is: Tea Timer 1.8&nbsp;(beta)!</p> <p>I resolved (or rejected ;-)) most of the issues that arose during the last couple of months. Unfortunately, I could not reproduce all bugs, so that’s why there is this&nbsp;beta.</p> <p>Things that have&nbsp;changed:</p> <ul class="simple"> <li>Alarms are now called asynchronously, so the sound, voice and Growl message appear all at the same time (instead of sequentially after each other as&nbsp;before).</li> <li>The list of alarm sounds and voices is now created dynamically and thus contain Lion’s new&nbsp;voices.</li> <li>Fixed a bug with <em>keep alarming</em> and sticky Growl messages. You won’t get flooded by them anymore. Instead, there will be only one sticky Growl message when <em>keep alarming</em> is&nbsp;activated.</li> <li>Editing the fields for the timer target (e.g. “Tea”) or <em>ready in / in</em> during a countdown will now longer reset the countdown. However, editing the countdown time during an active countdown will reset the countdown&nbsp;accordingly.</li> <li>Finally, there is a pink&nbsp;background!</li> </ul> <p>Thins I didn’t&nbsp;change:</p> <ul class="simple"> <li>I could not reproduce the rendering problems on&nbsp;Lion.</li> <li>I could not reproduce <a class="reference external" href="https://bitbucket.org/ssc/tea-timer/issue/15/timer-doesnt-make-a-sound">issue #15</a>.</li> </ul> <p>If you find any bugs and have <em>Dashcode</em> installed, please also test the widget there. It might help to reproducing (and fixing) the problems. Dashboard widgets are just <span class="caps">HTML</span>/<span class="caps">CSS</span>/JavaScript, so don’t be afraid.&nbsp;;-)</p> <p>You can find the issue tracker at <a class="reference external" href="https://bitbucket.org/ssc/tea-timer/issues?status=new&amp;status=open">bitbucket</a>.</p> <p><a class="reference external" href="https://bitbucket.org/ssc/tea-timer/get/1.8b1.zip">Download Tea Timer&nbsp;1.8b1</a></p> django-lastfm 1.0.12011-05-01T13:59:40+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2011-05-01:2011/05/01/django-lastfm-101/<p><em>Django-lastfm</em> is a small Django app that allows you to embed your charts or recently listened tracks from <a class="reference external" href="http://last.fm">Last.fm</a> into your website. You can see the widget in action in the sidebar of this&nbsp;website.</p> <p>Version 1.0.1 includes a few minor bugfixes and a more comprehensive test&nbsp;coverage.</p> <p>You can find <em>django-lastfm</em> in the <a class="reference external" href="http://pypi.python.org/pypi/django-lastfm">Cheese Shop</a> or at <a class="reference external" href="http://bitbucket.org/ssc/django-lastfm">Bitbucket</a>.</p> A New Design2011-04-08T23:12:20+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2011-04-08:2011/04/08/new-design/<p>After running the old design for three and a half years, I now present the first redesign of this blog. I actually started to work on a new design in July 2010, but abandoned it until December (because I didn’t have a really good idea for&nbsp;it).</p> <p>I kept the blue and green color tones, but removed nearly every graphic (I’m not a designer anyway). However, I’m using quite a few <span class="caps">CCS3</span> features (like rounded borders, gradients, shadows or web-fonts) and created a <a class="reference external" href="http://hicksdesign.co.uk/journal/finally-a-fluid-hicksdesign">fluid layout</a> that should come in handy especially for smart phone or netbook&nbsp;users.</p> Building NumPy, SciPy & Matplotlib for Python 2.7 on Snow Leopard2010-11-17T08:30:39+01:00Stefan Scherfketag:stefan.sofa-rockers.org,2010-11-17:2010/11/17/building-numpy-scipy-matplotlib-python-27-snow-leo/<p><a class="reference external" href="/2010/11/11/building-scipy-snow-leopard-python-27/">A few days ago</a> I wrote about how to build SciPy for Python 2.7 on Mac <span class="caps">OS</span> 10.6 Snow&nbsp;Leopard.</p> <p>Usually you want to install <a class="reference external" href="http://numpy.scipy.org/">NumPy</a>, <a class="reference external" href="http://www.scipy.org/">SciPy</a> and <a class="reference external" href="http://matplotlib.sourceforge.net/">Matplotlib</a>. After reading <a class="reference external" href="http://www.scipy.org/Installing_SciPy/Mac_OS_X">Installing SciPy/Mac <span class="caps">OS</span> X</a>, the <a class="reference external" href="http://matplotlib.sourceforge.net/users/installing.html">Matplotlib installation instructions</a> and the <a class="reference external" href="http://blog.hyperjeff.net/?p=160">HJBlog</a> you might come to the conclusion, that it’s not trivial to build them on your own and that you better use the 32bit binaries for Python 2.6 or get them via <a class="reference external" href="http://trac.macports.org/browser/trunk/dports/python/">MacPorts</a>.</p> <p>But actually it’s really easy. The only dependencies that you need to install are <a class="reference external" href="http://developer.apple.com/technologies/xcode.html">Xcode</a> (for gcc and X11) and <a class="reference external" href="http://r.research.att.com/tools/">gfortran</a>.</p> <p>To simplify the the installation, I wrote a small <a class="reference external" href="http://stefan.sofa-rockers.org/downloads/Makefile_nsm">Makefile</a> that downloads all packages (except for Xcode) and builds/installs&nbsp;them:</p> <ol class="arabic"> <li><p class="first">Rename* the downloaded makefile to <tt class="docutils literal">Makefile</tt>, open <em>Terminal</em> and <tt class="docutils literal">cd</tt> to the diretory with the&nbsp;makefile.</p> </li> <li><p class="first">Download gfortran and start the graphical&nbsp;installer:</p> <div class="highlight"><pre><span class="nv">$ </span>make fortran </pre></div> </li> <li><p class="first">Download and install NumPy, SciPy and&nbsp;Matplotlib:</p> <div class="highlight"><pre><span class="nv">$ </span>make </pre></div> </li> <li><p class="first">Run NumPy’s and SciPy’s test suite and delete the temporary build&nbsp;diretory:</p> <div class="highlight"><pre><span class="nv">$ </span>make <span class="nb">test </span>clean </pre></div> </li> </ol> <p>That’s how the makefile looks&nbsp;like:</p> <div class="highlight"><pre><span class="c"># Download all packages to this directory.</span> <span class="nv">TMP_DIR</span><span class="o">=</span>./PYTMP <span class="c"># Select Python version.</span> <span class="nv">PYVERSION</span><span class="o">=</span>2.7 <span class="nv">PYTHON</span><span class="o">=</span>python<span class="k">${</span><span class="nv">PYVERSION</span><span class="k">}</span> <span class="c"># Package to download frm http://r.research.att.com/</span> <span class="nv">FORTRANPACKAGE</span><span class="o">=</span>gfortran-42-5664.pkg <span class="c"># Select which versions of the packages you want to install.</span> <span class="nv">NUMPYVERSION</span><span class="o">=</span>1.5.0 <span class="nv">SCIPYVERSION</span><span class="o">=</span>0.8.0 <span class="nv">MATPLOTLIBMAJORVERSION</span><span class="o">=</span>1.0 <span class="nv">MATPLOTLIBVERSION</span><span class="o">=</span><span class="k">${</span><span class="nv">MATPLOTLIBMAJORVERSION</span><span class="k">}</span>.0 <span class="c"># Normally, you shouldn’t need change this.</span> <span class="nv">OSX_SDK_VER</span><span class="o">=</span>10.6 <span class="nv">ARCH_FLAGS</span><span class="o">=</span>-arch i386 -arch x86_64 <span class="c"># Values for some environment variables. You shouldn’t need to change this.</span> <span class="nv">MACOSX_DEPLOYMENT_TARGET</span><span class="o">=</span><span class="k">${</span><span class="nv">OSX_SDK_VER</span><span class="k">}</span> <span class="nv">CFLAGS</span><span class="o">=</span><span class="s2">&quot;${ARCH_FLAGS} -I/usr/X11/include -I/usr/X11/include/freetype2 -isysroot /Developer/SDKs/MacOSX${OSX_SDK_VER}.sdk&quot;</span> <span class="nv">LDFLAGS</span><span class="o">=</span><span class="s2">&quot;-Wall -undefined dynamic_lookup -bundle ${ARCH_FLAGS} -L/usr/X11/lib -syslibroot,/Developer/SDKs/MacOSX${OSX_SDK_VER}.sdk&quot;</span> <span class="nv">FFLAGS</span><span class="o">=</span><span class="s2">&quot;${ARCH_FLAGS}&quot;</span> <span class="nf">default</span><span class="o">:</span> <span class="m">all</span> <span class="c"># Create a temporary directory for the build process</span> <span class="nf">_tmp_dir</span><span class="o">:</span> mkdir -p <span class="k">${</span><span class="nv">TMP_DIR</span><span class="k">}</span> <span class="c"># Clean the the temporary build directory</span> <span class="nf">clean</span><span class="o">:</span> rm -rf <span class="k">${</span><span class="nv">TMP_DIR</span><span class="k">}</span> <span class="c"># Download gfortran and start the installer</span> <span class="nf">fortran</span><span class="o">:</span> <span class="m">_tmp_dir</span> <span class="nb">cd</span> <span class="k">${</span><span class="nv">TMP_DIR</span><span class="k">}</span> <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="k">${</span><span class="nv">PYTHON</span><span class="k">}</span> -c <span class="s2">&quot;import urllib; urllib.urlretrieve(&#39;http://r.research.att.com/${FORTRANPACKAGE}&#39;, &#39;${FORTRANPACKAGE}&#39;)&quot;</span> <span class="o">&amp;&amp;</span> <span class="se">\</span> open <span class="k">${</span><span class="nv">FORTRANPACKAGE</span><span class="k">}</span> <span class="c"># Download all required packages</span> <span class="nf">fetch</span><span class="o">:</span> <span class="m">_tmp_dir</span> <span class="nb">cd</span> <span class="k">${</span><span class="nv">TMP_DIR</span><span class="k">}</span> <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="k">${</span><span class="nv">PYTHON</span><span class="k">}</span> -c <span class="s2">&quot;import urllib; urllib.urlretrieve(&#39;http://sourceforge.net/projects/numpy/files/NumPy/${NUMPYVERSION}/numpy-${NUMPYVERSION}.tar.gz/download&#39;, &#39;numpy-${NUMPYVERSION}.tar.gz&#39;)&quot;</span> <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="k">${</span><span class="nv">PYTHON</span><span class="k">}</span> -c <span class="s2">&quot;import urllib; urllib.urlretrieve(&#39;http://sourceforge.net/projects/scipy/files/scipy/${SCIPYVERSION}/scipy-${SCIPYVERSION}.tar.gz/download&#39;, &#39;scipy-${SCIPYVERSION}.tar.gz&#39;)&quot;</span> <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="k">${</span><span class="nv">PYTHON</span><span class="k">}</span> -c <span class="s2">&quot;import urllib; urllib.urlretrieve(&#39;http://sourceforge.net/projects/matplotlib/files/matplotlib/matplotlib-${MATPLOTLIBMAJORVERSION}/matplotlib-${MATPLOTLIBVERSION}.tar.gz/download&#39;, &#39;matplotlib-${MATPLOTLIBVERSION}.tar.gz&#39;)&quot;</span> <span class="c"># Extract, build and install NumPy</span> <span class="nf">numpy</span><span class="o">:</span> <span class="nb">cd</span> <span class="k">${</span><span class="nv">TMP_DIR</span><span class="k">}</span> <span class="o">&amp;&amp;</span> <span class="se">\</span> rm -rf numpy-<span class="k">${</span><span class="nv">NUMPYVERSION</span><span class="k">}</span> <span class="o">&amp;&amp;</span> <span class="se">\</span> tar -xf numpy-<span class="k">${</span><span class="nv">NUMPYVERSION</span><span class="k">}</span>.tar.gz <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="nb">cd </span>numpy-<span class="k">${</span><span class="nv">NUMPYVERSION</span><span class="k">}</span> <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="nb">export </span><span class="nv">MACOSX_DEPLOYMENT_TARGET</span><span class="o">=</span><span class="k">${</span><span class="nv">MACOSX_DEPLOYMENT_TARGET</span><span class="k">}</span> <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="nb">export </span><span class="nv">CFLAGS</span><span class="o">=</span><span class="k">${</span><span class="nv">CFLAGS</span><span class="k">}</span> <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="nb">export </span><span class="nv">LDFLAGS</span><span class="o">=</span><span class="k">${</span><span class="nv">LDFLAGS</span><span class="k">}</span> <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="nb">export </span><span class="nv">FFLAGS</span><span class="o">=</span><span class="k">${</span><span class="nv">FFLAGS</span><span class="k">}</span> <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="k">${</span><span class="nv">PYTHON</span><span class="k">}</span> setup.py build <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="k">${</span><span class="nv">PYTHON</span><span class="k">}</span> setup.py install <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="nb">cd</span> .. <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="k">${</span><span class="nv">PYTHON</span><span class="k">}</span> -c <span class="s2">&quot;import numpy; print &#39;Installed NumPy&#39;, numpy.__version__&quot;</span> <span class="c"># Extract, build and install SciPy</span> <span class="nf">scipy</span><span class="o">:</span> <span class="nb">cd</span> <span class="k">${</span><span class="nv">TMP_DIR</span><span class="k">}</span> <span class="o">&amp;&amp;</span> <span class="se">\</span> rm -rf scipy-<span class="k">${</span><span class="nv">SCIPYVERSION</span><span class="k">}</span> <span class="o">&amp;&amp;</span> <span class="se">\</span> tar -xf scipy-<span class="k">${</span><span class="nv">SCIPYVERSION</span><span class="k">}</span>.tar.gz <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="nb">cd </span>scipy-<span class="k">${</span><span class="nv">SCIPYVERSION</span><span class="k">}</span> <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="nb">export </span><span class="nv">MACOSX_DEPLOYMENT_TARGET</span><span class="o">=</span><span class="k">${</span><span class="nv">MACOSX_DEPLOYMENT_TARGET</span><span class="k">}</span> <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="nb">export </span><span class="nv">CFLAGS</span><span class="o">=</span><span class="k">${</span><span class="nv">CFLAGS</span><span class="k">}</span> <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="nb">export </span><span class="nv">LDFLAGS</span><span class="o">=</span><span class="k">${</span><span class="nv">LDFLAGS</span><span class="k">}</span> <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="nb">export </span><span class="nv">FFLAGS</span><span class="o">=</span><span class="k">${</span><span class="nv">FFLAGS</span><span class="k">}</span> <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="k">${</span><span class="nv">PYTHON</span><span class="k">}</span> setup.py build <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="k">${</span><span class="nv">PYTHON</span><span class="k">}</span> setup.py install <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="nb">cd</span> .. <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="k">${</span><span class="nv">PYTHON</span><span class="k">}</span> -c <span class="s2">&quot;import scipy; print &#39;Installed SciPy&#39;, scipy.__version__&quot;</span> <span class="c"># Extract, build and install Matplotlib</span> <span class="nf">matplotlib</span><span class="o">:</span> <span class="nb">cd</span> <span class="k">${</span><span class="nv">TMP_DIR</span><span class="k">}</span> <span class="o">&amp;&amp;</span> <span class="se">\</span> rm -rf matplotlib-<span class="k">${</span><span class="nv">MATPLOTLIBVERSION</span><span class="k">}</span> <span class="o">&amp;&amp;</span> <span class="se">\</span> tar -xf matplotlib-<span class="k">${</span><span class="nv">MATPLOTLIBVERSION</span><span class="k">}</span>.tar.gz <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="nb">cd </span>matplotlib-<span class="k">${</span><span class="nv">MATPLOTLIBVERSION</span><span class="k">}</span> <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="nb">export </span><span class="nv">MACOSX_DEPLOYMENT_TARGET</span><span class="o">=</span><span class="k">${</span><span class="nv">MACOSX_DEPLOYMENT_TARGET</span><span class="k">}</span> <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="nb">export </span><span class="nv">CFLAGS</span><span class="o">=</span><span class="k">${</span><span class="nv">CFLAGS</span><span class="k">}</span> <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="nb">export </span><span class="nv">LDFLAGS</span><span class="o">=</span><span class="k">${</span><span class="nv">LDFLAGS</span><span class="k">}</span> <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="k">${</span><span class="nv">PYTHON</span><span class="k">}</span> setup.py build <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="k">${</span><span class="nv">PYTHON</span><span class="k">}</span> setup.py install <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="nb">cd</span> .. <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="k">${</span><span class="nv">PYTHON</span><span class="k">}</span> -c <span class="s2">&quot;import matplotlib; print &#39;Installed Matplotlib&#39;, matplotlib.__version__&quot;</span> <span class="c"># Fetch and build NumPy, SciPy and Matplotlib</span> <span class="nf">all</span><span class="o">:</span> <span class="m">fetch numpy scipy matplotlib</span> <span class="nb">echo</span> <span class="s2">&quot;all done&quot;</span> <span class="c"># Execute tests for NumPy and SciPy</span> <span class="nf">test</span><span class="o">:</span> <span class="m">_tmp_dir</span> <span class="nb">export </span><span class="nv">MACOSX_DEPLOYMENT_TARGET</span><span class="o">=</span><span class="k">${</span><span class="nv">MACOSX_DEPLOYMENT_TARGET</span><span class="k">}</span> <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="nb">export </span><span class="nv">CFLAGS</span><span class="o">=</span><span class="k">${</span><span class="nv">CFLAGS</span><span class="k">}</span> <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="nb">export </span><span class="nv">LDFLAGS</span><span class="o">=</span><span class="k">${</span><span class="nv">LDFLAGS</span><span class="k">}</span> <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="nb">export </span><span class="nv">FFLAGS</span><span class="o">=</span><span class="k">${</span><span class="nv">FFLAGS</span><span class="k">}</span> <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="nb">cd</span> <span class="k">${</span><span class="nv">TMP_DIR</span><span class="k">}</span> <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="k">${</span><span class="nv">PYTHON</span><span class="k">}</span> -c <span class="s2">&quot;import numpy; numpy.test(&#39;1&#39;, &#39;10&#39;)&quot;</span> <span class="o">&amp;&amp;</span> <span class="se">\</span> <span class="k">${</span><span class="nv">PYTHON</span><span class="k">}</span> -c <span class="s2">&quot;import scipy; scipy.test(&#39;1&#39;, &#39;10&#39;)&quot;</span> </pre></div> <p>I have only tested it on my local machine yet. Hope, it helps and you don’t run into any&nbsp;troubles.</p> <p>* If you don’t rename the makefile, you must pass it’s name to <em>make:</em> <tt class="docutils literal">make <span class="pre">-f</span> Makefile_nsm &lt;target&gt;</tt>.</p> Building SciPy on Snow Leopard with Python 2.72010-11-11T09:30:08+01:00Stefan Scherfketag:stefan.sofa-rockers.org,2010-11-11:2010/11/11/building-scipy-snow-leopard-python-27/<p>I recently had some struggle to build and install <em>SciPy 0.8.0</em> on <em>Mac <span class="caps">OS</span> X 10.6 Snow Leopard</em>, but actually it’s quite&nbsp;easy.</p> <p>I used <a class="reference external" href="http://www.scipy.org/Installing_SciPy/Mac_OS_X">the instructions on scipy.org</a> and <a class="reference external" href="http://blog.hyperjeff.net/?p=160">the HJBlog</a> as source. Since there are builds of <em>NumPy 1.5.0</em> for <em>Python 2.7</em> <a class="reference external" href="http://sourceforge.net/projects/numpy/files/">available</a>, you don’t need to install <em>fftw</em> and <em>umfpack</em>&nbsp;manually.</p> <p>You only need to install <em>gfortran</em> from <a class="reference external" href="http://r.research.att.com/tools/">this site</a>. Pick the latest build for <em>Snow Leopard</em> (e.g. <a class="reference external" href="http://r.research.att.com/gfortran-42-5664.pkg">this one</a>).</p> <p>Next, install <em>NumPy</em> with the disk image from SourceForge or with <em>pip</em>:</p> <div class="highlight"><pre><span class="nv">$ </span>pip install numpy </pre></div> <p>To build and install <em>SciPy</em>, download the latest version from <a class="reference external" href="http://sourceforge.net/projects/scipy/files/">SourceForge</a> and do the&nbsp;following:</p> <div class="highlight"><pre><span class="nv">$ </span>tar -xf scipy-0.8.0.tar.gz <span class="nv">$ </span><span class="nb">cd </span>scipy-0.8.0 <span class="nv">$ </span><span class="nb">export </span><span class="nv">CFLAGS</span><span class="o">=</span><span class="s2">&quot;-arch i386 -arch x86_64&quot;</span> <span class="nv">$ </span><span class="nb">export </span><span class="nv">LDFLAGS</span><span class="o">=</span><span class="s2">&quot;-Wall -undefined dynamic_lookup -bundle -arch i386 -arch x86_64&quot;</span> <span class="nv">$ </span><span class="nb">export </span><span class="nv">FFLAGS</span><span class="o">=</span><span class="s2">&quot;-arch i386 -arch x86_64&quot;</span> <span class="nv">$ </span><span class="nb">export </span><span class="nv">MACOSX_DEPLOYMENT_TARGET</span><span class="o">=</span>10.6 <span class="nv">$ </span>python setup.py build <span class="nv">$ </span>python setup.py install </pre></div> <p>Everything in one&nbsp;command:</p> <div class="highlight"><pre><span class="nv">$ CFLAGS</span><span class="o">=</span><span class="s2">&quot;-arch i386 -arch x86_64&quot;</span> <span class="nv">LDFLAGS</span><span class="o">=</span><span class="s2">&quot;-Wall -undefined dynamic_lookup -bundle -arch i386 -arch x86_64&quot;</span> <span class="nv">FFLAGS</span><span class="o">=</span><span class="s2">&quot;-arch i386 -arch x86_64&quot;</span> <span class="nv">MACOSX_DEPLOYMENT_TARGET</span><span class="o">=</span>10.6 python setup.py build install </pre></div> <p>I hope this works for you as well as it did for&nbsp;me.</p> django-lastfm 1.02010-09-13T15:09:21+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2010-09-13:2010/09/13/django-lastfm-10/<p><em>Django-lastfm</em> is a small Django app that allows you to embed your charts or recently listened tracks from <a class="reference external" href="http://last.fm">Last.fm</a> into your website. You can see the widget in action in the sidebar of this&nbsp;website.</p> <p>I raised its version to 1.0 since there have been no problems for a long time and there are also no features I want to&nbsp;include.</p> <p>You can find <em>django-lastfm</em> in the <a class="reference external" href="http://pypi.python.org/pypi/django-lastfm">Cheese Shop</a> or at <a class="reference external" href="http://bitbucket.org/ssc/django-lastfm">Bitbucket</a>.</p> django-sphinxdoc 1.02010-09-12T19:49:47+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2010-09-12:2010/09/12/django-sphinxdoc-v10/<p>Most Python projects use <a class="reference external" href="http://sphinx.pocoo.org/">Sphinx</a> for their documentation. And many (most?) Python powered websites use <a class="reference external" href="http://www.djangoproject.com/">Django</a> as&nbsp;framework.</p> <p>So there might be some people who use both Sphinx and Django. If you belong to this group and want to integrate the documentation of your projects into your Django powered website, <em>django-sphinxdoc</em> might be the app you’re searching&nbsp;for.</p> <p><em>Django-sphinxdoc</em> can build and import your Sphinxdocumentation and provides views for browsing and searching it. You can see <em>django-sphinxdoc</em> in action be reading <a class="reference external" href="/docs/django-sphinxdoc/">its documentation</a>.</p> <p>What’s new in this&nbsp;version?</p> <ul class="simple"> <li>You can now search the documentation (via <a class="reference external" href="http://haystacksearch.org/">Haystack</a>).</li> <li>New management command <a class="reference external" href="/docs/django-sphinxdoc/ref/management/#module-sphinxdoc.management.commands.updatedoc">updatedoc</a> for importing and building <span class="caps">JSON</span> files from your documentation and updating Haystack’s search&nbsp;index.</li> <li>New model <a class="reference external" href="/docs/django-sphinxdoc/ref/models/#sphinxdoc.models.Document">Document</a> for <span class="caps">JSON</span>&nbsp;files.</li> <li>Renamed the <em>App</em> model to <a class="reference external" href="/docs/django-sphinxdoc/ref/models/#sphinxdoc.models.Project">Project</a></li> </ul> <p>What’s planned for the&nbsp;future?</p> <ul class="simple"> <li>Allow users to comment the&nbsp;documentation.</li> </ul> <p>You can find <em>django-sphinxdoc</em> in the <a class="reference external" href="http://pypi.python.org/pypi/django-sphinxdoc">Cheese Shop</a> or at <a class="reference external" href="http://bitbucket.org/ssc/django-sphinxdoc">Bitbucket</a>.</p> Tea Timer 1.7.12010-07-18T13:21:17+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2010-07-18:2010/07/18/tea-timer-171/<p>Tea Timer 1.7.1 is final now. Thank you for testing. Tea Timer also rembers now if you don’t want to update to a specific version and won’t show the update dialog until the next&nbsp;update.</p> <p><a class="reference external" href="http://stefan.sofa-rockers.org/downloads/Tea_Timer-1.7.1.zip">Download</a> | <a class="reference external" href="/teatimer/#changelog">ChangeLog</a></p> Adding sections with folding for CSS in TextMate2010-07-14T18:09:48+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2010-07-14:2010/07/14/adding-sections-folding-css-textmate/<p>In <span class="caps">CSS</span> there is no concept of sections or nested blocks. But since <span class="caps">CSS</span> files usually become very long, you may want to fold groups of style definitions to get a better&nbsp;overview:</p> <div class="highlight"><pre><span class="c">/* start foldme */</span> <span class="nt">p</span> <span class="p">{</span> <span class="k">color</span><span class="o">:</span> <span class="m">#000</span> <span class="p">}</span> <span class="c">/* end foldme */</span> </pre></div> <p>Since this is not possible with the <span class="caps">CSS</span> bundle shiped with Textmate, I added this on&nbsp;myself:</p> <ol class="arabic"> <li><p class="first">I first added a snipped called <em>section</em> to the <span class="caps">CSS</span>&nbsp;bundle:</p> <div class="highlight"><pre>/* start ${1:section} */ $0 /* end $1 */ </pre></div> <p>It gets activated by the tab trigger <tt class="docutils literal">sec</tt>. The scope selector is <tt class="docutils literal">source.css</tt>.</p> </li> <li><p class="first">I then modified the language definition and added <tt class="docutils literal"><span class="pre">|/\*</span> start \w+ \*/</tt> to the <em>foldingStartMarker</em> and <tt class="docutils literal"><span class="pre">|/\*</span> end \w+ \*/</tt> to the <em>foldingStopMarker</em>. They should look like this&nbsp;now:</p> <div class="highlight"><pre><span class="n">foldingStartMarker</span> <span class="o">=</span> <span class="s">&#39;/\*\*(?!\*)|\{\s*($|/\*(?!.*?\*/.*\S))|/\* start \w+ \*/&#39;</span><span class="p">;</span> <span class="n">foldingStopMarker</span> <span class="o">=</span> <span class="s">&#39;(?&lt;!\*)\*\*/|^\s*\}|/\* end \w+ \*/&#39;</span><span class="p">;</span> </pre></div> </li> </ol> <p>That’s all you need to&nbsp;do.</p> Tea Timer 1.7.1-RC12010-07-12T21:56:58+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2010-07-12:2010/07/12/tea-timer-171-rc1/<p>Sorry for that buggy 1.7 release. Thanks to you, who helped me fixing them by giving good error descriptions. Also thanks you, who trolled and thus showed me how much you need this little widget.&nbsp;;-)</p> <p>To be sure, I really fixed everything, I’ve decided let you test the new release, before I make it final. Please download the curren <a class="reference external" href="http://bitbucket.org/ssc/tea-timer/get/tip.zip">tip</a> from the <a class="reference external" href="http://bitbucket.org/ssc/tea-timer/downloads">repository at bitbucket.org</a>. If you find anything, please create an <a class="reference external" href="http://bitbucket.org/ssc/tea-timer/issues?status=new&amp;status=open">issue</a> or just write a comment here.&nbsp;Thanks!</p> <p><span class="caps">PS</span>: I’ve decided to leave the update message the way it is. Normally, you only have to see it once or twice a year and it takes just one click to get rid of it.&nbsp;:)</p> Tea Timer 1.72010-07-04T20:09:10+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2010-07-04:2010/07/04/tea-timer-17/<p>I just released <a class="reference external" href="/teatimer/">Tea Timer 1.7</a> which implements a few wishes from&nbsp;you:</p> <ul class="simple"> <li>There’s now a rewind button for resetting an ongoing countdown to its initial&nbsp;value.</li> <li>The timer will be reset to the initial value, not the last pause value, after the&nbsp;alarm.</li> <li>The current widget’s settings will also be saved as default settings for new&nbsp;widgets.</li> </ul> <p>Unfortunately it was not possible (neither with Python nor with Apple Script) to implement a backlink from Growl to Tea Timer to stop the alarm when you click on the Growl&nbsp;notification.</p> <p>Anyway, <a class="reference external" href="http://stefan.sofa-rockers.org/downloads/Tea_Timer-1.7.zip">Download Tea Timer 1.7</a> and&nbsp;enjoy!</p> Collectors 1.02010-06-24T19:52:59+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2010-06-24:2010/06/24/collectors-10/<p>It took me nearly three months to fix five small issues with the documentation. But now I finally released <em>Collectors v1.0</em>.&nbsp;:-)</p> <p>You can read everything important in the <a class="reference external" href="/2010/04/03/collectors-10-rc1/"><span class="caps">RC1</span> posting</a>.</p> Collectors 1.0-rc12010-04-03T10:39:15+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2010-04-03:2010/04/03/collectors-10-rc1/<p><a class="reference external" href="/collectors/">Collectors</a> made a huge jump von v0.1 to v1.0 over the last weeks, since we added lots of changes and consider what we’ve done as stable. If we don’t find any bugs, we’ll release the final v1.0 at the end of next week. The changes&nbsp;are:</p> <ul class="simple"> <li>[<span class="caps">NEW</span>]&nbsp;Documentation!</li> <li>[<span class="caps">NEW</span>]&nbsp;Tests!</li> <li>[<span class="caps">NEW</span>] Collectors can use different storage backends&nbsp;now</li> <li>[<span class="caps">NEW</span>] Storage backend for&nbsp;PyTables</li> <li>[<span class="caps">NEW</span>] Storage backend for <span class="caps">MS</span>&nbsp;Excel</li> <li>[<span class="caps">NEW</span>] <tt class="docutils literal">Collector.collect()</tt> as alias to <tt class="docutils literal">Collector.__call__()</tt></li> <li>[<span class="caps">CHANGE</span>] <tt class="docutils literal">Monitor</tt> is now called <tt class="docutils literal">Collector</tt></li> <li>[<span class="caps">CHANGE</span>] Shortcut functions moved to <tt class="docutils literal">shortcuts</tt> package</li> </ul> <p>You can download and install <em>Collectors</em> via <a class="reference external" href="http://pypi.python.org/pypi/Collectors/">PyPI</a>. If you find any bugs or have ideas for further enhancement, please <a class="reference external" href="http://bitbucket.org/ssc/collectors/issues/">let us know</a>.</p> <p>The Documentation can be found <a class="reference external" href="/docs/collectors">here</a>.</p> Updates for django-lastfm and django-sphinxdoc2010-03-12T09:37:22+01:00Stefan Scherfketag:stefan.sofa-rockers.org,2010-03-12:2010/03/12/updates-django-lastfm-and-django-sphinxdoc/<p>After reading <a class="reference external" href="http://guide.python-distribute.org/index.html">The Hitchhiker’s Guide to Packaging</a> I update my packages for <a class="reference external" href="/django-lastfm/">django-lastfm</a> and <a class="reference external" href="/django-sphinxdoc/">django-sphinxdoc</a>.</p> <p>There are no functionional improvements, so you don’t need to update&nbsp;them.</p> Django: Highlighting with ReST using Pygments2010-01-13T21:20:20+01:00Stefan Scherfketag:stefan.sofa-rockers.org,2010-01-13:2010/01/13/django-highlighting-rest-using-pygments/<p>In my last post I wrote about code highlighting in <span class="caps">HTML</span> formatted text, but since I’m now using <a class="reference external" href="http://docutils.sourceforge.net/rst.html">reStructuredText</a> for markup things, I want to share how I added synthax highlighting for&nbsp;this.</p> <p>Within your project directory (the same that contains the <tt class="docutils literal">settings.py</tt>) create a file called <tt class="docutils literal">rst_directive.py</tt> and fill it with the following&nbsp;code:</p> <div class="highlight"><pre><span class="c"># -*- coding: utf-8 -*-</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> The Pygments reStructuredText directive</span> <span class="sd"> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~</span> <span class="sd"> This fragment is a Docutils_ 0.5 directive that renders source code</span> <span class="sd"> (to HTML only, currently) via Pygments.</span> <span class="sd"> To use it, adjust the options below and copy the code into a module</span> <span class="sd"> that you import on initialization. The code then automatically</span> <span class="sd"> registers a ``sourcecode`` directive that you can use instead of</span> <span class="sd"> normal code blocks like this::</span> <span class="sd"> .. sourcecode:: python</span> <span class="sd"> My code goes here.</span> <span class="sd"> If you want to have different code styles, e.g. one with line numbers</span> <span class="sd"> and one without, add formatters with their names in the VARIANTS dict</span> <span class="sd"> below. You can invoke them instead of the DEFAULT one by using a</span> <span class="sd"> directive option::</span> <span class="sd"> .. sourcecode:: python</span> <span class="sd"> :linenos:</span> <span class="sd"> My code goes here.</span> <span class="sd"> Look at the `directive documentation`_ to get all the gory details.</span> <span class="sd"> .. _Docutils: http://docutils.sf.net/</span> <span class="sd"> .. _directive documentation:</span> <span class="sd"> http://docutils.sourceforge.net/docs/howto/rst-directives.html</span> <span class="sd"> :copyright: Copyright 2006-2009 by the Pygments team, see AUTHORS.</span> <span class="sd"> :license: BSD, see LICENSE for details.</span> <span class="sd">&quot;&quot;&quot;</span> <span class="c"># Options</span> <span class="c"># ~~~~~~~</span> <span class="c"># Set to True if you want inline CSS styles instead of classes</span> <span class="n">INLINESTYLES</span> <span class="o">=</span> <span class="bp">False</span> <span class="kn">from</span> <span class="nn">pygments.formatters</span> <span class="kn">import</span> <span class="n">HtmlFormatter</span> <span class="c"># The default formatter</span> <span class="n">DEFAULT</span> <span class="o">=</span> <span class="n">HtmlFormatter</span><span class="p">(</span><span class="n">noclasses</span><span class="o">=</span><span class="n">INLINESTYLES</span><span class="p">)</span> <span class="c"># Add name -&gt; formatter pairs for every variant you want to use</span> <span class="n">VARIANTS</span> <span class="o">=</span> <span class="p">{</span> <span class="c"># &#39;linenos&#39;: HtmlFormatter(noclasses=INLINESTYLES, linenos=True),</span> <span class="p">}</span> <span class="kn">from</span> <span class="nn">docutils</span> <span class="kn">import</span> <span class="n">nodes</span> <span class="kn">from</span> <span class="nn">docutils.parsers.rst</span> <span class="kn">import</span> <span class="n">directives</span><span class="p">,</span> <span class="n">Directive</span> <span class="kn">from</span> <span class="nn">pygments</span> <span class="kn">import</span> <span class="n">highlight</span> <span class="kn">from</span> <span class="nn">pygments.lexers</span> <span class="kn">import</span> <span class="n">get_lexer_by_name</span><span class="p">,</span> <span class="n">TextLexer</span> <span class="k">class</span> <span class="nc">Pygments</span><span class="p">(</span><span class="n">Directive</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot; Source code syntax hightlighting.</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="n">required_arguments</span> <span class="o">=</span> <span class="mi">1</span> <span class="n">optional_arguments</span> <span class="o">=</span> <span class="mi">0</span> <span class="n">final_argument_whitespace</span> <span class="o">=</span> <span class="bp">True</span> <span class="n">option_spec</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">([(</span><span class="n">key</span><span class="p">,</span> <span class="n">directives</span><span class="o">.</span><span class="n">flag</span><span class="p">)</span> <span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">VARIANTS</span><span class="p">])</span> <span class="n">has_content</span> <span class="o">=</span> <span class="bp">True</span> <span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">assert_has_content</span><span class="p">()</span> <span class="k">try</span><span class="p">:</span> <span class="n">lexer</span> <span class="o">=</span> <span class="n">get_lexer_by_name</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">arguments</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span> <span class="k">except</span> <span class="ne">ValueError</span><span class="p">:</span> <span class="c"># no lexer found - use the text one instead of an exception</span> <span class="n">lexer</span> <span class="o">=</span> <span class="n">TextLexer</span><span class="p">()</span> <span class="c"># take an arbitrary option if more than one is given</span> <span class="n">formatter</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">options</span> <span class="ow">and</span> <span class="n">VARIANTS</span><span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">options</span><span class="o">.</span><span class="n">keys</span><span class="p">()[</span><span class="mi">0</span><span class="p">]]</span> <span class="ow">or</span> <span class="n">DEFAULT</span> <span class="n">parsed</span> <span class="o">=</span> <span class="n">highlight</span><span class="p">(</span><span class="s">u&#39;</span><span class="se">\n</span><span class="s">&#39;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">content</span><span class="p">),</span> <span class="n">lexer</span><span class="p">,</span> <span class="n">formatter</span><span class="p">)</span> <span class="k">return</span> <span class="p">[</span><span class="n">nodes</span><span class="o">.</span><span class="n">raw</span><span class="p">(</span><span class="s">&#39;&#39;</span><span class="p">,</span> <span class="n">parsed</span><span class="p">,</span> <span class="n">format</span><span class="o">=</span><span class="s">&#39;html&#39;</span><span class="p">)]</span> <span class="n">directives</span><span class="o">.</span><span class="n">register_directive</span><span class="p">(</span><span class="s">&#39;sourcecode&#39;</span><span class="p">,</span> <span class="n">Pygments</span><span class="p">)</span> </pre></div> <p>Open the file <tt class="docutils literal">__init__.py</tt> in the same directory and add the following&nbsp;line:</p> <div class="highlight"><pre><span class="kn">import</span> <span class="nn">rst_directive</span> </pre></div> <p>Furthermore you might want to create a <span class="caps">CSS</span> file containing all the style definitions in your static media directory. The following python script will do the&nbsp;job:</p> <div class="highlight"><pre><span class="c"># Call it this way:</span> <span class="c"># python gen_css.py pygments.css</span> <span class="kn">import</span> <span class="nn">sys</span> <span class="kn">from</span> <span class="nn">pygments.formatters</span> <span class="kn">import</span> <span class="n">HtmlFormatter</span> <span class="n">f</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="s">&#39;w&#39;</span><span class="p">)</span> <span class="c"># You can change style and the html class here:</span> <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">HtmlFormatter</span><span class="p">(</span><span class="n">style</span><span class="o">=</span><span class="s">&#39;colorful&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get_style_defs</span><span class="p">(</span><span class="s">&#39;.highlight&#39;</span><span class="p">))</span> <span class="n">f</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> </pre></div> <p>Now you can highlight code using the <tt class="docutils literal">sourcecode</tt> directive:</p> <div class="highlight"><pre>*Hello World!* in Python is so easy: .. sourcecode:: python print &#39;Hello World!&#39; </pre></div> <p>That’s it! If you have any questions, leave a&nbsp;comment.</p> Django: Highlighting in HTML using Pygments and Beautiful Soup2010-01-13T20:13:58+01:00Stefan Scherfketag:stefan.sofa-rockers.org,2010-01-13:2010/01/13/django-highlighting-html-using-pygments-and-beauti/<p>Two months ago I <a class="reference external" href="/2009/10/24/beautifulsoup-django-and-pygments/">wrote</a> about <a class="reference external" href="http://www.djangoproject.com/">Django</a>, <a class="reference external" href="http://pygments.org/">Pygments</a> and <a class="reference external" href="http://www.crummy.com/software/BeautifulSoup/">Beautiful Soup</a>.</p> <p>A few days after the post I switched from <span class="caps">HTML</span> markup to <a class="reference external" href="http://docutils.sourceforge.net/rst.html">reStructuredText</a> and thus didn’t need my filter anymore and forgot to post more about&nbsp;it.</p> <p>Today I received a comment asking me to publish the source code of the template filter—so here it&nbsp;is:</p> <div class="highlight"><pre><span class="c"># encoding: utf-8</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd">A filter to highlight code blocks in html with Pygments and BeautifulSoup.</span> <span class="sd"> {% load highlight_code %}</span> <span class="sd"> {{ var_with_code|highlight|safe }}</span> <span class="sd">&quot;&quot;&quot;</span> <span class="kn">from</span> <span class="nn">BeautifulSoup</span> <span class="kn">import</span> <span class="n">BeautifulSoup</span> <span class="kn">from</span> <span class="nn">django</span> <span class="kn">import</span> <span class="n">template</span> <span class="kn">from</span> <span class="nn">django.template.defaultfilters</span> <span class="kn">import</span> <span class="n">stringfilter</span> <span class="kn">import</span> <span class="nn">pygments</span> <span class="kn">import</span> <span class="nn">pygments.formatters</span> <span class="kn">import</span> <span class="nn">pygments.lexers</span> <span class="n">register</span> <span class="o">=</span> <span class="n">template</span><span class="o">.</span><span class="n">Library</span><span class="p">()</span> <span class="nd">@register.filter</span> <span class="nd">@stringfilter</span> <span class="k">def</span> <span class="nf">highlight</span><span class="p">(</span><span class="n">html</span><span class="p">):</span> <span class="n">soup</span> <span class="o">=</span> <span class="n">BeautifulSoup</span><span class="p">(</span><span class="n">html</span><span class="p">)</span> <span class="n">codeblocks</span> <span class="o">=</span> <span class="n">soup</span><span class="o">.</span><span class="n">findAll</span><span class="p">(</span><span class="s">&#39;pre&#39;</span><span class="p">)</span> <span class="k">for</span> <span class="n">block</span> <span class="ow">in</span> <span class="n">codeblocks</span><span class="p">:</span> <span class="k">if</span> <span class="n">block</span><span class="o">.</span><span class="n">has_key</span><span class="p">(</span><span class="s">&#39;class&#39;</span><span class="p">):</span> <span class="k">try</span><span class="p">:</span> <span class="n">code</span> <span class="o">=</span> <span class="s">&#39;&#39;</span><span class="o">.</span><span class="n">join</span><span class="p">([</span><span class="nb">unicode</span><span class="p">(</span><span class="n">item</span><span class="p">)</span> <span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">block</span><span class="o">.</span><span class="n">contents</span><span class="p">])</span> <span class="n">lexer</span> <span class="o">=</span> <span class="n">pygments</span><span class="o">.</span><span class="n">lexers</span><span class="o">.</span><span class="n">get_lexer_by_name</span><span class="p">(</span><span class="n">block</span><span class="p">[</span><span class="s">&#39;class&#39;</span><span class="p">])</span> <span class="n">formatter</span> <span class="o">=</span> <span class="n">pygments</span><span class="o">.</span><span class="n">formatters</span><span class="o">.</span><span class="n">HtmlFormatter</span><span class="p">()</span> <span class="n">code_hl</span> <span class="o">=</span> <span class="n">pygments</span><span class="o">.</span><span class="n">highlight</span><span class="p">(</span><span class="n">code</span><span class="p">,</span> <span class="n">lexer</span><span class="p">,</span> <span class="n">formatter</span><span class="p">)</span> <span class="n">block</span><span class="o">.</span><span class="n">contents</span> <span class="o">=</span> <span class="p">[</span><span class="n">BeautifulSoup</span><span class="p">(</span><span class="n">code_hl</span><span class="p">)]</span> <span class="n">block</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="s">&#39;code&#39;</span> <span class="k">except</span><span class="p">:</span> <span class="k">raise</span> <span class="k">return</span> <span class="nb">unicode</span><span class="p">(</span><span class="n">soup</span><span class="p">)</span> </pre></div> <p>Copy the code into a file called <tt class="docutils literal">templatetags/highlight_code.py</tt> within a new or an existing Django&nbsp;app.</p> <p><tt class="docutils literal">highlight()</tt> searches the passed <span class="caps">HTML</span> code for <tt class="docutils literal">&lt;pre&gt;</tt>-tags with a class denoting the <a class="reference external" href="http://pygments.org/docs/lexers/">lexer</a> to be used,&nbsp;e.g.:</p> <div class="highlight"><pre><span class="nt">&lt;p&gt;&lt;em&gt;</span>Hello World!<span class="nt">&lt;/em&gt;</span> in Python:<span class="nt">&lt;/p&gt;</span> <span class="nt">&lt;pre</span> <span class="na">class=</span><span class="s">&quot;python&quot;</span><span class="nt">&gt;</span> print &#39;Hello World&#39; <span class="nt">&lt;/pre&gt;</span> </pre></div> <p>Furthermore you might want to create a <span class="caps">CSS</span> file containing all the style definitions in your static media directory. The following python script will do the&nbsp;job:</p> <div class="highlight"><pre><span class="c"># Call it this way:</span> <span class="c"># python gen_css.py pygments.css</span> <span class="kn">import</span> <span class="nn">sys</span> <span class="kn">from</span> <span class="nn">pygments.formatters</span> <span class="kn">import</span> <span class="n">HtmlFormatter</span> <span class="n">f</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="s">&#39;w&#39;</span><span class="p">)</span> <span class="c"># You can change style and the html class here:</span> <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">HtmlFormatter</span><span class="p">(</span><span class="n">style</span><span class="o">=</span><span class="s">&#39;colorful&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get_style_defs</span><span class="p">(</span><span class="s">&#39;.highlight&#39;</span><span class="p">))</span> <span class="n">f</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> </pre></div> <p>I hope I explained all this well enough—if not, leave a comment and I’ll update this post.&nbsp;:-)</p> Django-sphinxdoc 0.2 now with documentation!2009-12-30T19:42:08+01:00Stefan Scherfketag:stefan.sofa-rockers.org,2009-12-30:2009/12/30/django-sphinxdoc-02-now-documentation/<p>Just released version 0.2 of <a class="reference external" href="/django-sphinxdoc/">django-sphinxdoc</a>. It can now display the documentation itself, the general index and the module index. The source and static views as well as the search functionality is not implemented&nbsp;yet.</p> <p>I’m still pondering whether to use a Google custom search or the Sphinx JavaScript search. Maybe it would be even better if I used <a class="reference external" href="http://haystacksearch.org/">Haystack</a>&nbsp;…</p> <p>You can find a quick guide as well as some other guides in <a class="reference external" href="/docs/django-sphinxdoc">django-sphinxdoc’s documentation section</a> (which is of course powered by django-sphinxdoc&nbsp;;-)).</p> <p>The download can be found at <a class="reference external" href="http://bitbucket.org/ssc/django-sphinxdoc/downloads/">bitbucket.org</a>.</p> Tea Timer 1.62009-12-22T19:31:26+01:00Stefan Scherfketag:stefan.sofa-rockers.org,2009-12-22:2009/12/22/tea-timer-16/<p>A new version of <a class="reference external" href="/teatimer/">Tea Timer</a> has just been released. It fixes as a few bugs and has two small new&nbsp;features:</p> <ol class="arabic simple"> <li>It now displays the current version number on the&nbsp;website.</li> <li>It can handle multiple instances of the widget, each with different settings. One side effect of this is, that each new instance starts with its built-in default settings. If you don’t close and reopen your widget every hour, this should not be a problem.&nbsp;;-)</li> </ol> <p><a class="reference external" href="http://stefan.sofa-rockers.org/downloads/Tea_Timer-1.6.zip">Download Tea Timer&nbsp;1.6</a></p> <p>One feature I did not add, was a way to easily add more background colors. Since each background has its own image, you can’t easily add a box for a hex color code. But if you realy want more or different colors, here is a quick guide to achive&nbsp;this.</p> <ol class="arabic simple"> <li>Clone the <a class="reference external" href="http://bitbucket.org/ssc/tea-timer/">Tea Timer repository</a> or just <a class="reference external" href="http://bitbucket.org/ssc/tea-timer/get/tip.zip">download</a> the latest&nbsp;version.</li> <li>Open the bundle <tt class="docutils literal">TemplateWidget.djproj</tt> with <em>Dashcode</em>.</li> <li>You can find the image unter <tt class="docutils literal">TemplateWidget/front/frontImg</tt>. Adjust its size to 320×120px and color it as you&nbsp;want.</li> <li>Copy the image to clipboard, pasted in your favourite editor and save it in the Widgets image folder (<tt class="docutils literal">Tea Timer.wdgt/Images/</tt>) as <tt class="docutils literal">Front_mycolor.png</tt>.</li> <li>Open <tt class="docutils literal">Tea Timer.wdgt/TeaTimer.html</tt> in a text editor. Around line 65 you’ll find the options for all colors. Add your color&nbsp;there:</li> </ol> <div class="highlight"><pre>... <span class="nt">&lt;option</span> <span class="na">value=</span><span class="s">&quot;silver&quot;</span><span class="nt">&gt;</span>Silver<span class="nt">&lt;/option&gt;</span> <span class="nt">&lt;option</span> <span class="na">value=</span><span class="s">&quot;mycolor&quot;</span><span class="nt">&gt;</span>My Color<span class="nt">&lt;/option&gt;</span> </pre></div> <ol class="arabic simple" start="6"> <li>Save and exit the&nbsp;editor.</li> <li>Install the new version by double clicking on&nbsp;it.</li> <li>That’s it. Ask if you have any questions.&nbsp;:-)</li> </ol> <p><strong>[Update]</strong> Tea Timer does <em>not</em> run under Mac <span class="caps">OS</span> X 10.4. I forgot to change this at my last submission to Apple and now I have to wait until I release the next&nbsp;version.</p> Documentation for django-lastfm with django-sphinxdoc2009-12-20T11:59:03+01:00Stefan Scherfketag:stefan.sofa-rockers.org,2009-12-20:2009/12/20/documentation-django-lastfm-django-sphinxdoc/<p>Yesterday I finished a first version of <a class="reference external" href="/django-sphinxdoc">django-sphinxdoc</a> that integrates <a class="reference external" href="http://sphinx.pocoo.org/index.html">Sphinx documentation</a> into a Django powered website. Its based on Django’s documentation app, but can manage the documenation for more then one app. I’ll post more on this&nbsp;later.</p> <p>What’s more important is, that I have put the <a class="reference external" href="/docs/django-lastfm">documentation for django-lastfm</a> online with it.&nbsp;:-)</p> django-lastfm2009-11-22T16:58:15+01:00Stefan Scherfketag:stefan.sofa-rockers.org,2009-11-22:2009/11/22/django-lastfm/<p>On my old Wordpress blog I had a widget that let me display some of my last.fm stats. Since there was no such widget for Django powered sites (or I didn’t search well enough), I created my own as you can see in the right&nbsp;column.</p> <p>You currently can choose between your recently listened tracks, your weekly artist chart and your top artists. I’ve created a <a class="reference external" href="http://bitbucket.org/ssc/django-lastfm/">bitbucket project</a> for its further development. So go clone it and give some&nbsp;feedback.</p> <p><span class="caps">PS</span>: I’ll update/write the documentation within the next few days …&nbsp;;-)</p> A BeautifulSoup with Django and Pygments2009-10-24T18:06:05+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2009-10-24:2009/10/24/beautifulsoup-django-and-pygments/<p>Just added syntax highlighting using <a class="reference external" href="http://www.crummy.com/software/BeautifulSoup">BeautifulSoup</a> and <a class="reference external" href="http://pygments.org/">Pygments</a>. I took the <a class="reference external" href="http://www.saltycrane.com/blog/2008/08/django-blog-project-12-adding-pygments-syntax-highlighting/">SaltyCrane Blog</a> for inspiration, but in contrast to it, I implemented it as a template filter in a separate application. This is surely not the most performant way, but I’m planning to use memcache, so I think this is&nbsp;ok.</p> <p>Here an example how to use the&nbsp;filter:</p> <div class="highlight"><pre><span class="cp">{%</span> <span class="k">load</span> <span class="nv">highlight_code</span> <span class="cp">%}</span> <span class="cp">{{</span> <span class="nv">my_var_with_code</span><span class="o">|</span><span class="nf">highlight</span><span class="o">|</span><span class="nf">safe</span> <span class="cp">}}</span> </pre></div> More Feeds2009-10-20T16:24:00+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2009-10-20:2009/10/20/more-feeds/<p>Just added Atom Feeds for post categories and post comments. They are available in the category detail and post detail&nbsp;views.</p> Comments and an Atom Feed2009-10-18T19:47:58+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2009-10-18:2009/10/18/comments-and-atom-feed/<p>Small update for this site: Comments are now working and there is an <a class="reference external" href="http://stefan.sofa-rockers.org/feeds/latest/">Atom Feed</a> for the latest posts in this&nbsp;blog.</p> <p>The behaviour and the style of the comments is not very polished yet, but I’ll work on that later. I’ll also add more feeds, e.g. for post comments, categories and&nbsp;tags.</p> Magic! Python! Django! Whee!2009-10-14T09:26:43+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2009-10-14:2009/10/14/magic-python-django-whee/<p>This blog is now run by <a class="reference external" href="http://www.djangoproject.com/">Django</a>. I didn’t really like <a class="reference external" href="http://wordpress.org/">Wordpress</a> which I used before. And also I don’t like <span class="caps">PHP</span> anymore (I really liked it some years ago, but everything changed, when I learned Python&nbsp;…).</p> <p>When I told a friend that I wanted to switch from Wordpress to something else, he just said: «Use Django.» So I took a look at the <a class="reference external" href="http://docs.djangoproject.com/en/dev/intro/tutorial01/">Tutorial</a> and was instantly&nbsp;thrilled.</p> <p>At first, I wanted to use an existing weblog app, but I also wanted to code an app for my own and since I din’t find a weblog app that I 100% liked, I just decided to write my own. So here it is (far from finished&nbsp;though)!</p> <p>Features so&nbsp;far:</p> <ul class="simple"> <li>Basic post model (title, slug, pub date, modify date, status, category, <span class="caps">HTML</span>&nbsp;body)</li> <li>Post manager for post counts (per year, month, category (and&nbsp;tag))</li> <li>Hierarchical category model – imho, this is the highlight of my&nbsp;app.</li> <li><a class="reference external" href="/archive/">Archive</a> and <a class="reference external" href="/category/">Categories</a></li> <li>Some special template&nbsp;tags.</li> <li>Usage of a <a class="reference external" href="http://www.undefinedfire.com/lab/recursion-django-templates/">recursive template tag</a> for the&nbsp;categories</li> <li>Unit- and doctests for the models and template&nbsp;tags</li> </ul> <p>Features to&nbsp;come:</p> <ul class="simple"> <li>Comments</li> <li>Atom&nbsp;feeds</li> <li>Sidebar</li> <li>Static&nbsp;pages</li> <li><a class="reference external" href="http://trac.edgewall.org/">Trac</a> integration for my&nbsp;projects</li> <li>Search</li> <li><a class="reference external" href="http://last.fm">Last.fm</a> sidebar&nbsp;widget</li> <li>Tags for posts (via <a class="reference external" href="http://code.google.com/p/django-tagging/">django-tagging</a>?)</li> <li><a class="reference external" href="http://ckeditor.com/">CKEditor</a>&nbsp;integration</li> <li><a class="reference external" href="http://pygments.org/">Pygments</a>&nbsp;integration</li> <li>Improved image/media support for&nbsp;posts</li> <li>Admin actions for posts (e.g. change status and&nbsp;category)</li> <li>Preview function for new&nbsp;posts</li> </ul> <p>That’s it so far. When I have Trac running, I’ll also publish the source of my applications.&nbsp;:-)</p> <div class="figure align-center"> <img alt="Magic! Python! Django! Whee!" src="http://media.djangopony.com/img/magic-pony-django-wallpaper.png" style="width: 400px;" /> </div> Oh noes! Tea Timer 1.5.1 hot fix2009-07-30T08:22:14+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2009-07-30:2009/07/30/oh-noes-tea-timer-151-hot-fix/<p>There were two bugs in Tea Timer 1.5 (one minor and one major – try to change the alarm settings while »keep alarming« is activated&nbsp;:-D).</p> <p>So I just released version 1.5.1 that fixes both issues. Thanks to Lakestone for the&nbsp;heads-up!</p> <p>Download <a class="reference external" href="http://stefan.sofa-rockers.org/downloads/TeaTimer-1.5.1.zip">Tea Timer&nbsp;1.5.1</a></p> Tea Timer 1.52009-07-28T22:33:01+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2009-07-28:2009/07/28/tea-timer-15/<p>It’s been a while since I last upated Tea Timer but I have quite a lot to do with my diploma thesis. But finally the new version is done. I implemented two feature requests and updated jQuery: Tea Timer now remembers the time you entered and resets the timer to that value after the countdown. In previous versions, the counter was reset to 00:00:00. The other feature allows you to let Tea Timer keep alarming you every 5 seconds. To stop the alarm, you have to click on the alarm clock, which will show an additional stop icon then. You can change this setting in the alarm&nbsp;preferences.</p> <div class="figure align-center"> <img alt="Tea Timer 1.5" src="http://stefan.sofa-rockers.org/images/Tea-Timer-1.5.png" style="width: 320px;" /> </div> <p>Download <a class="reference external" href="http://stefan.sofa-rockers.org/downloads/TeaTimer-1.5.zip">Tea Timer&nbsp;1.5</a></p> TeaTimer 1.42009-02-15T20:30:48+01:00Stefan Scherfketag:stefan.sofa-rockers.org,2009-02-15:2009/02/15/teatimer-14/<p>Actually, I finished the work on TeaTimer 1.4 nearly one month ago, but since I moved to a new server and had some problems with the domain transfer, I delayed the release of the new&nbsp;version.</p> <p>In this release, I implemented a few user requested features, upgraded the jQuery version and fixed a minor bug. TeaTimer now remembers the last timer target and it is now possible to switch off the <em>… is ready</em> in the alarm message. If you click on the <em>ready in</em> label, a select box opens. If you choose <em>ready in</em>, TeaTimer will alarm you with e.g. «Tea is ready». If you choose <em>in</em>, it will just say e.g. «Coffee&nbsp;Time».</p> <div class="figure align-center"> <img alt="Tea Timer 1.4" src="http://stefan.sofa-rockers.org/images/teatimer-14.png" style="width: 320px;" /> </div> <p>If you want to keep the Growl message open until you click it, go to <em>System Preferences » Growl » Applications</em>, select <em>TeaTimer</em> and edit the settings as you&nbsp;wish.</p> <p>Download <a class="reference external" href="http://stefan.sofa-rockers.org/downloads/TeaTimer-1.4.zip">TeaTimer&nbsp;1.4</a></p> Tea Timer 1.32008-12-13T22:27:18+01:00Stefan Scherfketag:stefan.sofa-rockers.org,2008-12-13:2008/12/13/tea-timer-13/<p>This is another bug fix release. Tea Timer couldn’t handle the values “08” and “09” for hours/minutes/seconds correctly, which is now fixed. The JavaScript function <tt class="docutils literal">parseInt()</tt> handles strings beginning with zero as octal numbers and there simply is no octal 8 or 9. So I just had to tell <tt class="docutils literal">parseInt()</tt> that it must handle all strings as decimal numbers to fix this&nbsp;issue.</p> <p>Thx to <em>pat</em> for his bug&nbsp;report.</p> <p><a class="reference external" href="http://stefan.sofa-rockers.org/downloads/TeaTimer-1.3.zip">Download Tea Timer&nbsp;1.3</a></p> Tea Timer 1.22008-12-08T19:00:11+01:00Stefan Scherfketag:stefan.sofa-rockers.org,2008-12-08:2008/12/08/tea-timer-12/<p>This is fastes new version of Tea Timer ever! ;-) I just found a small issue with one label of the update panel and added a link to this website on the widget’s back. I’m sorry if you just downloaded version 1.1 and have to update&nbsp;again.</p> <p><a class="reference external" href="http://stefan.sofa-rockers.org/downloads/TeaTimer-1.2.zip">Download Tea Timer&nbsp;1.2</a></p> Tea Timer 1.12008-12-08T16:29:11+01:00Stefan Scherfketag:stefan.sofa-rockers.org,2008-12-08:2008/12/08/tea-timer-11/<p>This release of Tea Timer fixes two little issues (Alex’ voice was missing and there was a problem with pausing the timer) and adds the ability to start the timer with the Enter or Return key. So Tea Timer is now fully keyboard controllable, once you have activated an input field.&nbsp;:-)</p> <p>An extensive <a class="reference external" href="/teatimer/#changelog">ChangeLog</a> can be found on <a class="reference external" href="/teatimer">Tea Timer’s project page</a>.</p> <p><a class="reference external" href="http://stefan.sofa-rockers.org/downloads/TeaTimer-1.1.zip">Download Tea Timer&nbsp;1.1</a></p> Olé! Tea Timer 1.0!2008-11-30T20:22:12+01:00Stefan Scherfketag:stefan.sofa-rockers.org,2008-11-30:2008/11/30/ole-tea-timer-10/<p>Tea Timer has finally gone final – all things I wanted in for version 1.0 are implemented (and thoroughly tested ;-)). Check out the <a class="reference external" href="/teatimer">project site</a> for further information, screenshots and the download&nbsp;link.</p> <p>Changes since version&nbsp;0.4:</p> <ul class="simple"> <li>[<span class="caps">NEW</span>] Alarm via&nbsp;Growl</li> <li>[<span class="caps">NEW</span>] Alarm via a selectable&nbsp;voice</li> <li>[<span class="caps">NEW</span>] Alarm via a selectable system&nbsp;sound</li> <li>[<span class="caps">NEW</span>] Fancy animation for pref-pane&nbsp;transitions</li> <li>[<span class="caps">FIX</span>] say-message now&nbsp;localized</li> </ul> <p>Things that might be implemented in the&nbsp;future:</p> <ul class="simple"> <li>Choose icon (pizza, tea, cake, eggs,&nbsp;…)</li> <li>Remember last entries for timer&nbsp;target</li> <li>List with teas and their required&nbsp;time</li> </ul> Tea Timer 0.42008-11-22T22:21:40+01:00Stefan Scherfketag:stefan.sofa-rockers.org,2008-11-22:2008/11/22/tea-timer-04/<p>I proudly present version 0.4 of the soon world dominating dashboard&nbsp;widget.</p> <p>Changes from version&nbsp;0.3:</p> <ul class="simple"> <li>[<span class="caps">NEW</span>] Info button and back&nbsp;side</li> <li>[<span class="caps">NEW</span>] Set color for the front background within the widget&nbsp;preferences.</li> <li>[<span class="caps">NEW</span>] Widget localization and English&nbsp;translation</li> <li>[<span class="caps">NEW</span>] Update notification shows current and available&nbsp;version.</li> <li>[<span class="caps">NEW</span>] Fancy drop shadows for text&nbsp;;-)</li> <li>[<span class="caps">CHANGE</span>] Overhauled default background and widget&nbsp;icon</li> </ul> <p>Remaining on the <span class="caps">TODO</span>&nbsp;list:</p> <ul class="simple"> <li>Action after countdown:<ul> <li>Sound</li> <li>Growl</li> <li>Dialog</li> </ul> </li> <li>Choose icon (pizza, tea, cake, eggs,&nbsp;…)</li> <li>Remember last entries for timer&nbsp;target</li> <li>List with tees and their required&nbsp;time</li> <li>Fading for tab transition in&nbsp;preferences</li> </ul> <p><a class="reference external" href="http://stefan.sofa-rockers.org/downloads/TeaTimer-0.4.zip">Download Tea Timer&nbsp;0.4</a></p> <div class="figure align-center"> <img alt="Tea Timer 0.4 (German) with ocean background" src="http://stefan.sofa-rockers.org/images/teatimer-04_1.png" style="width: 320px;" /> </div> <div class="figure align-center"> <img alt="Tea Timer 0.4 with backside showing the design preferences" src="http://stefan.sofa-rockers.org/images/teatimer-04_2.png" style="width: 320px;" /> </div> Tea Timer 0.32008-11-11T21:17:07+01:00Stefan Scherfketag:stefan.sofa-rockers.org,2008-11-11:2008/11/11/tea-timer-03/<p>After I’ve been forgetting my tea in the kitchen or my pizza in the oven way too often and not finding a nice Dashboard Widget to alarm myself in time, I created my one&nbsp;one:</p> <div class="figure align-center"> <img alt="Tea Timer 0.3" src="http://stefan.sofa-rockers.org/images/teatimer-03.png" style="width: 320px;" /> </div> <p>Tea Timer 0.3 has the following&nbsp;features:</p> <ul class="simple"> <li>Enter an arbitrary subject to wait for by clicking on&nbsp;«Tee».</li> <li>Change the time by clicking on the&nbsp;digits.</li> <li>Use the <em>up/down arrow</em> key to change the&nbsp;time.</li> <li>Use <em>Tab</em> and <em>Shift+Tab</em> to swich through input&nbsp;fields.</li> <li>Reminder by the <em>Victory Apple Voice</em>.</li> <li>Automatic check for&nbsp;updates.</li> <li>German user&nbsp;interface.</li> </ul> <p>Planned&nbsp;features:</p> <ul class="simple"> <li>Info button and back&nbsp;side</li> <li>Addition action after countdown end:<ul> <li>Custom&nbsp;sound</li> <li>Grow&nbsp;notification</li> <li>Dialog</li> </ul> </li> <li>Update notification shows available&nbsp;version</li> <li>Internationalization /&nbsp;Localization</li> <li>Overhaul widget&nbsp;icon</li> <li>Changeable background&nbsp;colors</li> <li>Let user choose an icon (pizza, tea, cake, eggs,&nbsp;…)</li> <li>Remember last entries for timer&nbsp;target</li> <li>List with tees and their required&nbsp;time</li> <li><span class="caps">CSS3</span>&nbsp;styling</li> </ul> <p>And, most importantly, the <a class="reference external" href="http://stefan.sofa-rockers.org/downloads/TeaTimer-0.3.zip">download link</a>.</p> It’s all better2007-09-15T13:13:22+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2007-09-15:2007/09/15/alles-ist-besser/<p>Put too much time into the styling. Again. But at least, everything looks a lot nicer now. the Last.fm widget is also running now.&nbsp;:-)</p> Hello world2007-09-13T22:44:12+02:00Stefan Scherfketag:stefan.sofa-rockers.org,2007-09-13:2007/09/13/hallo-welt/<p>Ohai, this is my new blog. It’s still work in progress. Don’t know yet, what we’ll see here&nbsp;…</p>