https://zameermanji.com/Zameer Manji's Blog2024-01-07T00:00:00ZZameer Manjihttps://zameermanji.comhttps://zameermanji.com/assets/images/favicon-128.pngtag:zameermanji.com,2024-01-07:/blog/2024/1/7/understanding-turbotax-tax-files/Understanding TurboTax .tax files2024-01-07T00:00:00Z2024-01-07T00:00:00Z<p>The start of a new year is the start of everyone’s favorite season:
<a href="https://www.youtube.com/watch?v=_57OWOU1Yc8">tax season</a>!.
Every year I use TurboTax to file my taxes and the online version offers
two downloads to archive a copy of the return:</p>
<ol type="1">
<li>A PDF of all of the forms.</li>
<li>A <code>.tax</code> file that can be later imported into
TurboTax</li>
</ol>
<p>The <code>.tax</code> file is an undocumented format and can only be
imported by TurboTax. It appears to contain the tax return data in a
structured format. This would be useful if you want to read select
fields of your tax return programmatically.</p>
<p>I prodded at the <code>.tax</code> file and realized it’s a zip file.
The <code>unzip</code> command can list the contents.</p>
<figure class="code-block shell-session"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>unzip -l TurboTaxReturn.tax2017
<span class="go">Archive: TurboTaxReturn.tax2017</span>
<span class="go"> Length Date Time Name</span>
<span class="go">--------- ---------- ----- ----</span>
<span class="go"> 976 01-05-2024 08:50 manifest.xml</span>
<span class="go"> 3114304 01-05-2024 08:50 tax:62edf303-b57a-46ad-b725-47a9188bf46d</span>
<span class="go">--------- -------</span>
<span class="go"> 3115280 2 files</span>
</code></pre></div>
</figure>
<p>However the <code>manifest.xml</code> file appears to not be a valid
xml file.</p>
<figure class="code-block shell-session"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>unzip -p TurboTaxReturn.tax2017 manifest.xml <span class="p">|</span> head -c <span class="m">16</span> <span class="p">|</span> xxd
<span class="go">00000000: a1b1 fefb 3718 dd9c 082d 9c86 2300 10fa ....7....-..#...</span>
</code></pre></div>
</figure>
<p>After some prodding of the TurboTax desktop application it appears to
be a AES CBC mode encrypted file. Using the powers of deduction I have
determined the keys to encrypt the <code>manifest.xml</code> are:</p>
<ul>
<li>For 2017 tax year files the key is <code>!QAZ2WSX#EDC4RFV</code>
with an IV of <code>5TGB@YHN7UJM(IK(</code></li>
<li>For 2019 tax year files the key is
<code>7HMT&BGM5KBNFH><</code> with an IV of
<code>#YBU7JLZ*JGL7MAR</code></li>
</ul>
<p>Using the <a href="https://pypi.org/project/pycryptodome/">pycryptodome</a> library
it’s pretty easy to get the contents of the <code>manifest.xml</code>
file.</p>
<figure class="code-block python"><div class="highlight"><pre><span></span><code><span class="ch">#!/usr/bin/env python3</span>
<span class="kn">import</span> <span class="nn">zipfile</span>
<span class="kn">import</span> <span class="nn">xml.dom.minidom</span>
<span class="kn">from</span> <span class="nn">Crypto.Cipher</span> <span class="kn">import</span> <span class="n">AES</span>
<span class="kn">from</span> <span class="nn">Crypto.Util.Padding</span> <span class="kn">import</span> <span class="n">unpad</span>
<span class="k">def</span> <span class="nf">main</span><span class="p">():</span>
<span class="n">zf</span> <span class="o">=</span> <span class="n">zipfile</span><span class="o">.</span><span class="n">ZipFile</span><span class="p">(</span><span class="s1">'./TurboTaxReturn.tax2017'</span><span class="p">)</span>
<span class="n">manifestf</span> <span class="o">=</span> <span class="n">zf</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="s1">'manifest.xml'</span><span class="p">)</span>
<span class="n">manifest_encrypted</span> <span class="o">=</span> <span class="n">manifestf</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
<span class="n">cipher</span> <span class="o">=</span> <span class="n">AES</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="s2">"5TGB@YHN7UJM(IK("</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">),</span> <span class="n">AES</span><span class="o">.</span><span class="n">MODE_CBC</span><span class="p">,</span> <span class="n">iv</span><span class="o">=</span><span class="s2">"!QAZ2WSX#EDC4RFV"</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">))</span>
<span class="n">manifest_decrypted</span> <span class="o">=</span> <span class="n">cipher</span><span class="o">.</span><span class="n">decrypt</span><span class="p">(</span><span class="n">manifest_encrypted</span><span class="p">)</span>
<span class="n">manifest</span> <span class="o">=</span> <span class="n">unpad</span><span class="p">(</span><span class="n">manifest_decrypted</span><span class="p">,</span> <span class="mi">16</span><span class="p">)</span>
<span class="n">parsed_manifest</span> <span class="o">=</span> <span class="n">xml</span><span class="o">.</span><span class="n">dom</span><span class="o">.</span><span class="n">minidom</span><span class="o">.</span><span class="n">parseString</span><span class="p">(</span><span class="n">manifest</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">parsed_manifest</span><span class="o">.</span><span class="n">toprettyxml</span><span class="p">(</span><span class="n">indent</span><span class="o">=</span><span class="s1">' '</span><span class="p">))</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
<span class="n">main</span><span class="p">()</span>
</code></pre></div>
</figure>
<p>Running the above results in</p>
<figure class="code-block shell-session"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>./decrypt.py
<span class="go"><?xml version="1.0" ?></span>
<span class="go"><Manifest xmlns="http://schemas.intuit.com/LocalCfpContainer.xsd"></span>
<span class="go"> <documents></span>
<span class="go"> <document></span>
<span class="go"> <entityType>/tax/income/individual/taxreturn/taxml</entityType></span>
<span class="go"> <entityKey>tax:62edf303-b57a-46ad-b725-47a9188bf46d</entityKey></span>
<span class="go"> <entityVer>2018-03-03T04:58:40.793Z</entityVer></span>
<span class="go"> <attributes></span>
<span class="go"> <attribute></span>
<span class="go"> <key>appId</key></span>
<span class="go"> <value>Intuit.ctg.tto.platform</value></span>
<span class="go"> </attribute></span>
<span class="go"> <attribute></span>
<span class="go"> <key>year</key></span>
<span class="go"> <value>2017</value></span>
<span class="go"> </attribute></span>
<span class="go"> <attribute></span>
<span class="go"> <key>deviceId</key></span>
<span class="go"> <value>TurboTax Online</value></span>
<span class="go"> </attribute></span>
<span class="go"> <attribute></span>
<span class="go"> <key>deviceName</key></span>
<span class="go"> <value>TurboTax Online</value></span>
<span class="go"> </attribute></span>
<span class="go"> <attribute></span>
<span class="go"> <key>appSku</key></span>
<span class="go"> <value>8</value></span>
<span class="go"> </attribute></span>
<span class="go"> <attribute></span>
<span class="go"> <key>formSetId</key></span>
<span class="go"> <value>US1040PER</value></span>
<span class="go"> </attribute></span>
<span class="go"> <attribute></span>
<span class="go"> <key>saveType</key></span>
<span class="go"> <value>endSession</value></span>
<span class="go"> </attribute></span>
<span class="go"> <attribute></span>
<span class="go"> <key>docName</key></span>
<span class="go"> <value>Zameer</value></span>
<span class="go"> </attribute></span>
<span class="go"> <attribute></span>
<span class="go"> <key>ceDataType</key></span>
<span class="go"> <value>TAXML</value></span>
<span class="go"> </attribute></span>
<span class="go"> </attributes></span>
<span class="go"> </document></span>
<span class="go"> </documents></span>
<span class="go"></Manifest></span>
</code></pre></div>
</figure>
<p>The manifest contains a list of all returns in the file and some
metadata about each return. Unfortunately the tax return file is
encrypted as well with a different key. Using powers of deduction the
keys to encrypt the <code>tax:</code> file are:</p>
<ul>
<li>For 2017 tax year files the key is <code>4TGB@YHN7UJM(IK(</code>
with an IV of <code>!ASZ2WSX#EDC4RFV</code></li>
<li>For 2019 tax year files the key is <code>8NV^RASJVG*(XSCB</code>
with an IV of <code>#BMBVVBD$FSZ6LSZ</code></li>
</ul>
<p>Adjusting the above script to use the above keys on a
<code>tax:</code> file results in an XML representation of the tax
return.</p>
<figure class="code-block shell-session"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>./decrypt.py <span class="p">|</span> head
<span class="go"><?xml version="1.0" ?></span>
<span class="go"><?xml-stylesheet type="text/xsl" href="http://taxml.intuit.com/xml_stylesheets/taxml_default.xslt"?></span>
<span class="go"><TaxReturns xmlns="http://www.intuit.com/Ctg/Pta/TurboTax/TaxReturns"></span>
<span class="go"> <USIndividualTaxReturn xmlns="http://www.intuit.com/Ctg/Pta/TurboTax/USIndividualTaxReturn" xmlns:po="http://www.intuit.com/Ctg/Pta/TurboTax/PersistentObjects" xmlns:td="http://www.intuit.com/Ctg/Pta/TurboTax/TaxDataType" taxYear="2019" tpsEngineVersion="4.7.2 (Tps-Compatibility-Version: 2018.4)" uuid="63de8c2c-0d75-4f7e-9929-8a19f81d2e32"></span>
<span class="go"> <US1040PER xmlns="http://www.intuit.com/Ctg/Pta/TurboTax/US1040PER" checksum="1890142416" cidFormSet="USIndividual" dataVersion="771297" formsetAttribute="1" implementationVersion="2019.94.0.1" showSmartWorksheets="true" tpsPrefix="S2019" tpsType="formset" uuid="0e24cebc-5be7-400c-ad0f-f658e10918df"></span>
</code></pre></div>
</figure>
<p>Unfortunately the schema is not documented but it should be possible
to read certain fields out of the tax return with any standard XML
parser.</p>
tag:zameermanji.com,2024-01-05:/blog/2024/1/5/binding-to-privileged-ports-without-root-on-macos/Binding to privileged ports without root on macOS2024-01-05T00:00:00Z2024-01-05T00:00:00Z<p>Since macOS Mojave <a href="https://developer.apple.com/forums/thread/674179">it has been
possible</a> to bind to privileged ports without root on macOS. For
example running the following shows <code>nc</code> binding to port 81
without issue.</p>
<figure class="code-block shell-session"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>nc -l <span class="m">0</span>.0.0.0 <span class="m">81</span>
<span class="go">^C</span>
</code></pre></div>
</figure>
<p>However binding to a port on a specific interface still requires
root.</p>
<figure class="code-block shell-session"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>nc -l <span class="m">127</span>.0.0.1 <span class="m">81</span>
<span class="go">nc: Permission denied</span>
</code></pre></div>
</figure>
<p>This makes binding a process to a single interface like a WireGuard
network or localhost impossible without root permissions.</p>
<p>It’s possible to work around the above restriction by using
<code>launchd</code> socket activation instead. Instead of having a
process bind to ports itself, <code>launchd</code> will bind to those
ports and pass those file descriptors to the process. For example I have
the following <code>plist</code> to run Python’s <code>HTTPServer</code>
module listening on localhost only.</p>
<figure class="code-block xml"><div class="highlight"><pre><span></span><code><span class="cp"><?xml version="1.0" encoding="UTF-8"?></span>
<span class="cp"><!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"></span>
<span class="nt"><plist</span> <span class="na">version=</span><span class="s">"1.0"</span><span class="nt">></span>
<span class="nt"><dict></span>
<span class="nt"><key></span>KeepAlive<span class="nt"></key></span>
<span class="nt"><true/></span>
<span class="nt"><key></span>Label<span class="nt"></key></span>
<span class="nt"><string></span>zmanji.startpage<span class="nt"></string></span>
<span class="nt"><key></span>EnvironmentVariables<span class="nt"></key></span>
<span class="nt"><dict></span>
<span class="nt"><key></span>PYTHONUNBUFFERED<span class="nt"></key></span>
<span class="nt"><string></span>1<span class="nt"></string></span>
<span class="nt"></dict></span>
<span class="nt"><key></span>ProgramArguments<span class="nt"></key></span>
<span class="nt"><array></span>
<span class="nt"><string></span>/Users/zmanji/bin/startpage.py<span class="nt"></string></span>
<span class="nt"></array></span>
<span class="nt"><key></span>WorkingDirectory<span class="nt"></key></span>
<span class="nt"><string></span>/Users/zmanji/.config/startpage<span class="nt"></string></span>
<span class="nt"><key></span>RunAtLoad<span class="nt"></key></span>
<span class="nt"><true/></span>
<span class="nt"><key></span>Sockets<span class="nt"></key></span>
<span class="nt"><dict></span>
<span class="nt"><key></span>Listeners<span class="nt"></key></span>
<span class="nt"><dict></span>
<span class="nt"><key></span>SockNodeName<span class="nt"></key></span>
<span class="nt"><string></span>127.0.0.1<span class="nt"></string></span>
<span class="nt"><key></span>SockServiceName<span class="nt"></key></span>
<span class="nt"><string></span>80<span class="nt"></string></span>
<span class="nt"></dict></span>
<span class="nt"></dict></span>
<span class="nt"><key></span>StandardOutPath<span class="nt"></key></span>
<span class="nt"><string></span>/tmp/zmanji.startpage.stdout.log<span class="nt"></string></span>
<span class="nt"><key></span>StandardErrorPath<span class="nt"></key></span>
<span class="nt"><string></span>/tmp/zmanji.startpage.stderr.log<span class="nt"></string></span>
<span class="nt"></dict></span>
<span class="nt"></plist></span>
</code></pre></div>
<figcaption><p>Example launchd <code>plist</code> that has a <code>Sockets</code>
key</p></figcaption></figure>
<p>The key part of this <code>plist</code> is the <code>Sockets</code>
key which specifies the interface and port for <code>launchd</code> to
bind to.</p>
<p>Then in the program being run it needs to receive the socket from
<code>launchd</code> by calling the <a href="https://developer.apple.com/documentation/xpc/1505523-launch_activate_socket"><code>launch_activate_socket</code></a>
API. Once the sockets are obtained they can be used like any other
socket. For example my <code>startpage.py</code> script looks like
this.</p>
<figure class="code-block python"><div class="highlight"><pre><span></span><code><span class="ch">#!/usr/bin/python3</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">from</span> <span class="nn">socket</span> <span class="kn">import</span> <span class="n">socket</span>
<span class="kn">from</span> <span class="nn">http.server</span> <span class="kn">import</span> <span class="n">HTTPServer</span><span class="p">,</span> <span class="n">SimpleHTTPRequestHandler</span>
<span class="k">def</span> <span class="nf">launch_activate_socket</span><span class="p">(</span><span class="n">name</span><span class="p">:</span> <span class="nb">str</span><span class="p">):</span>
<span class="kn">from</span> <span class="nn">ctypes</span> <span class="kn">import</span> <span class="n">byref</span><span class="p">,</span> <span class="n">CDLL</span><span class="p">,</span> <span class="n">c_int</span><span class="p">,</span> <span class="n">c_size_t</span><span class="p">,</span> <span class="n">POINTER</span>
<span class="n">libc</span> <span class="o">=</span> <span class="n">CDLL</span><span class="p">(</span><span class="s1">'/usr/lib/libc.dylib'</span><span class="p">)</span>
<span class="n">fds</span> <span class="o">=</span> <span class="n">POINTER</span><span class="p">(</span><span class="n">c_int</span><span class="p">)()</span>
<span class="n">count</span> <span class="o">=</span> <span class="n">c_size_t</span><span class="p">()</span>
<span class="n">res</span> <span class="o">=</span> <span class="n">libc</span><span class="o">.</span><span class="n">launch_activate_socket</span><span class="p">(</span><span class="n">name</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">),</span> <span class="n">byref</span><span class="p">(</span><span class="n">fds</span><span class="p">),</span> <span class="n">byref</span><span class="p">(</span><span class="n">count</span><span class="p">))</span>
<span class="k">if</span> <span class="n">res</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">Exception</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">strerror</span><span class="p">(</span><span class="n">res</span><span class="p">))</span>
<span class="n">sockets</span> <span class="o">=</span> <span class="p">[</span><span class="n">socket</span><span class="p">(</span><span class="n">fileno</span><span class="o">=</span><span class="n">fds</span><span class="p">[</span><span class="n">s</span><span class="p">])</span> <span class="k">for</span> <span class="n">s</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">count</span><span class="o">.</span><span class="n">value</span><span class="p">)]</span>
<span class="n">libc</span><span class="o">.</span><span class="n">free</span><span class="p">(</span><span class="n">fds</span><span class="p">)</span>
<span class="k">return</span> <span class="n">sockets</span>
<span class="k">def</span> <span class="nf">main</span><span class="p">():</span>
<span class="c1"># Name must match key in Sockets dict of plist</span>
<span class="n">sockets</span> <span class="o">=</span> <span class="n">launch_activate_socket</span><span class="p">(</span><span class="s1">'Listeners'</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">sockets</span><span class="p">)</span> <span class="o">!=</span> <span class="mi">1</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">Exception</span><span class="p">(</span><span class="sa">f</span><span class="s2">"Too many sockets: </span><span class="si">{</span><span class="nb">len</span><span class="p">(</span><span class="n">sockets</span><span class="p">)</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="n">socket</span> <span class="o">=</span> <span class="n">sockets</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="c1"># Pass the socket to the HTTP server</span>
<span class="n">HTTPServer</span><span class="o">.</span><span class="n">address_family</span> <span class="o">=</span> <span class="n">socket</span><span class="o">.</span><span class="n">family</span>
<span class="k">with</span> <span class="n">HTTPServer</span><span class="p">(</span><span class="n">socket</span><span class="o">.</span><span class="n">getsockname</span><span class="p">(),</span> <span class="n">SimpleHTTPRequestHandler</span><span class="p">,</span> <span class="n">bind_and_activate</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span> <span class="k">as</span> <span class="n">httpd</span><span class="p">:</span>
<span class="n">httpd</span><span class="o">.</span><span class="n">socket</span> <span class="o">=</span> <span class="n">socket</span>
<span class="n">httpd</span><span class="o">.</span><span class="n">server_activate</span><span class="p">()</span>
<span class="n">host</span><span class="p">,</span> <span class="n">port</span> <span class="o">=</span> <span class="n">httpd</span><span class="o">.</span><span class="n">socket</span><span class="o">.</span><span class="n">getsockname</span><span class="p">()[:</span><span class="mi">2</span><span class="p">]</span>
<span class="n">url_host</span> <span class="o">=</span> <span class="sa">f</span><span class="s1">'[</span><span class="si">{</span><span class="n">host</span><span class="si">}</span><span class="s1">]'</span> <span class="k">if</span> <span class="s1">':'</span> <span class="ow">in</span> <span class="n">host</span> <span class="k">else</span> <span class="n">host</span>
<span class="nb">print</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">"Serving HTTP on </span><span class="si">{</span><span class="n">host</span><span class="si">}</span><span class="s2"> port </span><span class="si">{</span><span class="n">port</span><span class="si">}</span><span class="s2"> "</span>
<span class="sa">f</span><span class="s2">"(http://</span><span class="si">{</span><span class="n">url_host</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="n">port</span><span class="si">}</span><span class="s2">/) ..."</span>
<span class="p">)</span>
<span class="n">httpd</span><span class="o">.</span><span class="n">serve_forever</span><span class="p">()</span>
</code></pre></div>
<figcaption><p>Example Python program that obtains sockets from
<code>launchd</code></p></figcaption></figure>
<p>Once the <code>plist</code> above is placed into
<code>~/Library/LaunchAgents</code> it works like any other launch
agent. When activated the process will be listening on the specific
interface and port but no root permissions were required.</p>
tag:zameermanji.com,2023-03-26:/blog/2023/3/26/using-nix-without-root/Using nix without root2023-03-26T00:00:00Z2023-03-26T00:00:00Z<p>The <a href="https://github.com/NixOS/nix">nix</a> package manager is
a purely functional package manager that enforces a declarative and
hermetic style of building packages. I wanted to use nix to create per
project development environments and what I learned is conventional nix
installs come in two flavors.</p>
<ol type="1">
<li>Via <a href="https://nixos.org/">NixOS</a> where it is the default
and only package manager.</li>
<li>Via an installer script which requires root access on Linux or macOS
to create the <code>/nix</code> folder and in mutli-user situations a
persistent daemon.</li>
</ol>
<p>NixOS is a non starter for creating per project development
environments and requiring root to create a development environment is
not ideal. Instead I discovered it’s possible to setup nix as a non root
user and the rest of the post outlines how to do that.</p>
<h2 id="getting-the-binary">Getting the binary<a class="pilcrow" href="#getting-the-binary"></a></h2>
<p>The first problem to solve is getting the <code>nix</code> binary.
The easiest way to get a <code>nix</code> static build from the hydra CI
system<a href="#fn1" class="footnote-ref" id="fnref1" role="doc-noteref"><sup>1</sup></a>. A fully static binary is available
for the <code>2.14</code> branch of the <code>nix</code> command by
running:</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>curl -L https://hydra.nixos.org/job/nix/maintenance-2.14/buildStatic.x86_64-linux/latest/download-by-type/file/binary-dist > nix
<span class="gp">$ </span>chmod +x nix
<span class="gp">$ </span>./nix --version
<span class="go">nix (Nix) 2.14.1</span>
</code></pre></div>
</figure>
<h2 id="setting-up-the-nix-directory">Setting up the /nix directory<a class="pilcrow" href="#setting-up-the-nix-directory"></a></h2>
<p>The next problem is ensuring the <code>nix</code> binary has access
to <code>/nix</code> directory for storage. Although <code>nix</code>
can be configured to use any directory for storage, the official hydra
CI system build all nix packages with the nix store in
<code>/nix</code>. Using the same directory allows <code>nix</code> to
download cached artifacts instead of re-building every package locally.
There are two options to setup the <code>/nix</code> directory.</p>
<h3 id="option-1-using-the-built-in-chroot">Option 1: Using the built in
chroot</h3>
<p>The first option is to use the built in <a href="https://github.com/NixOS/nix/blob/f9b81935550fe7225f95d9c3f214a631e613eb28/src/nix/run.cc#L223">logic</a>
that creates <code>~/.local/share/nix/root</code> and uses linux
namespaces to bind mount that as <code>/nix</code>. This means simply
running the following will build GNU Hello and re-use cached artifacts
if available.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>./nix run <span class="s1">'nixpkgs/release-22.11#hello'</span>
</code></pre></div>
</figure>
<p>Running the above for the first time <code>nix</code> will
output.</p>
<blockquote>
<p>warning: ‘/nix/var/nix’ does not exist, so Nix will use
‘/home/zmanji/.local/share/nix/root’ as a chroot store</p>
</blockquote>
<p>Subsequent commands will reuse that folder as the <code>nix</code>
store and <code>nix</code> will use user namespaces to mount that
directory as <code>/nix</code> for each command.</p>
<p>Commands like <code>nix store ls</code> also work as expected as
well:</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>./nix store ls /nix/store/hrqijlzk6b56ij6g86vdh2b8mkv1i469-hello-2.12.1/bin/hello
</code></pre></div>
</figure>
<p>The drawback to this option is the directory is not customizable.</p>
<h2 id="option-2-running-with-bubblewrap">Option 2: Running with
bubblewrap<a class="pilcrow" href="#option-2-running-with-bubblewrap"></a></h2>
<p>The other option use to use the <a href="https://github.com/containers/bubblewrap">bubblewrap</a> tool to
setup the mounts instead of relying on the built in features of
<code>nix</code>. This allows customization of the underlying directory
for the <code>/nix</code> mount and can also prevent <code>nix</code>
from accessing certain directories as a security measure.</p>
<p>This can be done with the following command.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>mkdir -p ./mynixroot/var/nix
<span class="gp">$ </span>bwrap --unshare-user --uid <span class="k">$(</span>id -u<span class="k">)</span> --gid <span class="k">$(</span>id -g<span class="k">)</span> --proc /proc --dev /dev --tmpfs /tmp <span class="se">\</span>
--ro-bind /bin/ /bin/ <span class="se">\</span>
--ro-bind /etc/ /etc/ <span class="se">\</span>
--ro-bind /lib/ /lib/ <span class="se">\</span>
--ro-bind /lib64/ /lib64/ <span class="se">\</span>
--ro-bind /run/ /run/ <span class="se">\</span>
--ro-bind /usr/ /usr/ <span class="se">\</span>
--ro-bind /var /var <span class="se">\</span>
--bind <span class="k">$(</span><span class="nb">pwd</span><span class="k">)</span> <span class="k">$(</span><span class="nb">pwd</span><span class="k">)</span> <span class="se">\</span>
--bind ~/.config/nix ~/.config/nix <span class="se">\</span>
--bind ./mynixroot /nix <span class="se">\</span>
./nix build <span class="s1">'nixpkgs#'</span>
</code></pre></div>
</figure>
<p>This creates the <code>/nix</code> store under
<code>./mynixroot</code> and bind mounts most of the system as read
only. <code>nix</code> will use the <code>./mynixroot</code> for the nix
store. This can be useful in select environments where using
<code>~/.local/share/nix/root</code> is not appropriate.</p>
<h2 id="other-alternatives">Other Alternatives<a class="pilcrow" href="#other-alternatives"></a></h2>
<p>One alternative is the <a href="https://github.com/nix-community/nix-user-chroot">nix-user-chroot</a>
tool which also uses user namespaces to setup the <code>/nix</code>
directory. However at the time of writing it has a <a href="https://github.com/nix-community/nix-user-chroot/issues/102">bug</a>
which prevents strict sandboxing which means there is no guarantee
builds will be hermetic. I think the built in chroot functionality or
relying on bubblewrap is superior as it still allows <code>nix</code> to
enable strict sandboxing for builds.</p>
<h2 id="conclusion">Conclusion<a class="pilcrow" href="#conclusion"></a></h2>
<p>It’s very easy to setup <code>nix</code> without switching to NixOS
or without root access. Just with one static binary it’s possible to use
<code>nix</code> for a specific project.</p>
<section id="footnotes" class="footnotes footnotes-end-of-document" role="doc-endnotes">
<hr>
<ol>
<li id="fn1"><p>Strangely enough this is not mentioned on the <a href="https://nixos.org/download.html">download</a> page at all<a href="#fnref1" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
</ol>
</section>
tag:zameermanji.com,2023-01-30:/blog/2023/1/30/a-utility-for-building-wheels-in-reproducible-environments/A utility for building wheels in reproducible environments2023-01-30T00:00:00Z2023-01-30T00:00:00Z<p><a href="https://peps.python.org/pep-0517/">PEP 517</a> and <a href="https://peps.python.org/pep-0518/">PEP 518</a> improved Python
packaging by allowing projects to specify which dependencies are needed
to build a wheel. They specify a <code>build-system.requires</code>
section in a <code>pyproject.toml</code> file which acts as a
<code>requirements.txt</code> for build time dependencies. The benefit
of specifying build time dependencies is projects other than
<code>setuptools</code> can be used to build wheels.</p>
<p>Unfortunately there is no lock file equivalent of the dependencies in
<code>pyproject.toml</code> meaning tools like <code>pip</code> need to
re-resolve the dependencies every time a wheel needs to be built. As <a href="/blog/2021/6/14/building-wheels-with-pip-is-getting-slower/#what-is-pip-doing">I
have written before</a> not only does this slow building the wheel, it
also introduces non determinism. For example the following
<code>pyproject.toml</code> section results in non determinism.</p>
<figure class="code-block toml"><div class="highlight"><pre><span></span><code><span class="k">[build-system]</span><span class="w"></span>
<span class="n">requires</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="s">"setuptools>48"</span><span class="p">,</span><span class="w"> </span><span class="s">"wheel"</span><span class="p">]</span><span class="w"></span>
</code></pre></div>
</figure>
<p>Since these requirements don’t specify an exact version nor do they
pin transitive dependencies, the versions used in the build will be the
latest versions published on PyPI. It’s even possible to fail to build
entirely because of backwards incompatible changes.</p>
<h2 id="introducing-reproducible-wheel-builder">Introducing
<code>reproducible-wheel-builder</code><a class="pilcrow" href="#introducing-reproducible-wheel-builder"></a></h2>
<p>To fix the above issue I wrote <a href="https://github.com/zmanji/reproducible-wheel-builder"><code>reproducible-wheel-builder</code></a><a href="#fn1" class="footnote-ref" id="fnref1" role="doc-noteref"><sup>1</sup></a> a utility which combines <a href="https://github.com/pantsbuild/pex/"><code>pex</code></a> and <a href="https://github.com/pypa/build"><code>build</code></a> to create
reproducible environments for building wheels. Instead of relying on
<code>pip</code> to re-resolve the <code>build-system.requires</code>
for every build, <code>pex</code> and a <a href="https://blog.pantsbuild.org/introducing-pants-2-11/#pex-lockfiles">Pex
lockfile</a> are used to create a reproducible virtual environment for
the build. Then <code>build</code> uses that virtual environment to
build the wheel. Since a lockfile is used, the build environment is
reproducible.</p>
<p>It is distributed as a pex and can be downloaded from <a href="https://github.com/zmanji/reproducible-wheel-builder/releases">GitHub</a>.
Running it is as simple as executing <code>main.pex</code>.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>./main.pex --help
<span class="go">usage: __pex_executable__.py [-h] --lock LOCK --src SRC [--dist {wheel,sdist,editable}] --out OUT [--quiet] [--requires_config REQUIRES_CONFIG] [--build_config BUILD_CONFIG]</span>
<span class="go">options:</span>
<span class="go"> -h, --help show this help message and exit</span>
<span class="go"> --lock LOCK Path to the pex lock file</span>
<span class="go"> --src SRC Path to the source directory to build</span>
<span class="go"> --dist {wheel,sdist,editable}</span>
<span class="go"> Type of distribution to build</span>
<span class="go"> --out OUT Path to the output directory</span>
<span class="go"> --quiet Supress the output of the build backend</span>
<span class="go"> --requires_config REQUIRES_CONFIG</span>
<span class="go"> JSON file of config to pass to build backend when getting requires</span>
<span class="go"> --build_config BUILD_CONFIG</span>
<span class="go"> JSON file of config to pass to build backend when building</span>
</code></pre></div>
</figure>
<p>This can be used instead of <code>pip wheel</code> to build a
project.</p>
<h2 id="example">Example<a class="pilcrow" href="#example"></a></h2>
<p><a href="https://pypi.org/project/gevent/21.1.2/">gevent 21.1.2</a>
does not have wheels for Python 3.10. Using <code>pip wheel</code> on
the sdist results in the following error.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>pip --verbose wheel ./gevent-21.1.2
<span class="go">Processing ./gevent-21.1.2</span>
<span class="go"> ...</span>
<span class="go"> Error compiling Cython file:</span>
<span class="go"> -----------------------------------------------------------</span>
<span class="go"> ...</span>
<span class="go"> cdef load_traceback</span>
<span class="go"> cdef Waiter</span>
<span class="go"> cdef wait</span>
<span class="go"> cdef iwait</span>
<span class="go"> cdef reraise</span>
<span class="go"> cpdef GEVENT_CONFIG</span>
<span class="go"> ^</span>
<span class="go"> ------------------------------------------------------------</span>
<span class="go"> src/gevent/_gevent_cgreenlet.pxd:182:6: Variables cannot be declared with 'cpdef'. Use 'cdef' instead.</span>
<span class="go"> Compiling src/gevent/greenlet.py because it changed.</span>
<span class="go"> ...</span>
<span class="go"> × Getting requirements to build wheel did not run successfully.</span>
<span class="go"> │ exit code: 1</span>
<span class="go"> ╰─> See above for output.</span>
</code></pre></div>
<figcaption><p>Build failure output when trying to build a wheel for gevent
21.1.2</p></figcaption></figure>
<p>This is because the <code>pyproject.toml</code> has the following
<code>build-system.requires</code> section.</p>
<figure class="code-block toml"><div class="highlight"><pre><span></span><code><span class="n">requires</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="w"></span>
<span class="w"> </span><span class="s">"setuptools >= 40.8.0"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="s">"wheel"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="s">"Cython >= 3.0a6"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="s">"cffi >= 1.12.3 ; platform_python_implementation == 'CPython'"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="s">"greenlet >= 0.4.17, < 2.0 ; platform_python_implementation == 'CPython'"</span><span class="p">,</span><span class="w"></span>
<span class="p">]</span><span class="w"></span>
</code></pre></div>
</figure>
<p>The <code>Cython >= 3.0a6</code> requirement is problematic
because <a href="https://cython.readthedocs.io/en/latest/src/changes.html#id12">Cython
3.0a8</a> has the following change:</p>
<blockquote>
<p>Variables can no longer be declared with cpdef.</p>
</blockquote>
<p>This can be overcome by using <code>reproducible-wheel-builder</code>
with a pex lockfile that pins <code>Cython</code> to <code>3.0a7</code>.
Generating a <code>pex</code> lockfile for the dependencies can be done
with the <code>pex3</code> command.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>pex3 lock create --pip-version <span class="m">22</span>.3 --resolver-version pip-2020-resolver <span class="se">\</span>
--no-build --indent <span class="m">2</span> -o pex.lock <span class="se">\</span>
<span class="s1">'setuptools >= 40.8.0'</span> <span class="s1">'wheel'</span> <span class="s1">'Cython == 3.0a7'</span> <span class="s1">'cffi >= 1.12.3'</span> <span class="s1">'greenlet >= 0.4.17, < 2.0'</span>
</code></pre></div>
</figure>
<p>With the <code>pex.lock</code> file and the <code>main.pex</code> of
<code>reproducible-wheel-builder</code> a wheel can be built now.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>./main.pex --lock pex.lock --src ./gevent-21.1.2 --out ./out --quiet
</code></pre></div>
</figure>
<p>The above command successfully runs and outputs a wheel under
<code>./out</code>.</p>
<h2 id="conclusion">Conclusion<a class="pilcrow" href="#conclusion"></a></h2>
<p>PEP 517 and PEP 518 don’t specify a lockfile mechanism for specifying
the build environment. This leads to non determinism and possibly failed
builds when using <code>pip wheel</code>. This can be overcome with a
pex lockfile of the build environment and passing it to <a href="https://github.com/zmanji/reproducible-wheel-builder"><code>reproducible-wheel-builder</code></a>
to do the build.</p>
<section id="footnotes" class="footnotes footnotes-end-of-document" role="doc-endnotes">
<hr>
<ol>
<li id="fn1"><p>I am not good at naming things<a href="#fnref1" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
</ol>
</section>
tag:zameermanji.com,2023-01-14:/blog/2023/1/14/working-around-a-failing-configure-script/Working around a failing configure script2023-01-14T00:00:00Z2023-01-14T00:00:00Z<p>When building <a href="https://www.cairographics.org/">cairo</a> from
source the first step is to run the <code>./configure</code> script.
This script is generated by <a href="https://www.gnu.org/software/autoconf/">GNU Autoconfig</a> and the
purpose is to probe the local system for various features and settings
which are encoded in Makfiles that are used to build cairo. When using
the <a href="https://ziglang.org/learn/overview/#zig-is-also-a-c-compiler">zig
toolchain</a> to build cairo I get the following error from
<code>./configure</code>:</p>
<figure class="code-block shell"><div class="highlight"><pre><span></span><code>$ <span class="nv">CC</span><span class="o">=</span><span class="s1">'zig cc'</span> ./configure
...
checking whether float word ordering is bigendian... unknown
configure: error:
Unknown float word ordering. You need to manually preset
<span class="nv">ax_cv_c_float_words_bigendian</span><span class="o">=</span>no <span class="o">(</span>or yes<span class="o">)</span> according to your system.
</code></pre></div>
<figcaption><p>Error output from running <code>./configure</code> with the zig
toolchain</p></figcaption></figure>
<p>The error message is pretty unhelpful because it doesn’t say what the
check is testing for and how to determine the correct value. Fortunately
it’s possible to look up every configure check in the <a href="https://www.gnu.org/software/autoconf-archive/">Autoconf
Archive</a>. The <a href="https://www.gnu.org/software/autoconf-archive/ax_c_float_words_bigendian.html"><code>ax_cv_c_float_words_bigendian</code></a>
check is trying to determine the endianess of floats. I don’t know what
the correct answer is for <code>x86_64</code> systems, but it is
possible to replicate the check by looking at the <a href="https://git.savannah.gnu.org/gitweb/?p=autoconf-archive.git;a=blob_plain;f=m4/ax_c_float_words_bigendian.m4">source</a>
of the check. In this case it’s trying to compile the following C
program and checks if <code>noonsees</code> or <code>seesnoon</code> is
in the resulting binary.</p>
<figure class="code-block C"><div class="highlight"><pre><span></span><code><span class="kt">int</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kt">double</span><span class="w"> </span><span class="n">d</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mf">90904234967036810337470478905505011476211692735615632014797120844053488865816695273723469097858056257517020191247487429516932130503560650002327564517570778480236724525140520121371739201496540132640109977779420565776568942592.0</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
</figure>
<p>Compiling the above file on an <code>x86_64</code> system and
inspecting the resulting binary shows <code>seesnoon</code> is in the
resulting binary.</p>
<figure class="code-block shell"><div class="highlight"><pre><span></span><code>$ zig cc test.c
$ strings a.out <span class="p">|</span> grep sees
seesnoon
</code></pre></div>
</figure>
<p>According to the source this means that the float ordering is little
endian and the correct value of
<code>ax_cv_c_float_words_bigendian</code> is <code>no</code>.</p>
<p>Calling <code>./configure</code> with
<code>ax_cv_c_float_words_bigendian=no</code> as an argument now skips
the failing check and causes the <code>./configure</code> script to exit
successfully.</p>
<figure class="code-block shell"><div class="highlight"><pre><span></span><code>$ <span class="nv">CC</span><span class="o">=</span><span class="s1">'zig cc'</span> ./configure <span class="nv">ax_cv_c_float_words_bigendian</span><span class="o">=</span>no
...
cairo <span class="o">(</span>version <span class="m">1</span>.16.0 <span class="o">[</span>release<span class="o">])</span> will be compiled with:
...
</code></pre></div>
</figure>
<h2 id="conclusion">Conclusion<a class="pilcrow" href="#conclusion"></a></h2>
<p>It’s easy to work around a failing configure check. Every check is
documented in the <a href="https://www.gnu.org/software/autoconf-archive/">Autoconf
Archive</a> with a simple description and a link to the source code.
With the documentation and source it’s possible to replicate the check
and determine the correct value for the current system. The correct
value can be passed into the <code>./configure</code> script to skip the
check.</p>
tag:zameermanji.com,2023-01-10:/blog/2023/1/10/building-and-installing-an-npm-package-offline/Building and installing an NPM package offline2023-01-10T00:00:00Z2023-01-10T00:00:00Z<p>Recently I had to build an NPM package in an environment that did not
have any network access <a href="#fn1" class="footnote-ref" id="fnref1" role="doc-noteref"><sup>1</sup></a>. Normally if you run
<code>npm install --offline</code> the build would normally fail with
the following error.</p>
<figure class="code-block shell"><div class="highlight"><pre><span></span><code>npm ERR! code ENOTCACHED
</code></pre></div>
</figure>
<p>This happens because NPM’s cache is not hydrated.</p>
<p>However it is possible to build the package if it has a
<code>package-lock.json</code> and if all of the dependencies specified
in the lock file are placed into NPM’s cache before the build. This blog
post outlines how to do that so a NPM package can be built and installed
in an offline environment.</p>
<h2 id="example-package">Example package<a class="pilcrow" href="#example-package"></a></h2>
<p>For this blog post I have setup a very basic NPM package with a
single dependency on the <a href="https://www.npmjs.com/package/chalk">chalk package</a>. The
package is composed of the following files.</p>
<figure class="code-block shell"><div class="highlight"><pre><span></span><code>$ ls
index.js package-lock.json package.json
</code></pre></div>
<figcaption><p>The contents of the example package.</p></figcaption></figure>
<p>The <code>index.js</code> contains:</p>
<figure class="code-block javascript"><div class="highlight"><pre><span></span><code><span class="ch">#!/usr/bin/env node</span>
<span class="k">import</span> <span class="nx">chalk</span> <span class="kr">from</span> <span class="s1">'chalk'</span><span class="p">;</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">chalk</span><span class="p">.</span><span class="nx">magenta</span><span class="p">(</span><span class="s1">'Hello world!'</span><span class="p">));</span>
</code></pre></div>
</figure>
<p>The <code>package.json</code> contains:</p>
<figure class="code-block json"><div class="highlight"><pre><span></span><code><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"example-package"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"private"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"0.0.0"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"main"</span><span class="p">:</span><span class="w"> </span><span class="s2">"index.js"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"module"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"bin"</span><span class="p">:</span><span class="w"> </span><span class="s2">"index.js"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"dependencies"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nt">"chalk"</span><span class="p">:</span><span class="w"> </span><span class="s2">"5.2.0"</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
</figure>
<p>The <code>package-lock.json</code> was generated by running
<code>npm i --package-lock-only</code>.</p>
<h2 id="building-offline-with-docker">Building offline with Docker<a class="pilcrow" href="#building-offline-with-docker"></a></h2>
<p>Setting up an offline build environment with Docker is straight
forward with the following <code>Dockerfile</code>.</p>
<figure class="code-block dockerfile"><div class="highlight"><pre><span></span><code><span class="c"># syntax-docker/dockerfile:1</span>
<span class="k">FROM</span><span class="w"> </span><span class="s">node:lts-hydrogen</span>
<span class="k">WORKDIR</span><span class="w"> </span><span class="s">/app</span>
<span class="k">COPY</span><span class="w"> </span>package.json /app
<span class="k">COPY</span><span class="w"> </span>index.js /app
<span class="k">COPY</span><span class="w"> </span>package-lock.json /app
<span class="k">RUN</span><span class="w"> </span>npm i --offline
</code></pre></div>
</figure>
<p>Then running <code>docker build --network none .</code> triggers a
build without network access and returns the <code>ENOTCACHED</code>
error.</p>
<figure class="code-block shell"><div class="highlight"><pre><span></span><code>$ docker build --network none .
<span class="o">=</span>> ERROR <span class="o">[</span><span class="m">6</span>/6<span class="o">]</span> RUN npm i --offline
------
> <span class="o">[</span><span class="m">6</span>/6<span class="o">]</span> RUN npm i --offline:
<span class="c1">#10 0.338 npm ERR! code ENOTCACHED</span>
<span class="c1">#10 0.339 npm ERR! request to https://registry.npmjs.org/chalk/-/chalk-5.2.0.tgz failed: cache mode is 'only-if-cached' but no cached response is available.</span>
<span class="c1">#10 0.340</span>
</code></pre></div>
</figure>
<h2 id="hydrating-the-cache">Hydrating the cache<a class="pilcrow" href="#hydrating-the-cache"></a></h2>
<p>To overcome the <code>ENOTCACHED</code> dependency the
<code>chalk-5.2.0.tgz</code> file needs to be already placed into the
cache before the build starts. Assuming <code>chalk-5.2.0.tgz</code> is
already available then using <code>npm cache add</code> before
<code>npm install -offline</code> fixes the problem. The
<code>Dockerfile</code> can be changed to:</p>
<figure class="code-block dockerfile"><div class="highlight"><pre><span></span><code><span class="c"># syntax-docker/dockerfile:1</span>
<span class="k">FROM</span><span class="w"> </span><span class="s">node:lts-hydrogen</span>
<span class="k">WORKDIR</span><span class="w"> </span><span class="s">/tmp/</span>
<span class="c"># Copy dependencies</span>
<span class="k">COPY</span><span class="w"> </span>chalk-5.2.0.tgz /tmp/
<span class="k">RUN</span><span class="w"> </span>npm cache add chalk-5.2.0.tgz
<span class="k">WORKDIR</span><span class="w"> </span><span class="s">/app</span>
<span class="k">COPY</span><span class="w"> </span>package.json /app
<span class="k">COPY</span><span class="w"> </span>index.js /app
<span class="k">COPY</span><span class="w"> </span>package-lock.json /app
<span class="k">RUN</span><span class="w"> </span>npm i --offline
</code></pre></div>
</figure>
<p>By calling <code>npm cache add</code> on every dependency specified
in the <code>package-lock.json</code> before the build, the build can
occur offline.</p>
<figure class="code-block shell"><div class="highlight"><pre><span></span><code>$ docker build --no-cache --network none .
<span class="o">[</span>+<span class="o">]</span> Building <span class="m">1</span>.4s <span class="o">(</span><span class="m">14</span>/14<span class="o">)</span> <span class="nv">FINISHED</span>
<span class="o">=</span>> <span class="o">[</span>internal<span class="o">]</span> load build definition from Dockerfile <span class="m">0</span>.0s
<span class="o">=</span>> <span class="o">=</span>> transferring dockerfile: 38B <span class="m">0</span>.0s
<span class="o">=</span>> <span class="o">[</span>internal<span class="o">]</span> load .dockerignore <span class="m">0</span>.0s
<span class="o">=</span>> <span class="o">=</span>> transferring context: 2B <span class="m">0</span>.0s
<span class="o">=</span>> <span class="o">[</span>internal<span class="o">]</span> load metadata <span class="k">for</span> docker.io/library/node:lts-hydrogen <span class="m">0</span>.1s
<span class="o">=</span>> <span class="o">[</span><span class="m">1</span>/9<span class="o">]</span> FROM docker.io/library/node:lts-hydrogen@sha256:a403ff0ffe7a6a8fe90fdc70289ba39 <span class="m">0</span>.0s
<span class="o">=</span>> <span class="o">[</span>internal<span class="o">]</span> load build context <span class="m">0</span>.0s
<span class="o">=</span>> <span class="o">=</span>> transferring context: 133B <span class="m">0</span>.0s
<span class="o">=</span>> CACHED <span class="o">[</span><span class="m">2</span>/9<span class="o">]</span> WORKDIR /tmp/ <span class="m">0</span>.0s
<span class="o">=</span>> <span class="o">[</span><span class="m">3</span>/9<span class="o">]</span> COPY chalk-5.2.0.tgz /tmp/ <span class="m">0</span>.0s
<span class="o">=</span>> <span class="o">[</span><span class="m">4</span>/9<span class="o">]</span> RUN npm cache add chalk-5.2.0.tgz <span class="m">0</span>.3s
<span class="o">=</span>> <span class="o">[</span><span class="m">5</span>/9<span class="o">]</span> WORKDIR /app <span class="m">0</span>.0s
<span class="o">=</span>> <span class="o">[</span><span class="m">6</span>/9<span class="o">]</span> COPY package.json /app <span class="m">0</span>.1s
<span class="o">=</span>> <span class="o">[</span><span class="m">7</span>/9<span class="o">]</span> COPY index.js /app <span class="m">0</span>.0s
<span class="o">=</span>> <span class="o">[</span><span class="m">8</span>/9<span class="o">]</span> COPY package-lock.json /app <span class="m">0</span>.0s
<span class="o">=</span>> <span class="o">[</span><span class="m">9</span>/9<span class="o">]</span> RUN npm i --offline <span class="m">0</span>.4s
<span class="o">=</span>> exporting to image <span class="m">0</span>.2s
<span class="o">=</span>> <span class="o">=</span>> exporting layers <span class="m">0</span>.2s
<span class="o">=</span>> <span class="o">=</span>> writing image sha256:dcf3e70d01cabfd9a6065ac5990498865e623bce3a0c890ea8a9a5e424956 <span class="m">0</span>.0s
</code></pre></div>
</figure>
<p>Changing the entrypoint to <code>/app/index.js</code> shows the
package was installed correctly.</p>
<figure class="code-block shell"><div class="highlight"><pre><span></span><code>docker run -it --entrypoint /app/index.js --rm sha256:ea603bac106fe180070b608537a2f34b7ba3ce6e31018331b06f7437f781e
Hello world!
</code></pre></div>
</figure>
<h2 id="conclusion">Conclusion<a class="pilcrow" href="#conclusion"></a></h2>
<p>It’s possible to build NPM packages in an offline environment so long
as the package has a <code>package-lock.json</code> and all of the
locked dependencies are added to the NPM cache prior to running
<code>npm install --offline</code>. This enables building NPM packages
in constrained environments using standard tooling.</p>
<section id="footnotes" class="footnotes footnotes-end-of-document" role="doc-endnotes">
<hr>
<ol>
<li id="fn1"><p>A <a href="https://github.com/NixOS/nix">nix</a> build
sandbox.<a href="#fnref1" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
</ol>
</section>
tag:zameermanji.com,2022-09-17:/blog/2022/9/17/specifying-a-relative-interpreter-in-a-shebang/Specifying a relative interpreter in a shebang2022-09-17T00:00:00Z2022-09-17T00:00:00Z<h2 id="background">Background<a class="pilcrow" href="#background"></a></h2>
<p>In order to make a Python script executable it needs to have a
shebang line specifying the interpreter of the script.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>head myscript.py -n <span class="m">1</span>
<span class="gp">#</span>!/usr/bin/python3
</code></pre></div>
</figure>
<p>This enables passing the script directly to <code>execve</code> where
the kernel can eventually call <code>/usr/bin/python3</code> with the
script file as an argument. This works well but has the drawback of
harcoding the interpreter. If a different interpreter is needed, like
one located in a virtual environment, it cannot be used.</p>
<h2 id="using-env">Using <code>env</code><a class="pilcrow" href="#using-env"></a></h2>
<p>Instead of hardcoding an interpreter, the shebang line can use <a href="https://man7.org/linux/man-pages/man1/env.1.html"><code>env(1)</code></a>.
Passing <code>env</code> an argument cases <code>env</code> to lookup
that name in the <code>$PATH</code> and execute that as the
interpreter.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>head myscript.py -n <span class="m">1</span>
<span class="gp">#</span>!/usr/bin/env python3
</code></pre></div>
</figure>
<p>The user can then move their desired <code>python3</code> binary to
the front of the <code>$PATH</code> and that will be used to execute the
script.</p>
<h2 id="limitations-of-env">Limitations of <code>env</code><a class="pilcrow" href="#limitations-of-env"></a></h2>
<p>There are two limitations with using <code>env</code> like above.
First, it’s not possible to pass arguments to <code>python3</code> via
<code>env</code> because <code>env</code> interprets the additional
arguments as the name of the interpreter. For example trying to pass the
<code>-E</code> flag to <code>python3</code> results in the following
error.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>head myscript.py -n <span class="m">1</span>
<span class="gp">#</span>!/usr/bin/env python3 -E
<span class="gp">$ </span>./myscript.py
<span class="go">/usr/bin/env: ‘python3 -E’: No such file or directory</span>
</code></pre></div>
</figure>
<p>This is because <code>env</code> does not split the arguments by
spaces before passing them to <code>execve</code>.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>strace --trace<span class="o">=</span>execve ./myscript.py
<span class="go">execve("./myscript.py", ["./myscript.py"], 0x7ffde39948c8 /* 49 vars */) = 0</span>
<span class="go">execve("/usr/local/bin/python3 -E", ["python3 -E", "./myscript.py"], 0x7ffdb12cf6c8 /* 49 vars */) = -1 ENOENT (No such file or directory)</span>
<span class="go">execve("/usr/bin/python3 -E", ["python3 -E", "./myscript.py"], 0x7ffdb12cf6c8 /* 49 vars */) = -1 ENOENT (No such file or directory)</span>
<span class="go">execve("/bin/python3 -E", ["python3 -E", "./myscript.py"], 0x7ffdb12cf6c8 /* 49 vars */) = -1 ENOENT (No such file or directory)</span>
<span class="go">/usr/bin/env: ‘python3 -E’: No such file or directory</span>
</code></pre></div>
</figure>
<p>Second, this approach assumes the desired interpreter is on the front
of the <code>$PATH</code>. If it’s not possible to put the desired
interpreter on the <code>$PATH</code> then the correct interpreter will
not be used.</p>
<h2 id="using-env--s">Using <code>env -S</code><a class="pilcrow" href="#using-env--s"></a></h2>
<p>It’s possible to overcome the above limitations by using the
<code>-S</code> flag to <code>env</code>. The <a href="https://www.gnu.org/software/coreutils/manual/html_node/env-invocation.html">documentation</a>
explains that the <code>-S</code> flag will split arguments on
whitespace and pass them all to <code>execve</code>. This means instead
of just specifying a binary name, all of the values of
<code>execve</code> can be specified, such as specifying <code>-E</code>
to the interpreter.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>head myscript.py -n <span class="m">1</span>
<span class="gp">#</span>!/usr/bin/env -S python3 -E
</code></pre></div>
</figure>
<p>This can be verified by passing <code>-v</code> to get verbose output
from <code>env</code>.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>head myscript.py -n <span class="m">1</span>
<span class="gp">#</span>!/usr/bin/env -vS python3 -E
<span class="gp">$ </span>./myscript.py
<span class="go">split -S: ‘python3 -E’</span>
<span class="go"> into: ‘python3’</span>
<span class="go"> & ‘-E’</span>
<span class="go">executing: python3</span>
<span class="go"> arg[0]= ‘python3’</span>
<span class="go"> arg[1]= ‘-E’</span>
<span class="go"> arg[2]= ‘./myscript.py’</span>
<span class="go">...</span>
</code></pre></div>
</figure>
<h2 id="selecting-a-relative-interpreter">Selecting a relative
interpreter<a class="pilcrow" href="#selecting-a-relative-interpreter"></a></h2>
<p>Since the <code>-S</code> flag allows us to pass full arguments to an
interpreter, it’s possible then to encode an entire program in the
shebang line as an argument to another interpreter and have that execute
our desired interpreter.</p>
<p>For example we could use <code>env -S</code> to execute
<code>/bin/sh</code> and pass in a full script to <code>-c</code>, where
that script locates the desired interpreter relative to the script and
then executes that. The <a href="https://www.gnu.org/software/coreutils/manual/html_node/env-invocation.html">documentation</a>
explains that spaces can be escaped with <code>\_</code>.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>head myscript.py -n <span class="m">1</span>
<span class="gp">#</span>!/usr/bin/env -vS<span class="se">\_</span>/bin/sh<span class="se">\_</span>-xc<span class="se">\_</span><span class="s2">"exec\_\$(dirname\_"</span><span class="se">\$</span><span class="m">0</span><span class="s2">")/venv/bin/python3\_-E\_"</span><span class="se">\$</span><span class="m">0</span><span class="s2">"\_"</span><span class="se">\$</span>@<span class="s2">""</span>$ ./myscript.py
<span class="go">split -S: ‘\\_/bin/sh\\_-xc\\_"exec\\_\\$(dirname\\_"\\$0")/venv/bin/python3\\_-E\\_"\\$0"\\_"\\$@""’</span>
<span class="go"> into: ‘/bin/sh’</span>
<span class="go"> & ‘-xc’</span>
<span class="go"> & ‘exec $(dirname $0)/venv/bin/python3 -E $0 $@’</span>
<span class="go">executing: /bin/sh</span>
<span class="go"> arg[0]= ‘/bin/sh’</span>
<span class="go"> arg[1]= ‘-xc’</span>
<span class="go"> arg[2]= ‘exec $(dirname $0)/venv/bin/python3 -E $0 $@’</span>
<span class="go"> arg[3]= ‘./myscript.py’</span>
<span class="go">+ dirname ./myscript.py</span>
<span class="go">+ exec ./venv/bin/python3 -E ./myscript.py...</span>
</code></pre></div>
</figure>
<h2 id="conclusion">Conclusion<a class="pilcrow" href="#conclusion"></a></h2>
<p>By combining <code>env -S</code> and <code>sh</code> it’s possible to
create a shebang line that executes an interpreter relative to the
script. This could be used to have a script execute in a specific
virtual environment.</p>
tag:zameermanji.com,2022-08-17:/blog/2022/8/17/how-to-use-github-to-host-python-wheels/How to use GitHub to host Python wheels2022-08-17T00:00:00Z2022-08-17T00:00:00Z<p>When using Python libraries that contain C extensions I have needed
to build my own wheel instead of using an existing wheel on PyPI. The
typical reason is to rebuild an existing release but with different
flags to change the compiled C extension. This allows me to:</p>
<ul>
<li>Increase debug information to debug a crash.</li>
<li>Have the wheel dynamically link against a system library instead of
statically linking.</li>
<li>Change compiler or linker flags to change optimization levels.</li>
<li>Compile an older version of the wheel for a version of Python that
was released after the library was released.</li>
</ul>
<p>In a large enterprise environment it would be easy to upload the
wheels to a custom PyPI mirror and configure it to prefer uploaded
wheels over external wheels. However when working on a solo project or a
hobby project something less cumbersome is required, especially when a
single wheel needs to be rebuilt.</p>
<p>Instead of creating a custom PyPI mirror, it’s possible to host the
wheels on a GitHub release and configure <code>pip</code> to prefer
those wheels over the wheels in PyPI. To make it happen there are two
features needed to be combined:</p>
<ul>
<li>Leverage that wheels have a ‘build number’ and
<code>setuptools</code> allow setting the build number via
<code>bdist_wheel</code>.</li>
<li><code>pip</code> supports combining using an index like PyPI with
it’s own <code>--find-links</code> mechanism to supplement the
index.</li>
</ul>
<p>For this post I will be using <a href="https://github.com/animalize/pyzstd"><code>pyzstd</code></a> as an
example where I have <a href="https://github.com/zmanji/pyzstd-wheel-builder/blob/75ed8478e374a151a531a417ff15903f029ccd98/build.py#L39">rebuilt
it with dynamic linking</a>.</p>
<h2 id="setting-a-custom-build-number">Setting a custom build
number<a class="pilcrow" href="#setting-a-custom-build-number"></a></h2>
<p>To set a build number, the wheel needs to be built with the
<code>bdist_wheel</code> command. The <code>bdist_wheel</code> command
has a <code>--build-number</code> flag which can be used to set the
build number of the wheel.</p>
<p>For example with the <code>pyzstd</code> wheel without the
<code>--build-number</code> set looks like so.</p>
<figure class="code-block text"><div class="highlight"><pre><span></span><code>pyzstd-0.15.2-cp310-cp310-linux_x86_64.whl
</code></pre></div>
</figure>
<p>Where as setting the <code>--build-number</code> to <code>1</code>
when running <code>bdist_wheel</code> will produce a name like so.</p>
<figure class="code-block text"><div class="highlight"><pre><span></span><code>pyzstd-0.15.2-1-cp310-cp310-linux_x86_64.whl
</code></pre></div>
</figure>
<p>From the <a href="https://packaging.python.org/en/latest/specifications/binary-distribution-format/">wheel
specification</a> the build number acts as a tie breaker.</p>
<blockquote>
<p>Optional build number. Must start with a digit. Acts as a tie-breaker
if two wheel file names are the same in all other respects (i.e. name,
version, and other tags). Sort as an empty tuple if unspecified, else
sort as a two-item tuple with the first item being the initial digits as
an int, and the second item being the remainder of the tag as a str.</p>
</blockquote>
<p>Adding this build number means the wheel with the build number will
always be preferred over one without, which means it will be preferred
over the wheels on PyPI. Further it’s better than changing the version
number, since no code changes were made.</p>
<h2 id="using---find-links-to-discover-the-wheel">Using
<code>--find-links</code> to discover the wheel<a class="pilcrow" href="#using---find-links-to-discover-the-wheel"></a></h2>
<p>To have <code>pip</code> discover the wheels, assuming they are
uploaded to a GitHub release page, is to pass the URL to the
<code>--find-links</code> flag. With <code>pip</code>
<code>20.3.3</code> it’s possible to run the following command.</p>
<figure class="code-block shell"><div class="highlight"><pre><span></span><code>$ pip install --no-cache --only-binary <span class="s1">':all:'</span> --find-links <span class="s1">'https://github.com/zmanji/pyzstd-wheel-builder/releases/tag/v0.0.7'</span> <span class="s1">'pyzstd==0.15.2'</span>
Looking <span class="k">in</span> links: https://github.com/zmanji/pyzstd-wheel-builder/releases/tag/v0.0.7
Collecting <span class="nv">pyzstd</span><span class="o">==</span><span class="m">0</span>.15.2
Downloading https://github.com/zmanji/pyzstd-wheel-builder/releases/download/v0.0.7/pyzstd-0.15.2-1-cp310-cp310-linux_x86_64.whl <span class="o">(</span><span class="m">42</span> kB<span class="o">)</span>
<span class="p">|</span>████████████████████████████████<span class="p">|</span> <span class="m">42</span> kB <span class="m">5</span>.1 MB/s
Installing collected packages: pyzstd
Successfully installed pyzstd-0.15.2
</code></pre></div>
</figure>
<p>Note that in the current version of <code>pip</code>, pip no longer
prefers wheels found in <code>--find-links</code> over PyPI due <a href="https://github.com/pypa/pip/issues/9959">to this issue</a>. To fix
this the above command needs to have <code>--no-index</code>.</p>
<figure class="code-block shell"><div class="highlight"><pre><span></span><code>$ pip install --no-cache --only-binary <span class="s1">':all:'</span> --find-links <span class="s1">'https://github.com/zmanji/pyzstd-wheel-builder/releases/tag/v0.0.7'</span> <span class="s1">'pyzstd==0.15.2'</span> --no-index
Looking <span class="k">in</span> links: https://github.com/zmanji/pyzstd-wheel-builder/releases/tag/v0.0.7
Collecting <span class="nv">pyzstd</span><span class="o">==</span><span class="m">0</span>.15.2
Downloading https://github.com/zmanji/pyzstd-wheel-builder/releases/download/v0.0.7/pyzstd-0.15.2-1-cp310-cp310-linux_x86_64.whl <span class="o">(</span><span class="m">42</span> kB<span class="o">)</span>
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ <span class="m">42</span>.5/42.5 kB <span class="m">5</span>.1 MB/s eta <span class="m">0</span>:00:00
Installing collected packages: pyzstd
Successfully installed pyzstd-0.15.2
</code></pre></div>
</figure>
<p>Alternatively a newer version of <code>pip</code> can be used in
conjunction with <a href="https://github.com/uranusjr/simpleindex"><code>simpleindex</code></a>
where <code>simpleindex</code> doesn’t offer the library at all and pip
is forced to pull the wheel from the <code>--find-links</code> argument.
This can be done by creating a <code>simpleindex</code> configuration
that looks like</p>
<figure class="code-block toml"><div class="highlight"><pre><span></span><code><span class="k">[routes."pyzstd"]</span><span class="w"></span>
<span class="n">source</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"path"</span><span class="w"></span>
<span class="n">to</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"/dev/null"</span><span class="w"></span>
<span class="k">[routes."{project}"]</span><span class="w"></span>
<span class="n">source</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"http"</span><span class="w"></span>
<span class="n">to</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"https://pypi.org/simple/{project}/"</span><span class="w"></span>
<span class="k">[server]</span><span class="w"></span>
<span class="n">host</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"127.0.0.1"</span><span class="w"></span>
<span class="n">port</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">8000</span><span class="w"></span>
</code></pre></div>
</figure>
<p>Running <code>simpleindex</code> with the above configuration and
pointing the latest <code>pip</code> to it and passing
<code>--find-links</code> results in a successful install.</p>
<figure class="code-block shell"><div class="highlight"><pre><span></span><code>$ pip install --no-cache --only-binary <span class="s1">':all:'</span> -i <span class="s1">'http://127.0.0.1:8000'</span> --find-links <span class="s1">'https://github.com/zmanji/pyzstd-wheel-builder/releases/tag/v0.0.7'</span> <span class="s1">'pyzstd==0.15.2'</span>
Looking <span class="k">in</span> indexes: http://127.0.0.1:8000
Looking <span class="k">in</span> links: https://github.com/zmanji/pyzstd-wheel-builder/releases/tag/v0.0.7
Collecting <span class="nv">pyzstd</span><span class="o">==</span><span class="m">0</span>.15.2
Downloading https://github.com/zmanji/pyzstd-wheel-builder/releases/download/v0.0.7/pyzstd-0.15.2-1-cp310-cp310-linux_x86_64.whl <span class="o">(</span><span class="m">42</span> kB<span class="o">)</span>
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ <span class="m">42</span>.5/42.5 kB <span class="m">4</span>.3 MB/s eta <span class="m">0</span>:00:00
Installing collected packages: pyzstd
Successfully installed pyzstd-0.15.2
</code></pre></div>
</figure>
<h1 id="conclusion">Conclusion</h1>
<p>It’s possible to rebuild a Python library wheel with different flags
as a different artifact without changing the version number by using the
<code>--build-number</code> flag of <code>bdist_wheel</code>. After
uploading the wheels to a GitHub release the wheels can be consumed
easily with the <code>--find-links</code> flag with
<code>pip</code>.</p>
tag:zameermanji.com,2022-08-05:/blog/2022/8/5/using-fuse-without-root-on-linux/Using FUSE without root on Linux2022-08-05T00:00:00Z2022-08-05T00:00:00Z<p>I recently wrote a FUSE driver for Linux. My FUSE driver had to work
in an environment where root access was not permitted and SUID binaries
were also not permitted. Overcoming this constraint was tricky and this
blog post outlines how I did it.</p>
<h2 id="how-does-fuse-work">How does FUSE work?<a class="pilcrow" href="#how-does-fuse-work"></a></h2>
<p>FUSE on Linux requires the driver to conduct a very specific sequence
in order to create and mount the filesystem. First the driver needs to
open <code>/dev/fuse</code> for reading and writing. Opening the device
returns a file descriptor which will be used to communicate with the
kernel.</p>
<p>Next, the driver needs to initiate a <a href="https://man7.org/linux/man-pages/man2/mount.2.html"><code>mount(2)</code></a>
system call for a <code>fuse</code> filesystem and provide the desired
mountpoint. The driver also needs to provide an <code>fd</code> mount
option which is set to the file descriptor from above.</p>
<p>After, the above the driver can use the file descriptor to
communicate with the kernel. The kernel implements a RPC protocol over
the file descriptor which tells the driver which files are opened,
closed, etc.</p>
<p>The problem with this sequence is that issuing a
<code>mount(2)</code> system call requires the caller to have
<code>CAP_SYS_ADMIN</code> which a regular user does not have. Without
it, it’s not possible for a fuse driver to actually make the filesystem
available.</p>
<h2 id="typical-approach">Typical Approach<a class="pilcrow" href="#typical-approach"></a></h2>
<p>The typical way to solve this is to implement a FUSE driver using <a href="https://github.com/libfuse/libfuse">libfuse</a>. This library
allows a user to implement each file system operation as a C callback.
The user implements these functions and passes them to a libfuse main
function which does the mounting described above. It overcomes the
privilege issue above by being bundled with a SUID helper binary called
<code>fusermount</code>.</p>
<p><code>libfuse</code> expects <code>fusermount</code> binary to exist
and delegates the mounting operation to <code>fusermount</code>. A
driver using <code>libfuse</code> actually allocates a local unix domain
socket, passes one of them to <code>fusermount</code> which then
performs the <code>mount(2)</code> call and proxies the I/O between
<code>/dev/fuse</code> and the driver. The <code>fusermount</code>
process runs as long as the filesystem is mounted.</p>
<p>This is easy to observe by using a driver that uses
<code>libfuse</code>. For example mounting a filesystem with
<code>squashfuse</code>.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>squashfuse -o auto_unmount,allow_other,default_permissions ./alpine-minirootfs-3.15.4-x86_64.sqfs mount
</code></pre></div>
</figure>
<p>Once this command is run two processes will have been launched:</p>
<figure class="code-block text"><div class="highlight"><pre><span></span><code>root 2132739 0.0 0.0 2792 1032 ? Ss 03:02 0:00 fusermount -o rw,nosuid,nodev,allow_other,default_permissions,auto_unmount,subtype=squashfuse -- /home/zmanji/tmp/mount
zmanji 2132741 0.0 0.0 4752 344 ? Ss 03:02 0:00 squashfuse -o auto_unmount,allow_other,default_permissions ./alpine-minirootfs-3.15.4-x86_64.sqfs mount
</code></pre></div>
</figure>
<p>and they have established a bi-directional unix socket pair between
them.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>sudo lsof +E -aUc fuse
<span class="go">COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME</span>
<span class="go">fusermoun 2132739 root 4u unix 0xffff9d931e79a640 0t0 308828 type=STREAM ->INO=308829 2132741,squashfus,5u</span>
<span class="go">squashfus 2132741 zmanji 5u unix 0xffff9d931e79d940 0t0 308829 type=STREAM ->INO=308828 2132739,fusermoun,4u</span>
</code></pre></div>
</figure>
<p>We can see that this approach will fail to work if
<code>fusermount</code> can not execute.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>sudo chmod -x /usr/bin/fusermount3
<span class="gp">$ </span>squashfuse ./alpine-minirootfs-3.15.4-x86_64.sqfs mount
<span class="go">fuse: failed to exec fusermount: Permission denied</span>
</code></pre></div>
<figcaption><p>Preventing <code>fusermount</code> from executing prevents
<code>libfuse</code> from working at all</p></figcaption></figure>
<h2 id="alternative-approach">Alternative Approach<a class="pilcrow" href="#alternative-approach"></a></h2>
<p>The alternative to the above is to take advantage of a feature of
Linux introduced in 4.18: <a href="https://github.com/torvalds/linux/commit/4ad769f3c346ec3d458e255548dec26ca5284cf6">fuse:
Allow fully unprivileged mounts</a>. This change allows a root in a user
namespace to issue <code>mount(2)</code> for a FUSE filesystem.</p>
<p>With this any regular user can call <code>mount(2)</code> if they
created a new user and mount namespace. This technique even works with
<code>libfuse</code> which tries to use <code>mount(2)</code> directly
before falling back to using <code>fusermount</code>. Simply creating a
new user, pid and mount namespaces with <code>unshare</code> with the
same restrictions as before works.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>sudo chmod -x /usr/bin/fusermount3
<span class="gp">$ </span>squashfuse ./alpine-minirootfs-3.15.4-x86_64.sqfs mount
<span class="go">fuse: failed to exec fusermount: Permission denied</span>
<span class="gp">$ </span>unshare -pfr --user --mount --kill-child /bin/bash
<span class="gp"># </span>squashfuse ./alpine-minirootfs-3.15.4-x86_64.sqfs mount
<span class="gp"># </span>ls mount
<span class="go">bin dev etc home lib media mnt opt proc root run sbin srv sys tmp usr var</span>
</code></pre></div>
</figure>
<p>With the mount in a different namespace however it’s not possible to
see the contents of the mount outside of the namespace. With the above
mount, another shell session would only see an empty directory at the
mount point.</p>
<p>This is because <a href="https://man7.org/linux/man-pages/man7/mount_namespaces.7.html"><code>mount_namespaces(7)</code></a>
comes with two restrictions:</p>
<blockquote>
<pre><code> [1] Each mount namespace has an owner user namespace. As
explained above, when a new mount namespace is created, its
mount list is initialized as a copy of the mount list of
another mount namespace. If the new namespace and the
namespace from which the mount list was copied are owned by
different user namespaces, then the new mount namespace is
considered less privileged.</code></pre>
</blockquote>
<blockquote>
<pre><code> [2] When creating a less privileged mount namespace, shared
mounts are reduced to slave mounts. This ensures that
mappings performed in less privileged mount namespaces will
not propagate to more privileged mount namespaces.</code></pre>
</blockquote>
<p>The combination of the two points means it’s impossible to bind mount
the mount point in the mount namespace to the root namespace.</p>
<p>However the owner of the mount namespace can peek into the contents
of the mount from the root namespace through <code>procfs</code>. If the
user can get the pid of the process that called <code>mount(2)</code> or
any other process in the same mount namespace then running the following
will show the contents of the mount.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>ls /proc/2719682/root/home/zmanji/tmp/mount
<span class="go">bin dev etc home lib media mnt opt proc root run sbin srv sys tmp usr var</span>
</code></pre></div>
<figcaption><p>Viewing the mount in the mount namespace from outside the
namespace</p></figcaption></figure>
<h2 id="conclusion">Conclusion<a class="pilcrow" href="#conclusion"></a></h2>
<p>It’s possible to mount a FUSE filesystem without use of root
permissions or SUID binaries by doing the mount inside of a user
namespace. This comes with the issue that viewing the contents of the
mount from outside of the namespace is restricted to using
<code>procfs</code>.</p>
tag:zameermanji.com,2022-01-18:/blog/2022/1/18/ubuntu-21-10-makes-libgpiod-unusable/Ubuntu 21.10 makes libgpiod unusable2022-01-18T00:00:00Z2022-01-18T00:00:00Z<p>Linux provides two APIs for user space programs to interact with GPIO
lines, a legacy <code>sysfs</code> based API and a newer character
device based API using <code>ioctl</code>. The legacy <code>sysfs</code>
API <a href="https://www.kernel.org/doc/Documentation/ABI/obsolete/sysfs-gpio">is
documented</a> to be removed soon:</p>
<blockquote>
<p>This ABI is deprecated and will be removed after 2020. It is replaced
with the GPIO character device.</p>
</blockquote>
<p>The newer character device is controlled with <code>ioctl</code>
calls and has a complementary library called <a href="https://git.kernel.org/pub/scm/libs/libgpiod/libgpiod.git/"><code>libgpiod</code></a>.
It provides C library with C++ and Python bindings to manipulate GPIO
via the newer API. The library also ships with a few utility command
line tools like <code>gpioinfo</code>, <code>gpiofind</code> and
<code>gpiodetect</code> which expose basic functionality of
<code>libgpiod</code> over the CLI.</p>
<p>Unfortunately on Ubuntu 21.10 (Impish) <code>libgpiod</code> is not
usable at all. For example the <code>gpioinfo</code> tool fails with the
following error when trying to look up the details for a GPIO line.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>gpioinfo gpiochip1
<span class="go">gpioinfo: error creating line iterator: Invalid argument</span>
</code></pre></div>
</figure>
<p>Similarly <code>gpiofind</code> will fail as well with the same
error.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>gpiofind gpiochip0
<span class="go">gpiofind: error performing the line lookup: Invalid argument</span>
</code></pre></div>
</figure>
<p>The root cause of this is that there are <em>two</em>
<code>ioctl</code> based APIs. The first one was <a href="https://github.com/torvalds/linux/commit/3c702e9987e261042a07e43460a8148be254412e">introduced
in</a> Linux v4.6 and the second was <a href="https://github.com/torvalds/linux/commit/b53911aa872db462be2e5f1dd611b25c4c2e663b">introduced
in</a> Linux v5.10. The first one is <a href="https://github.com/torvalds/linux/commit/957ebb61a4761c1eb32c3f34db7ccfef2e1e95ae">behind
a configuration flag</a> which is called
<code>CONFIG_GPIO_CDEV_V1</code>.</p>
<p>Unfortunately Ubuntu 21.10 ships with <code>libpgiod</code> 1.6.2
which uses the v1 <code>ioctl</code> API. The V1 API is disabled in the
Ubuntu 21.10 generic arm64 kernel.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>grep GPIO_CDEV /boot/config-5.13.0-19-generic
<span class="go">CONFIG_GPIO_CDEV=y</span>
<span class="gp"># </span>CONFIG_GPIO_CDEV_V1 is not <span class="nb">set</span>
</code></pre></div>
</figure>
<p>As a consequence all of the <code>ioctl</code> calls that
<code>libgpiod</code> makes will result in an “Invalid Argument” error,
because the <code>ioctl</code> is not available.</p>
<p>There are two possible solutions:</p>
<ul>
<li><p>Build your own kernel with <code>CONFIG_GPIO_CDEV_V1</code>
enabled.</p></li>
<li><p>Use another library like <a href="https://github.com/warthog618/gpiod"><code>gpiod</code></a> which
uses the newer <code>ioctls</code>.</p></li>
</ul>
tag:zameermanji.com,2022-01-05:/blog/2022/1/5/a-fast-usb-storage-gadget-on-the-pine-a64/A fast USB storage gadget on the Pine A642022-01-05T00:00:00Z2022-01-05T00:00:00Z<p>While iterating on some code that runs on my <a href="https://www.pine64.org/devices/single-board-computers/pine-a64/">Pine
A64</a> I got tired of constantly moving the SD card between the Pine
A64 and my laptop to flash a new image. It’s cumbersome to move the SD
card back and forth, and also prevents remote development.</p>
<p>It turns out that it is possible turn the Pine64<a href="#fn1" class="footnote-ref" id="fnref1" role="doc-noteref"><sup>1</sup></a>
into a USB device which exposes the SD card<a href="#fn2" class="footnote-ref" id="fnref2" role="doc-noteref"><sup>2</sup></a> as
a mass storage device over USB. This allows the SD card to be accessed
over USB from another computer without removing it from the Pine A64.
There are two approaches to doing this, one relies on u-boot only and
another which relies using a Linux kernel module. This post covers how
to use both.</p>
<h2 id="fel-mode">FEL Mode<a class="pilcrow" href="#fel-mode"></a></h2>
<p>The key technology to this approach while keeping the SD card
untouched is the <a href="https://linux-sunxi.org/FEL">FEL mode</a> that
all Allwinner CPUs have. When there is no bootable media attached, the
boot ROM will turn the OTG USB port into a peripheral that has a special
protocol. This protocol allows the host to read and write to arbitrary
memory and execute code in memory.</p>
<p>The combination of these primitives means it’s possible to do the
following:</p>
<ul>
<li>Load the u-boot SPL and execute it</li>
<li>Load a full u-boot and execute it</li>
<li>Optionally load a Linux kernel image and initrd for u-boot to
load</li>
</ul>
<p>Since the Pine A64 can only boot from the SD card, entering FEL mode
is pretty easy since the boot ROM looks for a header on sector 16 of the
SD card, it’s easy to wipe out the boot header with the following
command:</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>dd <span class="k">if</span><span class="o">=</span>/dev/zero <span class="nv">of</span><span class="o">=</span>/dev/mmcblk0 <span class="nv">bs</span><span class="o">=</span><span class="m">1024</span> <span class="nv">count</span><span class="o">=</span><span class="m">1</span> <span class="nv">seek</span><span class="o">=</span><span class="m">8</span>
</code></pre></div>
<figcaption><p>The command to cause the next reboot to enter FEL mode</p></figcaption></figure>
<p>After running this and rebooting, the device will enter FEL mode.
Assuming the OTG port<a href="#fn3" class="footnote-ref" id="fnref3" role="doc-noteref"><sup>3</sup></a> is connected to another computer,
that computer should see a device with Allwinner’s USB vendor id
<code>0x1f3a</code>.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>lsusb -d 1f3a:
<span class="go">Bus 005 Device 066: ID 1f3a:efe8 Allwinner Technology sunxi SoC OTG connector in FEL/flashing mode</span>
</code></pre></div>
</figure>
<p>To interact the device in FEL mode, the primary tool is from the <a href="https://github.com/linux-sunxi/sunxi-tools">sunxi-tools</a>
repository. At the time of this post, building from latest master would
provide a <code>sunxi-fel</code> binary that can list the Pine A64 in
FEL mode.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>./sunxi-fel -l
<span class="go">USB device 005:066 Allwinner A64 92c001ba:44004620:78918300:4023030f</span>
</code></pre></div>
</figure>
<p><code>sunxi-fel</code> provides a <code>uboot</code> command that can
load u-boot into memory and boot into it.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>./sunxi-fel uboot u-boot-sunxi-with-spl.bin
</code></pre></div>
<figcaption><p>Loading u-boot over fel mode</p></figcaption></figure>
<p>The serial console should print out output from the SPL and then
eventually drop into the u-boot prompt.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="go">U-Boot SPL 2021.10 (Jan 01 1970 - 00:00:01 +0000)</span>
<span class="go">DRAM: 1024 MiB</span>
<span class="go">Trying to boot from FEL</span>
<span class="go">...</span>
</code></pre></div>
<figcaption><p>Output from the u-boot SPL as it starts execution over FEL mode.</p></figcaption></figure>
<h2 id="using-u-boot">Using U-Boot<a class="pilcrow" href="#using-u-boot"></a></h2>
<p>With u-boot it’s possible to expose the SD card over USB with the <a href="https://u-boot.readthedocs.io/en/latest/usage/ums.html"><code>ums</code></a>
command once it is loaded via FEL mode. <code>sunx-fel</code> also
supports injecting u-boot a script into the environment so long as the
file start with a special header of <code>#=uEnv</code> and is written
to a special memory location.</p>
<figure class="code-block text"><div class="highlight"><pre><span></span><code>#=uEnv
ums_and_reboot=echo 'entering ums...'; ums 0 mmc 0; echo 'about to reboot..'; reset;
bootcmd=run ums_and_reboot;
</code></pre></div>
<figcaption><p>Contents of the <code>uenv</code> file</p></figcaption></figure>
<p>With the u-boot script it’s possbile to load uboot and automatically
run the <code>ums</code> command.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>./sunxi-fel uboot u-boot-sunxi-with-spl.bin write-with-progress 0x43100000 uenv
</code></pre></div>
<figcaption><p>Loading u-boot over fel mode with an environment injected</p></figcaption></figure>
<p>The above command will cause a mass storage USB device to appear.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>lsusb -d 1f3a:
<span class="go">Bus 005 Device 079: ID 1f3a:1010 Allwinner Technology Android device in fastboot mode</span>
</code></pre></div>
</figure>
<p>This USB device can be written to or read from with <code>dd</code>.
Zeroing out the card entirely can be done with the following
command.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>dd <span class="k">if</span><span class="o">=</span>/dev/zero <span class="nv">of</span><span class="o">=</span>/dev/disk/by-id/usb-Linux_UMS_disk_0_92c001baea519a49-0:0 <span class="nv">status</span><span class="o">=</span>progress
</code></pre></div>
</figure>
<p>The <code>ums</code> command will keep running until a
<code>Ctrl-C</code> is dispatched to <code>u-boot</code> over the serial
console. Then <code>u-boot</code> will trigger a reboot. If a new image
was written to the SD card, then rebooting will boot into the new
image.</p>
<h2 id="using-linux">Using Linux<a class="pilcrow" href="#using-linux"></a></h2>
<p>The second option is to use u-boot to load a kernel image and an
initrd which prepares the <code>g_mass_storage</code> kernel module and
triggers a reboot once the USB device has been ejected. The
<code>g_mass_storage</code> module needs the following kernel
configuration options enabled.</p>
<figure class="code-block kconfig"><div class="highlight"><pre><span></span><code>CONFIG_USB_GADGET=y¬<span class="w"></span>
CONFIG_USB_MASS_STORAGE=y<span class="w"></span>
CONFIG_USB_ROLE_SWITCH=y<span class="w"></span>
</code></pre></div>
</figure>
<p>For performance reasons, we should be using the performance governor
so the USB stack will be running at the maximum speed. This will make
read and write operations much faster.</p>
<figure class="code-block kconfig"><div class="highlight"><pre><span></span><code>CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y<span class="w"></span>
</code></pre></div>
</figure>
<p>A small <a href="https://www.busybox.net/">busybox</a> initrd is
sufficient to prepare the <code>g_mass_storage</code> module.</p>
<h3 id="creating-a-busybox-based-initrd">Creating a busybox based
initrd</h3>
<p>Assuming that busybox has been compiled in to a single static
executable, creating an initrd is pretty straight forward. Simply create
all of the basic directories of the root fileystem and place the busybox
executable at <code>/bin/busybox</code>.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>mkdir -p root/bin/
<span class="gp">$ </span>mkdir -p root/dev/
<span class="gp">$ </span>mkdir -p root/etc/
<span class="gp">$ </span>mkdir -p root/lib/
<span class="gp">$ </span>mkdir -p root/mnt/
<span class="gp">$ </span>mkdir -p root/proc/
<span class="gp">$ </span>mkdir -p root/sbin/
<span class="gp">$ </span>mkdir -p root/sys/
<span class="gp">$ </span>mkdir -p root/tmp/
<span class="gp">$ </span>mkdir -p root/usr/bin/
<span class="gp">$ </span>mkdir -p root/usr/sbin/
<span class="gp">$ </span>mkdir -p root/var
<span class="gp">$ </span>cp busybox /root/bin/busybox
</code></pre></div>
</figure>
<p>Even though busybox comes with <code>init</code> functionality, I
think it’s easier just to create a shell script at
<code>/root/init</code> that sets up busybox, passes the SD card block
device to the <code>g_mass_storage</code> module and polls for
ejection.</p>
<figure class="code-block sh"><div class="highlight"><pre><span></span><code><span class="ch">#!/bin/busybox sh</span>
/bin/busybox --install -s <span class="c1"># Install's busybox symlinks</span>
<span class="c1"># Mount all required filesystems</span>
mount -t devtmpfs devtmpfs /dev
mount -t proc proc /proc
mount -t sysfs sysfs /sys
mount -t tmpfs tmpfs /tmp
mount -t tmpfs mdev /dev
mount -t configfs none /sys/kernel/config
mount -t debugfs none /sys/kernel/debug
<span class="nb">echo</span> /sbin/mdev > /proc/sys/kernel/hotplug
mdev -s
<span class="c1"># Switch the Pine A64 into peripheral mode</span>
<span class="nb">echo</span> <span class="s1">'peripheral'</span> > /sys/devices/platform/soc/1c19000.usb/musb-hdrc.2.auto/mode
<span class="c1"># Maximize the clock of the MMC</span>
<span class="nb">echo</span> <span class="m">150000000</span> > /sys/kernel/debug/mmc0/clock
<span class="c1"># Open the SD card as a file over USB</span>
<span class="nb">echo</span> /dev/mmcblk0 > /sys/class/udc/musb-hdrc.2.auto/device/gadget/lun0/file
<span class="c1"># Poll for ejection</span>
<span class="nb">echo</span> <span class="s1">'Polling for ejection...'</span>
<span class="k">while</span> <span class="o">[</span> ! -z <span class="k">$(</span>cat /sys/class/udc/musb-hdrc.2.auto/device/gadget/lun0/file<span class="k">)</span> <span class="o">]</span>
<span class="k">do</span>
sleep <span class="m">1</span>
<span class="k">done</span>
<span class="nb">echo</span> <span class="s1">'Ejected, rebooting...'</span>
reboot -f
</code></pre></div>
<figcaption><p>Contents of the <code>root/init</code> file</p></figcaption></figure>
<p>Finally the entire <code>root</code> directory can be placed into a
CPIO archive, compressed and have the u-boot header attached.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span><span class="nb">cd</span> root
<span class="gp">$ </span>find . <span class="p">|</span> cpio -ov --format<span class="o">=</span>newc <span class="p">|</span> lzma -9 >../initramfz
<span class="gp">$ </span><span class="nb">cd</span> ..
<span class="gp">$ </span>mkimage -A arm64 -O linux -T ramdisk -C lzma -d initramfz initramfs.uImage
</code></pre></div>
</figure>
<h3 id="enabling-the-gadget">Enabling the gadget</h3>
<p>With u-boot, a kernel with the <code>g_mass_storage</code> module and
the above initrd it’s possible to use FEL mode to load all of them and
boot into a Linux kernel which exposes the SD card over USB. The last
thing that is needed is a u-boot script which sets the kernel arguments
and immediately boots into the kernel.</p>
<figure class="code-block text"><div class="highlight"><pre><span></span><code>#=uEnv
set_boot_args=setenv bootargs quiet console=ttyS0,115200 g_mass_storage.luns=1 g_mass_storage.removable=1
bootcmd=run set_boot_args; booti $kernel_addr_r $ramdisk_addr_r $fdtcontroladdr;
</code></pre></div>
<figcaption><p>Contents of the uenv file</p></figcaption></figure>
<p>All of these files can be loaded and executed</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>sunxi-fel uboot u-boot-sunxi-with-spl.bin write 0x40080000 Image.lzma write 0x4FF00000 initramfs.uImage writes 0x43100000 uenv
</code></pre></div>
<figcaption><p>Loading u-boot over fel mode with an environment, initrd and kernel
image</p></figcaption></figure>
<p>The above command will cause a mass storage USB device to appear.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>lsusb -d <span class="m">0525</span>:
<span class="go">Bus 005 Device 089: ID 0525:a4a5 Netchip Technology, Inc. Pocketbook Pro 903 / Mobius 2 Action Cam / xDuoo X3 / PocketBook Pro 602</span>
</code></pre></div>
</figure>
<p>By default the USB vendor id is from Netchip. It can be changed by
specifying the <code>idVendor</code> module parameter. Like the mass
storage device created by u-boot this can be written to or read from
with <code>dd</code>. Since our initrd polls for ejection, ejecting the
drive from the host will trigger a reboot on the device. If a new image
was written, the reboot will trigger a boot of the new image.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>dd <span class="k">if</span><span class="o">=</span>/dev/zero <span class="nv">of</span><span class="o">=</span>/dev/disk/by-id/usb-Linux_File-Stor_Gadget-0:0 <span class="nv">status</span><span class="o">=</span>progress
<span class="gp">$ </span>eject -s /dev/disk/by-id/usb-Linux_File-Stor_Gadget-0:0
</code></pre></div>
</figure>
<h2 id="performance-metrics">Performance Metrics<a class="pilcrow" href="#performance-metrics"></a></h2>
<p>Regardless of how the gadget is implemented in u-boot or in Linux I
have found that the optimal block size is 4MB for most SD cards. I have
also found adding the <code>oflag=direct</code> flag when writing or the
<code>iflag=direct</code> when reading ensures the throughput is
higher.</p>
<table>
<thead>
<tr class="header">
<th style="text-align: center;">Operation</th>
<th style="text-align: center;">u-boot</th>
<th style="text-align: center;">Linux</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: center;">Read 1GB (MB/s)</td>
<td style="text-align: center;">13.2</td>
<td style="text-align: center;">30.1</td>
</tr>
<tr class="even">
<td style="text-align: center;">Write 1GB (MB/s)</td>
<td style="text-align: center;">9</td>
<td style="text-align: center;">18.5</td>
</tr>
</tbody>
</table>
<p>Since u-boot does not increase the clock speeds to maximum and since
u-boot does not initialize multiple cores, the implementation is
slower.</p>
<h2 id="conclusion">Conclusion<a class="pilcrow" href="#conclusion"></a></h2>
<p>Due to Allwinner CPUs design, it is possible to load in u-boot via
USB directly to RAM. Once loaded, u-boot can expose internal storage
over USB and the internal storage can be changed as needed.
Alternatively, a full kernel and initrd can be loaded over USB and
booted into, which can also expose the internal storage over USB. The
Linux approach allows for better performance which may matter depending
on the size of the internal storage.</p>
<section id="footnotes" class="footnotes footnotes-end-of-document" role="doc-endnotes">
<hr>
<ol>
<li id="fn1"><p>Also any other Allwinner CPU based device, and probably
any Rockchip or Amlogic device as well.<a href="#fnref1" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
<li id="fn2"><p>This applies to any internal storage such as eMMC or SPI
as well.<a href="#fnref2" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
<li id="fn3"><p>The upper port on the Pine A64<a href="#fnref3" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
</ol>
</section>
tag:zameermanji.com,2021-12-29:/blog/2021/12/29/customizing-u-boot-build-configuration/Customizing u-boot build configuration2021-12-29T00:00:00Z2021-12-29T00:00:00Z<p><a href="https://www.denx.de/wiki/U-Boot">u-boot</a> uses the <a href="https://www.kernel.org/doc/html/latest/kbuild/kconfig-macro-language.html">Kconfig</a>
build system like Linux. One issue I faced with Kconfig in u-boot is how
to alter the configuration of a build via a script. The typical u-boot
flow is to seed the build configuration by using a
<code>defconfig</code> for a specific board.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>make pine64_plus_defconfig
</code></pre></div>
</figure>
<p>The <code>.config</code> file is then populated with reasonable
default values. The next step is to interactively edit the configuration
via a ncurses application.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>make menuconfig
</code></pre></div>
</figure>
<p>This approach is not feasible in a script because
<code>make menuconfig</code> is interactive. Fortunately u-boot ships
with a utility script <code>./scripts/kconfig/merge_config.sh</code>
which allows merging of two different config files.</p>
<p>This script allows merging two sets of config files together. For
example to add on zstd compression to the
<code>pine64_plus_defconfig</code> configuration just do the
following:</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>make pine64_plus_defconfig
<span class="gp">$ </span><span class="nb">echo</span> <span class="s1">'CONFIG_ZSTD=y'</span> > ./.custom_config
<span class="gp">$ </span>./scripts/kconfig/merge_config.sh <span class="s1">'.config'</span> <span class="s1">'.custom_config'</span>
</code></pre></div>
</figure>
<p>After running the above the <code>CONFIG_ZSTD</code> option will be
merge into the <code>.config</code> file and set.</p>
<p>The same approach can be taken to disable certain configuration
values as well. For example to disable any networking related
functionality, it’s possible to flip the value of the
<code>CONFIG_NET</code> option.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>make pine64_plus_defconfig
<span class="gp">$ </span><span class="nb">echo</span> <span class="s1">'CONFIG_NET=n'</span> > ./.custom_config
<span class="gp">$ </span>./scripts/kconfig/merge_config.sh <span class="s1">'.config'</span> <span class="s1">'.custom_config'</span>
</code></pre></div>
</figure>
<p>After the script runs, any networking related options will be
disabled.</p>
<p>There are some limitations to this approach, with
<code>depends on</code> such as the following:</p>
<figure class="code-block kconfig"><div class="highlight"><pre><span></span><code><span class="k">config</span><span class="w"> </span>CMD_USB_MASS_STORAGE<span class="w"></span>
<span class="w"> </span><span class="nb">bool</span><span class="w"> </span><span class="s2">"UMS usb mass storage"</span><span class="w"></span>
<span class="w"> </span><span class="k">depends on</span><span class="w"> </span>USB_GADGET_DOWNLOAD<span class="w"></span>
<span class="w"> </span><span class="k">select</span><span class="w"> </span>USB_FUNCTION_MASS_STORAGE<span class="w"></span>
<span class="w"> </span><span class="k">depends on</span><span class="w"> </span>BLK<span class="w"> </span><span class="o">&&</span><span class="w"> </span>USB_GADGET<span class="w"></span>
<span class="w"> </span><span class="k">help</span>
<span class="w"> </span>Enables<span class="w"> </span>the<span class="w"> </span>command<span class="w"> </span><span class="s2">"ums"</span><span class="w"> </span>and<span class="w"> </span>the<span class="w"> </span>USB<span class="w"> </span>mass<span class="w"> </span>storage<span class="w"> </span>support<span class="w"> </span>to<span class="w"> </span>the<span class="w"></span>
<span class="w"> </span>export<span class="w"> </span>a<span class="w"> </span>block<span class="w"> </span>device:<span class="w"> </span>U-Boot,<span class="w"> </span>the<span class="w"> </span>USB<span class="w"> </span>device,<span class="w"> </span>acts<span class="w"> </span>as<span class="w"> </span>a<span class="w"> </span>simple<span class="w"></span>
<span class="w"> </span>external<span class="w"> </span>hard<span class="w"> </span>drive<span class="w"> </span>plugged<span class="w"> </span>on<span class="w"> </span>the<span class="w"> </span>host<span class="w"> </span>USB<span class="w"> </span>port.<span class="w"></span>
</code></pre></div>
</figure>
<p>The <code>pine64_plus_defconfig</code> does not set
<code>CMD_USB_MASS_STORAGE</code> but it also doesn’t set
<code>USB_GADGET_DOWNLOAD</code>. Trying to enable
<code>CMD_USB_MASS_STORAGE</code> without the dependencies results in
the following.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>make pine64_plus_defconfig
<span class="gp">$ </span><span class="nb">echo</span> <span class="s1">'CONFIG_CMD_USB_MASS_STORAGE=y'</span> > ./.custom_config
<span class="gp">$ </span>./scripts/kconfig/merge_config.sh <span class="s1">'.config'</span> <span class="s1">'.custom_config'</span>
<span class="go">Using .config as base</span>
<span class="go">Merging .custom_config</span>
<span class="go">scripts/kconfig/conf --alldefconfig Kconfig</span>
<span class="gp">#</span>
<span class="gp"># </span>configuration written to .config
<span class="gp">#</span>
<span class="go">Value requested for CONFIG_CMD_USB_MASS_STORAGE not in final .config</span>
<span class="go">Requested value: CONFIG_CMD_USB_MASS_STORAGE=y</span>
<span class="go">Actual value:</span>
</code></pre></div>
</figure>
<p>In order to actually enable this feature, all of the dependencies
need to be enabled as well in the final configuration.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>make pine64_plus_defconfig
<span class="gp">$ </span><span class="nb">echo</span> <span class="s1">'CONFIG_CMD_USB_MASS_STORAGE=y'</span> > ./.custom_config
<span class="gp">$ </span><span class="nb">echo</span> <span class="s1">'CONFIG_USB_GADGET_DOWNLOAD=y'</span> >> ./.custom_config
<span class="gp">$ </span>./scripts/kconfig/merge_config.sh <span class="s1">'.config'</span> <span class="s1">'.custom_config'</span>
<span class="go">Using .config as base</span>
<span class="go">Merging .custom_config</span>
<span class="go">Value of CONFIG_USB_GADGET_DOWNLOAD is redefined by fragment .custom_config:</span>
<span class="go">Previous value: # CONFIG_USB_GADGET_DOWNLOAD is not set</span>
<span class="go">New value: CONFIG_USB_GADGET_DOWNLOAD=y</span>
<span class="go">scripts/kconfig/conf --alldefconfig Kconfig</span>
<span class="gp">#</span>
<span class="gp"># </span>configuration written to .config
<span class="gp">#</span>
</code></pre></div>
</figure>
<p>Despite its limitations the <code>merge_config.sh</code> script is a
powerful way to alter the build configuration of u-boot via a
script.</p>
tag:zameermanji.com,2021-11-22:/blog/2021/11/22/combining-sunxi-boot-with-guid-partition-tables/Combining Sunxi Boot with GUID Partition Tables2021-11-22T00:00:00Z2021-11-22T00:00:00Z<p>The <a href="https://linux-sunxi.org/Main_Page">Sunxi family of
SoCs</a> from Allwinner, have a very specific boot process. When an SD
card is available, the boot ROM expects to find the SPL (Secondary
Program Loader) at exactly block 16 (8KB) of the SD card. This creates a
problem when using the (GPT) GUID Partition Table <a href="#fn1" class="footnote-ref" id="fnref1" role="doc-noteref"><sup>1</sup></a>
scheme because the entire partition table takes up 16KB.</p>
<p><picture><svg viewBox="-72.000004 -72.000005 148.332682 62.903791"><style><![CDATA[#embedtikzgpt__embedtikzgpt text.embedtikzgpt__f0{font-size:4.98132px;font-family:IBM Plex Serif;font-style:normal;font-weight:400}#embedtikzgpt__embedtikzgpt .embedtikzgpt__node-text text{fill:var(--color-text)}#embedtikzgpt__embedtikzgpt .embedtikzgpt__node-icon path,#embedtikzgpt__embedtikzgpt .embedtikzgpt__node-icon use,#embedtikzgpt__embedtikzgpt .embedtikzgpt__node-shape{stroke:var(--color-text)}#embedtikzgpt__embedtikzgpt .embedtikzgpt__node-icon.embedtikzgpt__hard-drive path,#embedtikzgpt__embedtikzgpt .embedtikzgpt__node-icon.embedtikzgpt__hard-drive use{stroke-width:.333}]]></style><g id="embedtikzgpt__embedtikzgpt" stroke="#000" stroke-miterlimit="10" stroke-width="0.4"><path fill="none" stroke-width=".398504" d="M-16.652-61.314h59.776V-71.8h-59.776Z" class="embedtikzgpt__node-shape"></path><g class="embedtikzgpt__node-text"><text x="13.236314" y="-66.557447" stroke="none" class="embedtikzgpt__f0" transform="translate(-9.053 1.863)">Unuse<tspan x="28.314772">d</tspan></text></g><path fill="none" stroke-width=".398504" d="M-16.652-50.827h59.776v-10.487h-59.776Z" class="embedtikzgpt__node-shape"></path><g class="embedtikzgpt__node-text"><text x="13.236314" y="-66.557447" stroke="none" class="embedtikzgpt__f0" transform="translate(-14.187 12.35)">GPT<tspan x="24.344661">He</tspan><tspan x="30.984764">ader</tspan></text></g><path fill="none" stroke-width=".398504" d="M-16.652-40.41h59.776v-10.417h-59.776Z" class="embedtikzgpt__node-shape"></path><g class="embedtikzgpt__node-text"><text x="13.236314" y="-66.557447" stroke="none" class="embedtikzgpt__f0" transform="translate(-18.604 22.767)">GPT<tspan x="24.344661">Entries</tspan><tspan x="42.511539">1-4</tspan></text></g><path fill="none" stroke-width=".398504" d="M-16.652-29.994h59.776V-40.41h-59.776Z" class="embedtikzgpt__node-shape"></path><g class="embedtikzgpt__node-text"><text x="13.236314" y="-66.557447" stroke="none" class="embedtikzgpt__f0" transform="translate(-18.604 33.184)">GPT<tspan x="24.344661">Entries</tspan><tspan x="42.511539">5-8</tspan></text></g><path fill="none" stroke-width=".398504" d="M-16.652-19.713h59.776v-10.281h-59.776Z" class="embedtikzgpt__node-shape"></path><g class="embedtikzgpt__node-text"><text x="13.236314" y="-66.557447" stroke="none" class="embedtikzgpt__f0" transform="translate(-2.005 41.959)">…</text></g><path fill="none" stroke-width=".398504" d="M-16.652-9.295h59.776v-10.417h-59.776Z" class="embedtikzgpt__node-shape"></path><g class="embedtikzgpt__node-text"><text x="13.236314" y="-66.557447" stroke="none" class="embedtikzgpt__f0" transform="translate(-24.582 53.88)">GPT<tspan x="24.344661">Entries</tspan><tspan x="42.511539">125-128</tspan></text></g><g class="embedtikzgpt__node-text"><text x="13.236314" y="-66.557447" stroke="none" class="embedtikzgpt__f0" transform="translate(-60.685 1.863)">Blo<tspan x="20.927475">ck</tspan><tspan x="27.562597">0</tspan></text></g><g class="embedtikzgpt__node-text"><text x="13.236314" y="-66.557447" stroke="none" class="embedtikzgpt__f0" transform="translate(-60.685 12.35)">Blo<tspan x="20.927475">ck</tspan><tspan x="27.562597">1</tspan></text></g><g class="embedtikzgpt__node-text"><text x="13.236314" y="-66.557447" stroke="none" class="embedtikzgpt__f0" transform="translate(-60.685 22.802)">Blo<tspan x="20.927475">ck</tspan><tspan x="27.562597">2</tspan></text></g><g class="embedtikzgpt__node-text"><text x="13.236314" y="-66.557447" stroke="none" class="embedtikzgpt__f0" transform="translate(-60.685 33.218)">Blo<tspan x="20.927475">ck</tspan><tspan x="27.562597">3</tspan></text></g><g class="embedtikzgpt__node-text"><text x="13.236314" y="-66.557447" stroke="none" class="embedtikzgpt__f0" transform="translate(-54.032 41.959)">…</text></g><g class="embedtikzgpt__node-text"><text x="13.236314" y="-66.557447" stroke="none" class="embedtikzgpt__f0" transform="translate(-62.18 53.917)">Blo<tspan x="20.927475">ck</tspan><tspan x="27.562597">33</tspan></text></g></g></svg>
</picture></p>
<p>When using the GPT scheme, the boot ROM will instead read part of the
partition table instead of the SPL. This is still an issue even if there
are only a few partitions, the entire space needs to be reserved for the
table.</p>
<p>Fortunately, it’s possible to adjust the partition table so there is
a gap between the headers and entries. This is because in the GPT Header
there is a field that references the start of the GPT entries. Typically
it is Block 2, but it can be adjusted to be any block on the disk.</p>
<p>Tools like <a href="https://www.rodsbooks.com/gdisk/sgdisk-walkthrough.html">sgdisk</a>
can set this value. In <code>sgdisk</code>’s case it’s done with the
<code>-j</code> command. This means the following set of commands, will
create a new partition table and relocate the start of the entries to
sector 4064. This provides ample room to place the SPL and a boot
loader.</p>
<figure class="code-block bash"><div class="highlight"><pre><span></span><code>$ sgdisk -o /dev/sda
$ sgdisk -j <span class="m">4064</span> /dev/sda
</code></pre></div>
</figure>
<p>After the above commands the GPT looks like this.</p>
<p><picture><svg viewBox="-72.000004 -72.000006 148.332682 85.116453"><style><![CDATA[#embedtikzgpthole__embedtikzgpthole text.embedtikzgpthole__f0{font-size:4.98132px;font-family:IBM Plex Serif;font-style:normal;font-weight:400}#embedtikzgpthole__embedtikzgpthole .embedtikzgpthole__node-text text{fill:var(--color-text)}#embedtikzgpthole__embedtikzgpthole .embedtikzgpthole__node-icon path,#embedtikzgpthole__embedtikzgpthole .embedtikzgpthole__node-icon use,#embedtikzgpthole__embedtikzgpthole .embedtikzgpthole__node-shape{stroke:var(--color-text)}#embedtikzgpthole__embedtikzgpthole .embedtikzgpthole__node-icon.embedtikzgpthole__hard-drive path,#embedtikzgpthole__embedtikzgpthole .embedtikzgpthole__node-icon.embedtikzgpthole__hard-drive use{stroke-width:.333}]]></style><g id="embedtikzgpthole__embedtikzgpthole" stroke="#000" stroke-miterlimit="10" stroke-width="0.4"><path fill="none" stroke-width=".398504" d="M-16.652-61.314h59.776V-71.8h-59.776Z" class="embedtikzgpthole__node-shape"></path><g class="embedtikzgpthole__node-text"><text x="13.236314" y="-66.557447" stroke="none" class="embedtikzgpthole__f0" transform="translate(-9.053 1.863)">Unuse<tspan x="28.314772">d</tspan></text></g><path fill="none" stroke-width=".398504" d="M-16.652-50.827h59.776v-10.487h-59.776Z" class="embedtikzgpthole__node-shape"></path><g class="embedtikzgpthole__node-text"><text x="13.236314" y="-66.557447" stroke="none" class="embedtikzgpthole__f0" transform="translate(-14.187 12.35)">GPT<tspan x="24.344661">He</tspan><tspan x="30.984764">ader</tspan></text></g><path fill="none" stroke-width=".398504" d="M-16.652-39.653h59.776v-11.174h-59.776Z" class="embedtikzgpthole__node-shape"></path><g class="embedtikzgpthole__node-text"><text x="13.236314" y="-66.557447" stroke="none" class="embedtikzgpthole__f0" transform="translate(-7.705 22.528)">Empty</text></g><path fill="none" stroke-width=".398504" d="M-16.652-28.48h59.776v-11.173h-59.776Z" class="embedtikzgpthole__node-shape"></path><g class="embedtikzgpthole__node-text"><text x="13.236314" y="-66.557447" stroke="none" class="embedtikzgpthole__f0" transform="translate(-7.705 33.702)">Empty</text></g><path fill="none" stroke-width=".398504" d="M-16.652-18.199h59.776v-10.28h-59.776Z" class="embedtikzgpthole__node-shape"></path><g class="embedtikzgpthole__node-text"><text x="13.236314" y="-66.557447" stroke="none" class="embedtikzgpthole__f0" transform="translate(-2.005 43.473)">…</text></g><path fill="none" stroke-width=".398504" d="M-16.652-7.78h59.776v-10.418h-59.776Z" class="embedtikzgpthole__node-shape"></path><g class="embedtikzgpthole__node-text"><text x="13.236314" y="-66.557447" stroke="none" class="embedtikzgpthole__f0" transform="translate(-18.604 55.395)">GPT<tspan x="24.344661">Entries</tspan><tspan x="42.511539">1-4</tspan></text></g><path fill="none" stroke-width=".398504" d="M-16.652 2.5h59.776V-7.78h-59.776Z" class="embedtikzgpthole__node-shape"></path><g class="embedtikzgpthole__node-text"><text x="13.236314" y="-66.557447" stroke="none" class="embedtikzgpthole__f0" transform="translate(-2.005 64.171)">…</text></g><path fill="none" stroke-width=".398504" d="M-16.652 12.918h59.776V2.5h-59.776Z" class="embedtikzgpthole__node-shape"></path><g class="embedtikzgpthole__node-text"><text x="13.236314" y="-66.557447" stroke="none" class="embedtikzgpthole__f0" transform="translate(-24.582 76.094)">GPT<tspan x="24.344661">Entries</tspan><tspan x="42.511539">125-128</tspan></text></g><g class="embedtikzgpthole__node-text"><text x="13.236314" y="-66.557447" stroke="none" class="embedtikzgpthole__f0" transform="translate(-60.685 1.863)">Blo<tspan x="20.927475">ck</tspan><tspan x="27.562597">0</tspan></text></g><g class="embedtikzgpthole__node-text"><text x="13.236314" y="-66.557447" stroke="none" class="embedtikzgpthole__f0" transform="translate(-60.685 12.35)">Blo<tspan x="20.927475">ck</tspan><tspan x="27.562597">1</tspan></text></g><g class="embedtikzgpthole__node-text"><text x="13.236314" y="-66.557447" stroke="none" class="embedtikzgpthole__f0" transform="translate(-60.685 23.18)">Blo<tspan x="20.927475">ck</tspan><tspan x="27.562597">2</tspan></text></g><g class="embedtikzgpthole__node-text"><text x="13.236314" y="-66.557447" stroke="none" class="embedtikzgpthole__f0" transform="translate(-60.685 34.354)">Blo<tspan x="20.927475">ck</tspan><tspan x="27.562597">3</tspan></text></g><g class="embedtikzgpthole__node-text"><text x="13.236314" y="-66.557447" stroke="none" class="embedtikzgpthole__f0" transform="translate(-54.032 43.473)">…</text></g><g class="embedtikzgpthole__node-text"><text x="13.236314" y="-66.557447" stroke="none" class="embedtikzgpthole__f0" transform="translate(-65.169 55.431)">Blo<tspan x="20.927475">ck</tspan><tspan x="27.562597">4064</tspan></text></g><g class="embedtikzgpthole__node-text"><text x="13.236314" y="-66.557447" stroke="none" class="embedtikzgpthole__f0" transform="translate(-54.032 64.171)">…</text></g><g class="embedtikzgpthole__node-text"><text x="13.236314" y="-66.557447" stroke="none" class="embedtikzgpthole__f0" transform="translate(-65.169 76.13)">Blo<tspan x="20.927475">ck</tspan><tspan x="27.562597">4066</tspan></text></g></g></svg>
</picture></p>
<p>Once this is done, <code>sgdisk</code> can be used to add partitions
and it will preserve the empty space in the table. The SPL can be
written to the gap without issue so long as there is enough space.
Usiing this method it’s possible to have a GPT formatted disk contain
the SPL and the data, which is useful for devices that only have an SD
card to boot from.</p>
<section id="footnotes" class="footnotes footnotes-end-of-document" role="doc-endnotes">
<hr>
<ol>
<li id="fn1"><p>Usually when you want to use EFI boot or follow the <a href="https://systemd.io/DISCOVERABLE_PARTITIONS/">Discoverable
Partitions Specification</a><a href="#fnref1" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
</ol>
</section>
tag:zameermanji.com,2021-11-17:/blog/2021/11/17/fixing-wpa-supplicants-systemd-unit-file/Fixing wpa_supplicant’s systemd unit file2021-11-17T00:00:00Z2021-11-17T00:00:00Z<p>I recently started to use <code>wpa_supplicant</code> on Ubuntu 21.10
to manage a USB WiFi adapter. I noticed that when the adapter was not
attached to the computer during boot then the boot hangs for 90 seconds.
systemd displays the following errors after 90 seconds and continues the
boot.</p>
<figure class="code-block text"><div class="highlight"><pre><span></span><code>sys-subsystem-net-devices-wlan0.device: Job sys-subsystem-net-devices-wlan0.device/start timed out.
Timed out waiting for device /sys/subsystem/net/devices/wlan0.
Dependency failed for WPA supplicant daemon (interface-specific version).
wpa_supplicant@wlan0.service: Job wpa_supplicant@wlan0.service/start failed with result 'dependency'.
sys-subsystem-net-devices-wlan0.device: Job sys-subsystem-net-devices-wlan0.device/start failed with result 'timeout'
</code></pre></div>
</figure>
<p>I also noticed that after the boot was complete, inserting the
adapter would not start <code>wpa_supplicant</code> until I manually
started <code>wpa_supplicant</code>. Both of these issues comes from the
Ubuntu provided <code>wpa_supplicant</code> systemd unit.</p>
<figure class="code-block ini"><div class="highlight"><pre><span></span><code><span class="k">[Unit]</span><span class="w"></span>
<span class="na">Description</span><span class="o">=</span><span class="s">WPA supplicant daemon (interface-specific version)</span><span class="w"></span>
<span class="na">Requires</span><span class="o">=</span><span class="s">sys-subsystem-net-devices-%i.device</span><span class="w"></span>
<span class="na">After</span><span class="o">=</span><span class="s">sys-subsystem-net-devices-%i.device</span><span class="w"></span>
<span class="na">Before</span><span class="o">=</span><span class="s">network.target</span><span class="w"></span>
<span class="na">Wants</span><span class="o">=</span><span class="s">network.target</span><span class="w"></span>
<span class="k">[Service]</span><span class="w"></span>
<span class="na">Type</span><span class="o">=</span><span class="s">simple</span><span class="w"></span>
<span class="na">ExecStart</span><span class="o">=</span><span class="s">/sbin/wpa_supplicant -c/etc/wpa_supplicant/wpa_supplicant-%I.conf -Dnl80211,wext -i%I</span><span class="w"></span>
<span class="k">[Install]</span><span class="w"></span>
<span class="na">Alias</span><span class="o">=</span><span class="s">multi-user.target.wants/wpa_supplicant@%i.service</span><span class="w"></span>
</code></pre></div>
<figcaption><p>The systemd unit at
<code>/lib/systemd/system/wpa_supplicant@.service</code></p></figcaption></figure>
<p>This systemd unit file allows me to write my configuration for the
<code>wlan0</code> device at
<code>/etc/wpa_supplicant/wpa_supplicant-wlan0.conf</code> and enable
<code>wpa_supplicant</code> for the device by running:
<code>systemctl enable wpa_supplicant@wlan0</code>.</p>
<p>The two problems come from two different stanzas in the unit file.
The first is the <code>Requires=</code> stanza.</p>
<figure class="code-block ini"><div class="highlight"><pre><span></span><code><span class="na">Requires</span><span class="o">=</span><span class="s">sys-subsystem-net-devices-%i.device</span><span class="w"></span>
</code></pre></div>
</figure>
<p>According to the systemd <a href="https://www.freedesktop.org/software/systemd/man/systemd.unit.html">man
page</a>:</p>
<blockquote>
<p>If this unit gets activated, the units listed will be activated as
well. If one of the other units fails to activate, and an ordering
dependency After= on the failing unit is set, this unit will not be
started. Besides, with or without specifying After=, this unit will be
stopped if one of the other units is explicitly stopped.</p>
</blockquote>
<p>During boot systemd attempts to activate
<code>sys-subsystem-net-devices-wlan0.device</code> which is impossible
when the device is not connected. By default systemd will wait 90
seconds in this case before moving on. Usually this is not a problem on
it’s own but combined with the <code>Alias=</code> stanza.</p>
<figure class="code-block ini"><div class="highlight"><pre><span></span><code><span class="na">Alias</span><span class="o">=</span><span class="s">multi-user.target.wants/wpa_supplicant@%i.service</span><span class="w"></span>
</code></pre></div>
</figure>
<p>This creates a hard dependency between the
<code>multi-user.target</code> target and the service and unfortunately,
the <code>multi-user.target</code> is the default target systemd boots
into. Combined with the previous <code>Requires=</code> stanza this
blocks completing booting for 90 seconds.</p>
<p>To fix these issues, a drop-in unit can remove the dependency on the
<code>multi-user.target</code> and tie the lifecycle of the device and
<code>wpa_supplicant</code> together.</p>
<figure class="code-block ini"><div class="highlight"><pre><span></span><code><span class="k">[Unit]</span><span class="w"></span>
<span class="na">Requires</span><span class="o">=</span><span class="w"></span>
<span class="na">BindsTo</span><span class="o">=</span><span class="s">sys-subsystem-net-devices-wlan0.device</span><span class="w"></span>
<span class="k">[Install]</span><span class="w"></span>
<span class="na">Alias</span><span class="o">=</span><span class="w"></span>
<span class="na">WantedBy</span><span class="o">=</span><span class="w"></span>
<span class="na">WantedBy</span><span class="o">=</span><span class="s">sys-subsystem-net-devices-wlan0.device</span><span class="w"></span>
</code></pre></div>
<figcaption><p>The drop-in unit at
<code>/etc/systemd/system/wpa_supplicant@wlan0.service.d/override.conf </code></p></figcaption></figure>
<p>The combined <code>BindsTo=</code> and <code>WantedBy=</code> stanzas
tie the lifecyles of the device and <code>wpa_supplicant</code>
together. <code>wpa_supplicant</code> will only be started if the device
is inserted, and will be stopped if the device is removed. Further the
successful startup of <code>wpa_supplicant</code> is no longer required
for boot, fixing the blocking boot problem.</p>
tag:zameermanji.com,2021-09-16:/blog/2021/9/16/zfs-pool-configuration-for-a-home-nas/ZFS Pool Configuration for a Home NAS2021-09-16T00:00:00Z2021-09-16T00:00:00Z<p>I recently setup a home NAS where the storage filesystem is ZFS. Like
most home users, I don’t have a server with 10+ drive bays and a large
budget for storage. I only have 4 drive bays and want to minimize the
cost per TB of storage while being resilient to one drive failure.</p>
<p>Given the constraint of only having 4 drive bays and wanting to be
resilient to at least one disk failure, there are only two possible ZFS
configurations available, mirror vdevs or RADIDz vdevs.</p>
<p>Mirror vdevs offer flexibility as I do not need all four drives to
start and instead can expand the pool one mirror vdev at a time. For
example I could start with only two 4TB drives and mirror the data
between them.</p>
<p><picture><svg viewBox="-72.000004 -86.194169 246.362361 102.082449"><defs><path id="embedtikzhalfmirror__reuse-0" d="M16.5 9h-15m2.586 5.168L1.5 9V4.5A1.5 1.5 0 0 1 3 3h12a1.5 1.5 0 0 1 1.5 1.5V9l-2.586 5.168A1.504 1.504 0 0 1 12.57 15H5.43c-.57 0-1.09-.324-1.344-.832ZM4.5 6h.008M7.5 6h.008"></path></defs><style><![CDATA[#embedtikzhalfmirror__embedtikzhalfmirror text.embedtikzhalfmirror__f0,#embedtikzhalfmirror__embedtikzhalfmirror text.embedtikzhalfmirror__f1,#embedtikzhalfmirror__embedtikzhalfmirror text.embedtikzhalfmirror__f2{font-family:IBM Plex Mono;font-style:normal;font-weight:400}#embedtikzhalfmirror__embedtikzhalfmirror text.embedtikzhalfmirror__f0{font-size:8.966376px}#embedtikzhalfmirror__embedtikzhalfmirror text.embedtikzhalfmirror__f1{font-size:9.96264px}#embedtikzhalfmirror__embedtikzhalfmirror text.embedtikzhalfmirror__f2{font-size:6.973848px}#embedtikzhalfmirror__embedtikzhalfmirror .embedtikzhalfmirror__node-text text{fill:var(--color-text)}#embedtikzhalfmirror__embedtikzhalfmirror .embedtikzhalfmirror__node-icon path,#embedtikzhalfmirror__embedtikzhalfmirror .embedtikzhalfmirror__node-icon use,#embedtikzhalfmirror__embedtikzhalfmirror .embedtikzhalfmirror__node-shape{stroke:var(--color-text)}#embedtikzhalfmirror__embedtikzhalfmirror .embedtikzhalfmirror__node-icon.embedtikzhalfmirror__hard-drive path,#embedtikzhalfmirror__embedtikzhalfmirror .embedtikzhalfmirror__node-icon.embedtikzhalfmirror__hard-drive use{stroke-width:.333}]]></style><defs><clipPath id="embedtikzhalfmirror__clip1"><path d="M0 2h18v14H0Z"></path></clipPath></defs><g id="embedtikzhalfmirror__embedtikzhalfmirror" stroke="#000" stroke-miterlimit="10" stroke-width="0.4" transform="matrix(.99626 0 0 -.99626 -41.055 -13.923)"><g id="embedtikzhalfmirror__name-sda"><g fill="none" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.5" class="embedtikzhalfmirror__node-icon embedtikzhalfmirror__hard-drive" clip-path="url(#embedtikzhalfmirror__clip1)" transform="translate(-30.86 -27.711) scale(3.42709)"><use xlink:href="#embedtikzhalfmirror__reuse-0"></use></g><g class="embedtikzhalfmirror__node-text"><text x="-10.309788" y="-13.922932" stroke="none" class="embedtikzhalfmirror__f2" transform="matrix(1.00375 0 0 -1.00375 -6.452 -40.152)">/dev/sda</text></g></g><g id="embedtikzhalfmirror__name-sdb"><g fill="none" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.5" class="embedtikzhalfmirror__node-icon embedtikzhalfmirror__hard-drive" clip-path="url(#embedtikzhalfmirror__clip1)" transform="translate(30.86 -27.711) scale(3.42709)"><use xlink:href="#embedtikzhalfmirror__reuse-0"></use></g><g class="embedtikzhalfmirror__node-text"><text x="-41.055271" y="-13.922932" stroke="none" class="embedtikzhalfmirror__f2" transform="matrix(1.00375 0 0 -1.00375 86.13 -40.152)">/dev/sdb</text></g></g><g class="embedtikzhalfmirror__node-text"><text x="-41.055271" y="-13.922932" stroke="none" class="embedtikzhalfmirror__f1" transform="matrix(1.00375 0 0 -1.00375 21.07 17.88)">vdev<tspan x="-11.167351">mirror</tspan><tspan x="30.675737">(4TB)</tspan></text></g><path fill="none" stroke-width="0.1" d="M-30.86-29.723H92.581v72.512H-30.86Zm123.442 0h123.443v72.512H92.582Z" class="embedtikzhalfmirror__node-shape"></path><g class="embedtikzhalfmirror__node-text"><text x="-41.055271" y="-13.922932" stroke="none" class="embedtikzhalfmirror__f0" transform="matrix(1.00375 0 0 -1.00375 104.092 33.947)">zpool<tspan x="-8.776323">(4TB)</tspan></text></g><path fill="none" d="M-30.86-29.723h246.885v87.818H-30.86Z" class="embedtikzhalfmirror__node-shape"></path></g></svg>
</picture></p>
<p>Later I could create a second mirror with two more 4TB drives vdev
and add it to the pool. This would end up with about 8TB of usable
storage.</p>
<p><picture><svg viewBox="-72.000004 -86.194169 246.362361 102.082449"><defs><path id="embedtikzmirror__reuse-0" d="M16.5 9h-15m2.586 5.168L1.5 9V4.5A1.5 1.5 0 0 1 3 3h12a1.5 1.5 0 0 1 1.5 1.5V9l-2.586 5.168A1.504 1.504 0 0 1 12.57 15H5.43c-.57 0-1.09-.324-1.344-.832ZM4.5 6h.008M7.5 6h.008"></path></defs><style><![CDATA[#embedtikzmirror__embedtikzmirror text.embedtikzmirror__f0,#embedtikzmirror__embedtikzmirror text.embedtikzmirror__f1,#embedtikzmirror__embedtikzmirror text.embedtikzmirror__f2{font-family:IBM Plex Mono;font-style:normal;font-weight:400}#embedtikzmirror__embedtikzmirror text.embedtikzmirror__f0{font-size:8.966376px}#embedtikzmirror__embedtikzmirror text.embedtikzmirror__f1{font-size:9.96264px}#embedtikzmirror__embedtikzmirror text.embedtikzmirror__f2{font-size:6.973848px}#embedtikzmirror__embedtikzmirror .embedtikzmirror__node-text text{fill:var(--color-text)}#embedtikzmirror__embedtikzmirror .embedtikzmirror__node-icon path,#embedtikzmirror__embedtikzmirror .embedtikzmirror__node-icon use,#embedtikzmirror__embedtikzmirror .embedtikzmirror__node-shape{stroke:var(--color-text)}#embedtikzmirror__embedtikzmirror .embedtikzmirror__node-icon.embedtikzmirror__hard-drive path,#embedtikzmirror__embedtikzmirror .embedtikzmirror__node-icon.embedtikzmirror__hard-drive use{stroke-width:.333}]]></style><defs><clipPath id="embedtikzmirror__clip1"><path d="M0 2h18v14H0Z"></path></clipPath></defs><g id="embedtikzmirror__embedtikzmirror" stroke="#000" stroke-miterlimit="10" stroke-width="0.4" transform="matrix(.99626 0 0 -.99626 -41.055 -13.923)"><g id="embedtikzmirror__name-sda"><g fill="none" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.5" class="embedtikzmirror__node-icon embedtikzmirror__hard-drive" clip-path="url(#embedtikzmirror__clip1)" transform="translate(-30.86 -27.711) scale(3.42709)"><use xlink:href="#embedtikzmirror__reuse-0"></use></g><g class="embedtikzmirror__node-text"><text x="-10.309788" y="-13.922932" stroke="none" class="embedtikzmirror__f2" transform="matrix(1.00375 0 0 -1.00375 -6.452 -40.152)">/dev/sda</text></g></g><g id="embedtikzmirror__name-sdb"><g fill="none" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.5" class="embedtikzmirror__node-icon embedtikzmirror__hard-drive" clip-path="url(#embedtikzmirror__clip1)" transform="translate(30.86 -27.711) scale(3.42709)"><use xlink:href="#embedtikzmirror__reuse-0"></use></g><g class="embedtikzmirror__node-text"><text x="-41.055271" y="-13.922932" stroke="none" class="embedtikzmirror__f2" transform="matrix(1.00375 0 0 -1.00375 86.13 -40.152)">/dev/sdb</text></g></g><g class="embedtikzmirror__node-text"><text x="-41.055271" y="-13.922932" stroke="none" class="embedtikzmirror__f1" transform="matrix(1.00375 0 0 -1.00375 21.07 17.88)">vdev<tspan x="-11.167351">mirror</tspan><tspan x="30.675737">(4TB)</tspan></text></g><path fill="none" stroke-width="0.1" d="M-30.86-29.723H92.581v72.512H-30.86Z" class="embedtikzmirror__node-shape"></path><g id="embedtikzmirror__name-sdc"><g fill="none" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.5" class="embedtikzmirror__node-icon embedtikzmirror__hard-drive" clip-path="url(#embedtikzmirror__clip1)" transform="translate(92.582 -27.711) scale(3.42709)"><use xlink:href="#embedtikzmirror__reuse-0"></use></g><g class="embedtikzmirror__node-text"><text x="-41.055271" y="-13.922932" stroke="none" class="embedtikzmirror__f2" transform="matrix(1.00375 0 0 -1.00375 147.852 -40.152)">/dev/sdc</text></g></g><g id="embedtikzmirror__name-sdd"><g fill="none" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.5" class="embedtikzmirror__node-icon embedtikzmirror__hard-drive" clip-path="url(#embedtikzmirror__clip1)" transform="translate(154.304 -27.711) scale(3.42709)"><use xlink:href="#embedtikzmirror__reuse-0"></use></g><g class="embedtikzmirror__node-text"><text x="-41.055271" y="-13.922932" stroke="none" class="embedtikzmirror__f2" transform="matrix(1.00375 0 0 -1.00375 209.574 -40.152)">/dev/sdd</text></g></g><g class="embedtikzmirror__node-text"><text x="-41.055271" y="-13.922932" stroke="none" class="embedtikzmirror__f1" transform="matrix(1.00375 0 0 -1.00375 144.513 17.88)">vdev<tspan x="-11.167351">mirror</tspan><tspan x="30.675737">(4TB)</tspan></text></g><path fill="none" stroke-width="0.1" d="M92.582-29.723h123.443v72.512H92.582Z" class="embedtikzmirror__node-shape"></path><g class="embedtikzmirror__node-text"><text x="-41.055271" y="-13.922932" stroke="none" class="embedtikzmirror__f0" transform="matrix(1.00375 0 0 -1.00375 104.092 33.947)">zpool<tspan x="-8.776323">(8TB)</tspan></text></g><path fill="none" d="M-30.86-29.723h246.885v87.818H-30.86Z" class="embedtikzmirror__node-shape"></path></g></svg>
</picture></p>
<p>Although each pair can survive the loss of a drive, the resulting
pool can only survive the loss of a single drive. The loss of both
drives from a mirror would result in data loss. Compared to RAIDz vdevs
this has the higher cost per TB as the total usable storage is 50% of
the raw capacity of the drives.</p>
<p>The other configuration available is using a RADIz vdev. A RAIDz vdev
requires a minimum of three drives, and since RAIDz vdevs are not
expandable <a href="#fn1" class="footnote-ref" id="fnref1" role="doc-noteref"><sup>1</sup></a> all of the drives are required
upfront. In exchange there is more usable storage compared to using
mirror vdevs, about 75% of the raw capacity. If I were to get four 4TB
drives, I would end up with about 12TB of usable storage.</p>
<p><picture><svg viewBox="-72.000004 -84.355042 246.461998 102.132265"><defs><path id="embedtikzzraid__reuse-0" d="M16.5 9h-15m2.586 5.168L1.5 9V4.5A1.5 1.5 0 0 1 3 3h12a1.5 1.5 0 0 1 1.5 1.5V9l-2.586 5.168A1.504 1.504 0 0 1 12.57 15H5.43c-.57 0-1.09-.324-1.344-.832ZM4.5 6h.008M7.5 6h.008"></path></defs><style><![CDATA[#embedtikzzraid__embedtikzzraid text.embedtikzzraid__f0,#embedtikzzraid__embedtikzzraid text.embedtikzzraid__f1,#embedtikzzraid__embedtikzzraid text.embedtikzzraid__f2{font-family:IBM Plex Mono;font-style:normal;font-weight:400}#embedtikzzraid__embedtikzzraid text.embedtikzzraid__f0{font-size:8.966376px}#embedtikzzraid__embedtikzzraid text.embedtikzzraid__f1{font-size:11.955168px}#embedtikzzraid__embedtikzzraid text.embedtikzzraid__f2{font-size:6.973848px}#embedtikzzraid__embedtikzzraid .embedtikzzraid__node-text text{fill:var(--color-text)}#embedtikzzraid__embedtikzzraid .embedtikzzraid__node-icon path,#embedtikzzraid__embedtikzzraid .embedtikzzraid__node-icon use,#embedtikzzraid__embedtikzzraid .embedtikzzraid__node-shape{stroke:var(--color-text)}#embedtikzzraid__embedtikzzraid .embedtikzzraid__node-icon.embedtikzzraid__hard-drive path,#embedtikzzraid__embedtikzzraid .embedtikzzraid__node-icon.embedtikzzraid__hard-drive use{stroke-width:.333}]]></style><defs><clipPath id="embedtikzzraid__clip1"><path d="M0 2h18v14H0Z"></path></clipPath></defs><g id="embedtikzzraid__embedtikzzraid" stroke="#000" stroke-miterlimit="10" stroke-width="0.4" transform="matrix(.99626 0 0 -.99626 -41.005 -12.084)"><g id="embedtikzzraid__name-sda"><g fill="none" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.5" class="embedtikzzraid__node-icon embedtikzzraid__hard-drive" clip-path="url(#embedtikzzraid__clip1)" transform="translate(-30.86 -27.711) scale(3.42709)"><use xlink:href="#embedtikzzraid__reuse-0"></use></g><g class="embedtikzzraid__node-text"><text x="-10.259972" y="-12.083806" stroke="none" class="embedtikzzraid__f2" transform="matrix(1.00375 0 0 -1.00375 -6.502 -38.306)">/dev/sda</text></g></g><g id="embedtikzzraid__name-sdb"><g fill="none" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.5" class="embedtikzzraid__node-icon embedtikzzraid__hard-drive" clip-path="url(#embedtikzzraid__clip1)" transform="translate(30.86 -27.711) scale(3.42709)"><use xlink:href="#embedtikzzraid__reuse-0"></use></g><g class="embedtikzzraid__node-text"><text x="-41.005455" y="-12.083806" stroke="none" class="embedtikzzraid__f2" transform="matrix(1.00375 0 0 -1.00375 86.08 -38.306)">/dev/sdb</text></g></g><g id="embedtikzzraid__name-sdc"><g fill="none" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.5" class="embedtikzzraid__node-icon embedtikzzraid__hard-drive" clip-path="url(#embedtikzzraid__clip1)" transform="translate(92.582 -27.711) scale(3.42709)"><use xlink:href="#embedtikzzraid__reuse-0"></use></g><g class="embedtikzzraid__node-text"><text x="-41.005455" y="-12.083806" stroke="none" class="embedtikzzraid__f2" transform="matrix(1.00375 0 0 -1.00375 147.802 -38.306)">/dev/sdc</text></g></g><g id="embedtikzzraid__name-sdd"><g fill="none" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.5" class="embedtikzzraid__node-icon embedtikzzraid__hard-drive" clip-path="url(#embedtikzzraid__clip1)" transform="translate(154.304 -27.711) scale(3.42709)"><use xlink:href="#embedtikzzraid__reuse-0"></use></g><g class="embedtikzzraid__node-text"><text x="-41.005455" y="-12.083806" stroke="none" class="embedtikzzraid__f2" transform="matrix(1.00375 0 0 -1.00375 209.524 -38.306)">/dev/sdd</text></g></g><g class="embedtikzzraid__node-text"><text x="-41.005455" y="-12.083806" stroke="none" class="embedtikzzraid__f1" transform="matrix(1.00375 0 0 -1.00375 72.542 20.003)">vdev<tspan x="-5.139954">RAIDz</tspan><tspan x="37.898649">(12TB)</tspan></text></g><path fill="none" stroke-width="0.1" d="M-30.86-29.723h246.885v74.308H-30.86Z" class="embedtikzzraid__node-shape"></path><g class="embedtikzzraid__node-text"><text x="-41.005455" y="-12.083806" stroke="none" class="embedtikzzraid__f0" transform="matrix(1.00375 0 0 -1.00375 101.342 37.639)">zpool<tspan x="-8.726507">(12TB)</tspan></text></g><path fill="none" d="M-30.91-29.773h246.985v89.714H-30.91Z" class="embedtikzzraid__node-shape"></path></g></svg>
</picture></p>
<p>Considering the storage penalty using mirror vdevs have, I chose a
RAIDz configuration. I think this makes sense for most home users
because of the increased usable storage.</p>
<section id="footnotes" class="footnotes footnotes-end-of-document" role="doc-endnotes">
<hr>
<ol>
<li id="fn1"><p>Not expandable as of the time of writing although a
future release of OpenZFS might change that.<a href="#fnref1" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
</ol>
</section>
tag:zameermanji.com,2021-09-09:/blog/2021/9/9/installing-ubuntu-20-04-over-ssh/Installing Ubuntu 20.04 over SSH2021-09-09T00:00:00Z2021-09-09T00:00:00Z<p>I obtained a home server which I wanted to install Ubuntu <a href="#fn1" class="footnote-ref" id="fnref1" role="doc-noteref"><sup>1</sup></a> on. There was no serial port
available and the only video out is VGA. I don’t have a VGA capable
display so I was stumped on how to install Ubuntu. This post outlines my
failed and successful attempts in case anyone has the same problem.</p>
<h2 id="failed-attempt-1-installing-on-another-host">Failed Attempt 1:
Installing on Another Host<a class="pilcrow" href="#failed-attempt-1-installing-on-another-host"></a></h2>
<p>My first attempt was to do the installation on another computer and
then attach the hard drive to the server. This <em>might</em> work in
cases where the other computer conducting the installation and the
server are very similar. However, the only other available computers to
me are MacBooks. Using a MacBook to to run the installer created two
problems.</p>
<p>First, MacBooks only have EFI boot and the installer running on the
MacBook will boot via EFI. The installer detects that it has been booted
by EFI and partitions the hard drive for EFI<a href="#fn2" class="footnote-ref" id="fnref2" role="doc-noteref"><sup>2</sup></a>.
Unfortunately, in my case the server has no EFI support so this results
in an unbootable disk.</p>
<p>Second, my MacBook only has WiFi and no Ethernet whereas the server
only has Ethernet. The installer will create <a href="https://netplan.io/">netplan</a> files based on the network
devices available to the installer, meaning that there is no
<code>netplan</code> file created to activate the Ethernet link and
enable DHCP. Even if the hard drive was bootable on the server, it
wouldn’t be able to get an IP address on start.</p>
<p>The combination of above meant that I couldn’t run the installer on
my MacBook and then attach the hard drive to the server.</p>
<h2 id="failed-attempt-2-connecting-to-the-installer-over-ssh">Failed
Attempt 2: “Connecting to the installer over SSH”<a class="pilcrow" href="#failed-attempt-2-connecting-to-the-installer-over-ssh"></a></h2>
<p>The <a href="https://ubuntu.com/server/docs/install/general">Ubuntu
Server documentation</a> says that it’s possible to connect to the
installer over SSH:</p>
<blockquote>
<p>If the only available terminal is very basic, an alternative is to
connect via SSH. If the network is up by the time the installer starts,
instructions are offered on the initial screen in basic mode. Otherwise,
instructions are available from the help menu once networking is
configured.</p>
</blockquote>
<p>The help screen of the installer does list instructions on how to ssh
into the host as the <code>installer</code> user. However the <a href="https://github.com/canonical/subiquity/"><code>subiquity</code>
installer</a> has a <a href="https://github.com/canonical/subiquity/blob/9c47505ae06ca432c98f8cf297baee9b92d0d435/subiquity/server/server.py#L542-L550">code
snippet</a> to generate a random password every time the installer is
run:</p>
<figure class="code-block python"><div class="highlight"><pre><span></span><code><span class="n">passwd</span> <span class="o">=</span> <span class="n">rand_user_password</span><span class="p">()</span>
<span class="n">cp</span> <span class="o">=</span> <span class="n">run_command</span><span class="p">(</span><span class="s1">'chpasswd'</span><span class="p">,</span> <span class="nb">input</span><span class="o">=</span><span class="n">username</span> <span class="o">+</span> <span class="s1">':'</span><span class="o">+</span><span class="n">passwd</span><span class="o">+</span><span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">)</span>
</code></pre></div>
</figure>
<p>Without serial or VGA access to the host, it’s not possible to see
the generated password and therefore ssh into the installer.</p>
<h2 id="what-what-actually-worked">What what actually worked<a class="pilcrow" href="#what-what-actually-worked"></a></h2>
<p>The only thing that worked for me was to modify the ISO to hard code
the password of the <code>installer</code> user. It’s possible to do
this because the <a href="https://ubuntu.com/server/docs/install/autoinstall">Ubuntu Server
documentation</a> describes how the installation can be automated with
<a href="https://cloudinit.readthedocs.io/en/latest/"><code>cloud-init</code></a>.
The documentation suggests how to set up a <code>user-data</code> file
that can fully automate the installer like so:</p>
<figure class="code-block yaml"><div class="highlight"><pre><span></span><code><span class="c1">#cloud-config</span><span class="w"></span>
<span class="nt">autoinstall</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">version</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">1</span><span class="w"></span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">...</span><span class="w"></span>
</code></pre></div>
</figure>
<p>I didn’t want to autoinstall but I realized that the
<code>user-data</code> file also supports a <code>chpasswd</code>
section to set the password of a user. With this, it’s possible to
create a <code>user-data</code> file like this:</p>
<figure class="code-block yaml"><div class="highlight"><pre><span></span><code><span class="c1">#cloud-config</span><span class="w"></span>
<span class="nt">chpasswd</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">expire</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">false</span><span class="w"></span>
<span class="w"> </span><span class="nt">list</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">installer:$6$exDY1mhS4KUYCE/2$zmn9ToZwTKLhCw.b4/b.ZRTIZM30JZ4QrOQ2aOXJ8yk96xpcCof0kxKwuX1kqLG/ygbJ1f8wxED22bTL4F46P0</span><span class="w"></span>
</code></pre></div>
</figure>
<p>Which will have <code>cloud-init</code> set the password of the
<code>installer</code> user to <code>ubuntu</code>. To have the
installer use this file, I needed to provide a <a href="https://cloudinit.readthedocs.io/en/latest/topics/datasources/nocloud.html">nocloud</a>
<code>cloud-init</code> datasource. This means I needed to create a new
Ubuntu Server ISO which has the above <code>user-data</code> file in
it.</p>
<h3 id="creating-a-new-iso">Creating a new ISO</h3>
<p>To create a new ISO I first fetched a <a href="https://releases.ubuntu.com/20.04/"><code>live-server</code>
ISO</a>. Then extract it into a temporary directory:</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>7z -y x ubuntu-20.04.3-live-server-amd64.iso -oiso
</code></pre></div>
</figure>
<p>Inside the <code>iso</code> directory I created a new directory
called <code>nocloud</code> and placed the <code>user-data</code> file
and an empty <code>meta-data</code> file.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span><span class="nb">cd</span> iso
<span class="gp">$ </span>mkdir nocloud
<span class="gp">$ </span><span class="nb">cd</span> nocloud
<span class="gp">$ </span>vim user-data
<span class="gp">$ </span>touch <span class="sb">`</span>meta-data<span class="sb">`</span>
</code></pre></div>
</figure>
<p>Then for the following files in the ISO:</p>
<ul>
<li><code>isolinux/txt.cfg</code></li>
<li><code>boot/grub/grub.cfg</code></li>
<li><code>boot/grub/loopback.cfg</code></li>
</ul>
<p>I changed the kernel parameters to have
<code>ds=nocloud;s=/cdrom/nocloud</code> set. These flags will be passed
to <code>cloud-init</code> which will discover our
<code>user-data</code> file and run <code>chpasswd</code>. I then made a
new ISO by running:</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>mkisofs -o myiso.iso -ldots -allow-multidot -d -r -l -J -no-emul-boot -boot-load-size <span class="m">4</span> -boot-info-table -b isolinux/isolinux.bin -c isolinux/boot.cat iso
<span class="gp">$ </span>isohybrid myiso.iso
</code></pre></div>
</figure>
<p>The resulting <code>myiso.iso</code> can be imaged on to a USB key,
and inserted to the server. Once the server has obtained and IP address
it’s possible to ssh into the host with the user name
<code>installer</code> and password <code>ubuntu</code>. The
<code>subiquity</code> installer will start up automatically.</p>
<h2 id="conclusion">Conclusion<a class="pilcrow" href="#conclusion"></a></h2>
<p>It’s possible to install Ubuntu on a headless machine and use the
installer over SSH by building a new ISO. This ISO passes in a
configuration file for <code>cloud-init</code> which sets the password
to the <code>installer</code> user. Once the installer has booted on the
machine, just SSH in as the <code>installer</code> user.</p>
<section id="footnotes" class="footnotes footnotes-end-of-document" role="doc-endnotes">
<hr>
<ol>
<li id="fn1"><p>A HP ProLiant G7 N54L.<a href="#fnref1" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
<li id="fn2"><p>This is actually a pretty reasonable assumption to make
in 2021.<a href="#fnref2" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
</ol>
</section>
tag:zameermanji.com,2021-08-17:/blog/2021/8/17/monitoring-battery-status-on-macos-with-rust/Monitoring battery status on macOS with Rust2021-08-17T00:00:00Z2021-08-17T00:00:00Z<p>Recently I wanted to execute code<a href="#fn1" class="footnote-ref" id="fnref1" role="doc-noteref"><sup>1</sup></a> when my MacBook’s
battery charge reached a certain percentage remaining. I was using the
<a href="https://github.com/svartalf/rust-battery">rust-battery</a>
crate to check the status of the battery of my MacBook. The crate comes
with an example program called <a href="https://github.com/svartalf/rust-battery/blob/20233871e16b0e7083281df560875110a0cac93b/battery/examples/simple.rs"><code>simple.rs</code></a>
which shows how to poll for the battery state. However the problem with
this example program is the main loop.</p>
<figure class="code-block rust"><div class="highlight"><pre><span></span><code><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"{:?}"</span><span class="p">,</span><span class="w"> </span><span class="n">battery</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">thread</span>::<span class="n">sleep</span><span class="p">(</span><span class="n">Duration</span>::<span class="n">from_secs</span><span class="p">(</span><span class="mi">1</span><span class="p">));</span><span class="w"></span>
<span class="w"> </span><span class="n">manager</span><span class="p">.</span><span class="n">refresh</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="n">battery</span><span class="p">)</span><span class="o">?</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<figcaption><p>The loop inside <code>main</code> of <code>simple.rs</code></p></figcaption></figure>
<p>On my laptop <code>simple.rs</code> prints out the following:</p>
<figure class="code-block text"><div class="highlight"><pre><span></span><code>Battery { impl: MacOSDevice { source: PowerSource { io_object: IoObject(2563) } }, vendor: Some("SMP"), model: Some("bq20z451"), serial_number: None, technology: Unknown, state: Discharging, capacity: 0.8180468, temperature: Some(303.96 K^1), percentage: 0.6009494, cycle_count: Some(731), energy: 164085.69 m^2 kg^1 s^-2, energy_full: 273044.1 m^2 kg^1 s^-2, energy_full_design: 333775.63 m^2 kg^1 s^-2, energy_rate: 30.065012 m^2 kg^1 s^-3, voltage: 10.59 m^2 kg^1 s^-3 A^-1, time_to_full: None, time_to_empty: Some(6300.0 s^1) }
Battery { impl: MacOSDevice { source: PowerSource { io_object: IoObject(2563) } }, vendor: Some("SMP"), model: Some("bq20z451"), serial_number: None, technology: Unknown, state: Discharging, capacity: 0.8180468, temperature: Some(303.96 K^1), percentage: 0.6009494, cycle_count: Some(731), energy: 164085.69 m^2 kg^1 s^-2, energy_full: 273044.1 m^2 kg^1 s^-2, energy_full_design: 333775.63 m^2 kg^1 s^-2, energy_rate: 30.065012 m^2 kg^1 s^-3, voltage: 10.59 m^2 kg^1 s^-3 A^-1, time_to_full: None, time_to_empty: Some(6300.0 s^1) }
Battery { impl: MacOSDevice { source: PowerSource { io_object: IoObject(2563) } }, vendor: Some("SMP"), model: Some("bq20z451"), serial_number: None, technology: Unknown, state: Discharging, capacity: 0.8180468, temperature: Some(303.96 K^1), percentage: 0.6009494, cycle_count: Some(731), energy: 164085.69 m^2 kg^1 s^-2, energy_full: 273044.1 m^2 kg^1 s^-2, energy_full_design: 333775.63 m^2 kg^1 s^-2, energy_rate: 30.065012 m^2 kg^1 s^-3, voltage: 10.59 m^2 kg^1 s^-3 A^-1, time_to_full: None, time_to_empty: Some(6300.0 s^1) }
...
</code></pre></div>
<figcaption><p>Output of <code>simple.rs</code> from <code>rust-battery</code></p></figcaption></figure>
<p>Each iteration of the loop results in the same output as the previous
run, since the polling interval is more frequent than the battery data
updates from macOS. A better program would only refresh the battery data
once the OS has signalled some new information is available.</p>
<p>Fortunately on macOS it’s possible to be signalled when the battery
state has been updated using the <code>notify.h</code> API in the macOS
SDK. The rest of this post will show how to use <code>notify.h</code>
safely from Rust and integrate it with <a href="https://tokio.rs/">Tokio</a> so battery data can be refreshed
asynchronously.</p>
<h2 id="what-is-the-notifyh-api-">What is the <code>notify.h</code> API
?<a class="pilcrow" href="#what-is-the-notifyh-api-"></a></h2>
<p>The man page for this API is called <code>notify(3)</code>. To quote
from the man page:</p>
<blockquote>
<p>These routines allow processes to exchange stateless notification
events. Processes post notifications to a single system-wide
notification server, which then distributes notifications to client
processes that have registered to receive those notifications, including
processes run by other users.</p>
</blockquote>
<p>This API allows processes to subscribe or publish messages to a macOS
wide event bus<a href="#fn2" class="footnote-ref" id="fnref2" role="doc-noteref"><sup>2</sup></a>. Each event is given a unique string
identifier such as <code>com.apple.system.config.network_change</code><a href="#fn3" class="footnote-ref" id="fnref3" role="doc-noteref"><sup>3</sup></a>. The event is published from a
single process and arbitrary processes can subscribe to them. The
typical use that I have seen for these APIs is subscribing to OS
configuration changes such as network, timezone, hostname, power source
and other changes<a href="#fn4" class="footnote-ref" id="fnref4" role="doc-noteref"><sup>4</sup></a>.</p>
<p>For subscribing to battery charge state or power source changes, the
event name is <code>com.apple.system.powersources.timeremaining</code>.
This event will be published if the machine switches between battery
power and AC Power, and if there are any changes to the battery charge
level.</p>
<p>The entire API is exposed in the <code>notify.h</code> header, and
there are only a few functions.The entire lifecycle of subscribing to,
receiving and unsubscribing from an event is covered by the following
functions<a href="#fn5" class="footnote-ref" id="fnref5" role="doc-noteref"><sup>5</sup></a>.</p>
<figure class="code-block c"><div class="highlight"><pre><span></span><code><span class="cm">/*</span>
<span class="cm"> * Request notification by a write to a file descriptor.</span>
<span class="cm"> *</span>
<span class="cm"> * Notifications are delivered by a write to a file descriptor.</span>
<span class="cm"> * By default, a new file descriptor is created and a pointer to it</span>
<span class="cm"> * is returned as the value of "notify_fd". A file descriptor created</span>
<span class="cm"> * by a previous call to this routine may be used for notifications if</span>
<span class="cm"> * a pointer to that file descriptor is passed in to the routine and</span>
<span class="cm"> * NOTIFY_REUSE is set in the flags parameter.</span>
<span class="cm"> *</span>
<span class="cm"> * Note that the kernel limits the buffer space for queued writes on a</span>
<span class="cm"> * file descriptor. If it is important that notifications should not be</span>
<span class="cm"> * lost due to queue overflow, clients should service messages quickly,</span>
<span class="cm"> * and be careful about using the same file descriptor for notifications</span>
<span class="cm"> * for more than one name.</span>
<span class="cm"> *</span>
<span class="cm"> * Notifications are delivered by an integer value written to the</span>
<span class="cm"> * file descriptor. The value will match the notification token</span>
<span class="cm"> * for which the notification was generated.</span>
<span class="cm"> */</span><span class="w"></span>
<span class="n">OS_EXPORT</span><span class="w"> </span><span class="kt">uint32_t</span><span class="w"> </span><span class="n">notify_register_file_descriptor</span><span class="p">(</span><span class="k">const</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="n">name</span><span class="p">,</span><span class="w"> </span><span class="kt">int</span><span class="w"></span>
<span class="w"> </span><span class="o">*</span><span class="n">notify_fd</span><span class="p">,</span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">flags</span><span class="p">,</span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="o">*</span><span class="n">out_token</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="cm">/*</span>
<span class="cm"> * Cancel notification and free resources associated with a notification</span>
<span class="cm"> * token. Mach ports and file descriptor associated with a token are released</span>
<span class="cm"> * (deallocated or closed) when all registration tokens associated with</span>
<span class="cm"> * the port or file descriptor have been cancelled.</span>
<span class="cm"> */</span><span class="w"></span>
<span class="n">OS_EXPORT</span><span class="w"> </span><span class="kt">uint32_t</span><span class="w"> </span><span class="n">notify_cancel</span><span class="p">(</span><span class="kt">int</span><span class="w"> </span><span class="n">token</span><span class="p">);</span><span class="w"></span>
<span class="cm">/*</span>
<span class="cm"> * Suspend delivery of notifications for a token. Notifications for this token will be</span>
<span class="cm"> * pended and coalesced, then delivered following a matching call to notify_resume.</span>
<span class="cm"> * Calls to notify_suspend may be nested. Notifications remain suspended until</span>
<span class="cm"> * an equal number of calls have been made to notify_resume.</span>
<span class="cm"> */</span><span class="w"></span>
<span class="n">OS_EXPORT</span><span class="w"> </span><span class="kt">uint32_t</span><span class="w"> </span><span class="n">notify_suspend</span><span class="p">(</span><span class="kt">int</span><span class="w"> </span><span class="n">token</span><span class="p">);</span><span class="w"></span>
<span class="cm">/*</span>
<span class="cm"> * Removes one level of suspension for a token previously suspended</span>
<span class="cm"> * by a call to notify_suspend. Notifications will resume when a matching</span>
<span class="cm"> * call to notify_resume is made for each previous call to notify_suspend.</span>
<span class="cm"> * Notifications posted while a token is suspended are coalesced into</span>
<span class="cm"> * a single notification sent following a resumption.</span>
<span class="cm"> */</span><span class="w"></span>
<span class="n">OS_EXPORT</span><span class="w"> </span><span class="kt">uint32_t</span><span class="w"> </span><span class="n">notify_resume</span><span class="p">(</span><span class="kt">int</span><span class="w"> </span><span class="n">token</span><span class="p">);</span><span class="w"></span>
</code></pre></div>
<figcaption><p>Copied from <code>notify.h</code> in the macOS SDK</p></figcaption></figure>
<p>From the comments of the header it should be possible to do the
following:</p>
<ul>
<li>Use <code>notify_register_file_descriptor</code> to get a file
descriptor that will be readable every time an event fires.</li>
<li>The value readable should be the same as the token returned with the
file descriptor.</li>
<li>To close the file descriptor and stop getting events use
<code>notify_cancel</code>.</li>
</ul>
<p>The file descriptor doesn’t represent a file on disk, so we should be
careful to never write to it or to accidentally close it.</p>
<h2 id="creating-safe-bindings">Creating safe bindings<a class="pilcrow" href="#creating-safe-bindings"></a></h2>
<p>The comments in <code>notify.h</code> hint on what a safe Rust API
could look like. First a file descriptor and token are returned from
<code>notify_register_file_descriptor</code> and this file descriptor
can only be closed with <code>notify_cancel</code>. That tells us that a
constructor in Rust would need to call
<code>notify_register_file_descriptor</code> and return a struct with a
<a href="https://doc.rust-lang.org/std/ops/trait.Drop.html"><code>Drop</code></a>
implementation that calls <code>notify_cancel</code>.</p>
<p>Second, the file descriptor returned is not backed by a file. Passing
this file descriptor to <a href="https://doc.rust-lang.org/std/fs/struct.File.html"><code>File</code></a>
would be not appropriate since that offers methods that could write to
the file descriptor. Instead, we likely need to implement the <a href="https://doc.rust-lang.org/std/io/trait.Read.html"><code>Read</code></a>
trait on our struct.</p>
<p>With out two hints, using the <code>notify.h</code> from safe Rust
looks like something below.</p>
<figure class="code-block rust"><div class="highlight"><pre><span></span><code><span class="k">extern</span><span class="w"> </span><span class="k">crate</span><span class="w"> </span><span class="n">battery</span><span class="p">;</span><span class="w"></span>
<span class="c1">// Need the libc crate to read from a raw file descriptor</span>
<span class="k">extern</span><span class="w"> </span><span class="k">crate</span><span class="w"> </span><span class="n">libc</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">error</span>::<span class="n">Error</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">ffi</span>::<span class="n">CString</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">io</span>::<span class="n">Read</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">mem</span>::<span class="n">MaybeUninit</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">os</span>::<span class="n">raw</span>::<span class="p">{</span><span class="n">c_char</span><span class="p">,</span><span class="w"> </span><span class="n">c_int</span><span class="p">,</span><span class="w"> </span><span class="n">c_void</span><span class="p">};</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">os</span>::<span class="n">unix</span>::<span class="n">io</span>::<span class="n">RawFd</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">result</span>::<span class="nb">Result</span><span class="p">;</span><span class="w"></span>
<span class="k">const</span><span class="w"> </span><span class="n">KEY</span>: <span class="kp">&</span><span class="kt">str</span> <span class="o">=</span><span class="w"> </span><span class="s">"com.apple.system.powersources.timeremaining"</span><span class="p">;</span><span class="w"></span>
<span class="k">struct</span> <span class="nc">NotifyFd</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">fd</span>: <span class="nc">RawFd</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="n">token</span>: <span class="nc">c_int</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">impl</span><span class="w"> </span><span class="n">NotifyFd</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">fn</span> <span class="nf">new</span><span class="p">(</span><span class="n">key</span>: <span class="kp">&</span><span class="kt">str</span><span class="p">)</span><span class="w"> </span>-> <span class="nb">Result</span><span class="o"><</span><span class="bp">Self</span><span class="p">,</span><span class="w"> </span><span class="nb">Box</span><span class="o"><</span><span class="k">dyn</span><span class="w"> </span><span class="n">Error</span><span class="o">>></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">token</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">MaybeUninit</span>::<span class="o"><</span><span class="n">c_int</span><span class="o">></span>::<span class="n">uninit</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">nfd</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">MaybeUninit</span>::<span class="o"><</span><span class="n">RawFd</span><span class="o">></span>::<span class="n">uninit</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="k">unsafe</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">key</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">CString</span>::<span class="n">new</span><span class="p">(</span><span class="n">key</span><span class="p">).</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">r</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">notify_register_file_descriptor</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="n">key</span><span class="p">.</span><span class="n">as_ptr</span><span class="p">(),</span><span class="w"></span>
<span class="w"> </span><span class="n">nfd</span><span class="p">.</span><span class="n">as_mut_ptr</span><span class="p">(),</span><span class="w"></span>
<span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">token</span><span class="p">.</span><span class="n">as_mut_ptr</span><span class="p">(),</span><span class="w"></span>
<span class="w"> </span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">r</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nb">Err</span><span class="p">(</span><span class="s">"notify_register_file_descriptor failed"</span><span class="p">.</span><span class="n">into</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">token</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">unsafe</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">token</span><span class="p">.</span><span class="n">assume_init</span><span class="p">()</span><span class="w"> </span><span class="p">};</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">nfd</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">unsafe</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">nfd</span><span class="p">.</span><span class="n">assume_init</span><span class="p">()</span><span class="w"> </span><span class="p">};</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nb">Ok</span><span class="p">(</span><span class="n">NotifyFd</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">fd</span>: <span class="nc">nfd</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">token</span>: <span class="nc">token</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">});</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">impl</span><span class="w"> </span><span class="n">Read</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">NotifyFd</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">fn</span> <span class="nf">read</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">buf</span>: <span class="kp">&</span><span class="nc">mut</span><span class="w"> </span><span class="p">[</span><span class="kt">u8</span><span class="p">])</span><span class="w"> </span>-> <span class="nc">std</span>::<span class="n">io</span>::<span class="nb">Result</span><span class="o"><</span><span class="kt">usize</span><span class="o">></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">unsafe</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">r</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">libc</span>::<span class="n">read</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">fd</span><span class="p">,</span><span class="w"> </span><span class="n">buf</span><span class="p">.</span><span class="n">as_mut_ptr</span><span class="p">()</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="o">*</span><span class="k">mut</span><span class="w"> </span><span class="n">c_void</span><span class="p">,</span><span class="w"> </span><span class="n">buf</span><span class="p">.</span><span class="n">len</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">r</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="o">-</span><span class="mi">1</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nb">Err</span><span class="p">(</span><span class="n">std</span>::<span class="n">io</span>::<span class="n">Error</span>::<span class="n">last_os_error</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nb">Ok</span><span class="p">(</span><span class="n">r</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="kt">usize</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">impl</span><span class="w"> </span><span class="nb">Drop</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">NotifyFd</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">fn</span> <span class="nf">drop</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">unsafe</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">r</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">notify_cancel</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">token</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">r</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="fm">panic!</span><span class="p">(</span><span class="s">"notify_cancel failed"</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span>-> <span class="nb">Result</span><span class="o"><</span><span class="p">(),</span><span class="w"> </span><span class="nb">Box</span><span class="o"><</span><span class="k">dyn</span><span class="w"> </span><span class="n">Error</span><span class="o">>></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">manager</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">battery</span>::<span class="n">Manager</span>::<span class="n">new</span><span class="p">()</span><span class="o">?</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">battery</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">manager</span><span class="p">.</span><span class="n">batteries</span><span class="p">()</span><span class="o">?</span><span class="p">.</span><span class="n">next</span><span class="p">().</span><span class="n">ok_or</span><span class="p">(</span><span class="s">"no battery found"</span><span class="p">)</span><span class="o">??</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">nfd</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">NotifyFd</span>::<span class="n">new</span><span class="p">(</span><span class="n">KEY</span><span class="p">).</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">buf</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="p">;</span><span class="w"> </span><span class="mi">4</span><span class="p">];</span><span class="w"></span>
<span class="w"> </span><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"percent charge: {:?}"</span><span class="p">,</span><span class="w"> </span><span class="n">battery</span><span class="p">.</span><span class="n">state_of_charge</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="n">nfd</span><span class="p">.</span><span class="n">read_exact</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="n">buf</span><span class="p">).</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">v</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">c_int</span>::<span class="n">from_be_bytes</span><span class="p">(</span><span class="n">buf</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">v</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">nfd</span><span class="p">.</span><span class="n">token</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">manager</span><span class="p">.</span><span class="n">refresh</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="n">battery</span><span class="p">)</span><span class="o">?</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nb">Err</span><span class="p">(</span><span class="s">"Unknown token in file descriptor!"</span><span class="p">.</span><span class="n">into</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">extern</span><span class="w"> </span><span class="s">"C"</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">notify_register_file_descriptor</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="n">name</span>: <span class="o">*</span><span class="k">const</span><span class="w"> </span><span class="n">c_char</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">notify_fd</span>: <span class="o">*</span><span class="k">mut</span><span class="w"> </span><span class="n">c_int</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">flags</span>: <span class="nc">c_int</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">out_token</span>: <span class="o">*</span><span class="k">mut</span><span class="w"> </span><span class="n">c_int</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">)</span><span class="w"> </span>-> <span class="kt">u32</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">notify_cancel</span><span class="p">(</span><span class="n">token</span>: <span class="nc">c_int</span><span class="p">)</span><span class="w"> </span>-> <span class="kt">u32</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<figcaption><p>Using safe bindings to <code>notify.h</code> to signal when to
refresh battery state</p></figcaption></figure>
<p>On my laptop I get the following output after sometime on battery
power.</p>
<figure class="code-block text"><div class="highlight"><pre><span></span><code>percent charge: 0.5685927
percent charge: 0.5514522
...
</code></pre></div>
<figcaption><p>Output of using <code>notify.h</code> to poll for battery state
changes</p></figcaption></figure>
<p>Observe that every line has a different value, indicating the program
is not polling for battery state too frequently. The only drawback with
this approach is that using the <code>Read</code> trait blocks the
<code>main</code> until the next event arrives, preventing
concurrency.</p>
<h2 id="adding-async-with-tokio">Adding async with Tokio<a class="pilcrow" href="#adding-async-with-tokio"></a></h2>
<p>To prevent blocking while waiting for the next event, we can
integrate the above <code>NotifyFd</code> struct with <a href="https://tokio.rs/">Tokio</a>. Tokio, unlike other Rust async
runtimes, allows for arbitrary file descriptors to be added to it’s
reactor with <a href="https://docs.rs/tokio/1.10.0/tokio/io/unix/struct.AsyncFd.html"><code>AsyncFd</code></a>.
The documentation for <code>AsyncFd</code> says:</p>
<blockquote>
<p>Associates an IO object backed by a Unix file descriptor with the
tokio reactor, allowing for readiness to be polled. The file descriptor
must be of a type that can be used with the OS polling facilities (ie,
poll, epoll, kqueue, etc), such as a network socket or pipe, and the
file descriptor must have the nonblocking mode set to true.</p>
</blockquote>
<p>This implies so long as we correctly change the file descriptor to
non blocking mode and pass it to Tokio, it should be possible to
asynchronously wait for events. Based on the documentation of the trait
the code needs to do the following:</p>
<ul>
<li>Implement the <a href="https://doc.rust-lang.org/std/os/unix/io/trait.AsRawFd.html"><code>AsRawFd</code></a>
trait on a struct so Tokio can extract the underlying file descriptor to
pass to <code>kqueue</code>.</li>
<li>Before passing the struct to Tokio, the code should put the file
descriptor in non blocking mode and drain it<a href="#fn6" class="footnote-ref" id="fnref6" role="doc-noteref"><sup>6</sup></a> of
all existing data so it can be safely passed to
<code>kqueue</code>.</li>
<li>Implement the <a href="https://docs.rs/tokio/1.10.0/tokio/io/trait.AsyncRead.html"><code>AsyncRead</code></a>
to allow us to call <code>.read</code> in an async manner.</li>
</ul>
<p>A full implementation based on the code above is below. The primary
additions is adding the <code>AsRawFd</code> trait to the
<code>NotifyFd</code> struct, as well as a new struct called
<code>AsyncNotifyFd</code> which implements the integration with Tokio.
The <code>main</code> function is now async.</p>
<figure class="code-block rust"><div class="highlight"><pre><span></span><code><span class="k">extern</span><span class="w"> </span><span class="k">crate</span><span class="w"> </span><span class="n">battery</span><span class="p">;</span><span class="w"></span>
<span class="k">extern</span><span class="w"> </span><span class="k">crate</span><span class="w"> </span><span class="n">futures</span><span class="p">;</span><span class="w"></span>
<span class="k">extern</span><span class="w"> </span><span class="k">crate</span><span class="w"> </span><span class="n">libc</span><span class="p">;</span><span class="w"></span>
<span class="k">extern</span><span class="w"> </span><span class="k">crate</span><span class="w"> </span><span class="n">tokio</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">error</span>::<span class="n">Error</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">ffi</span>::<span class="n">CString</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">io</span>::<span class="n">Read</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">mem</span>::<span class="n">MaybeUninit</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">os</span>::<span class="n">raw</span>::<span class="p">{</span><span class="n">c_char</span><span class="p">,</span><span class="w"> </span><span class="n">c_int</span><span class="p">,</span><span class="w"> </span><span class="n">c_void</span><span class="p">};</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">os</span>::<span class="n">unix</span>::<span class="n">io</span>::<span class="n">AsRawFd</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">os</span>::<span class="n">unix</span>::<span class="n">io</span>::<span class="n">RawFd</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">pin</span>::<span class="n">Pin</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">result</span>::<span class="nb">Result</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">task</span>::<span class="n">Context</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">task</span>::<span class="n">Poll</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">tokio</span>::<span class="n">io</span>::<span class="n">unix</span>::<span class="n">AsyncFd</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">tokio</span>::<span class="n">io</span>::<span class="n">Interest</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">tokio</span>::<span class="n">io</span>::<span class="n">ReadBuf</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">tokio</span>::<span class="n">io</span>::<span class="p">{</span><span class="n">AsyncRead</span><span class="p">,</span><span class="w"> </span><span class="n">AsyncReadExt</span><span class="p">};</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">futures</span>::<span class="n">ready</span><span class="p">;</span><span class="w"></span>
<span class="k">const</span><span class="w"> </span><span class="n">KEY</span>: <span class="kp">&</span><span class="kt">str</span> <span class="o">=</span><span class="w"> </span><span class="s">"com.apple.system.powersources.timeremaining"</span><span class="p">;</span><span class="w"></span>
<span class="k">struct</span> <span class="nc">NotifyFd</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">fd</span>: <span class="nc">RawFd</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="n">token</span>: <span class="nc">c_int</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">impl</span><span class="w"> </span><span class="n">NotifyFd</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">fn</span> <span class="nf">new</span><span class="p">(</span><span class="n">key</span>: <span class="kp">&</span><span class="kt">str</span><span class="p">)</span><span class="w"> </span>-> <span class="nb">Result</span><span class="o"><</span><span class="bp">Self</span><span class="p">,</span><span class="w"> </span><span class="nb">Box</span><span class="o"><</span><span class="k">dyn</span><span class="w"> </span><span class="n">Error</span><span class="o">>></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Same as above ...</span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">impl</span><span class="w"> </span><span class="n">Read</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">NotifyFd</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">fn</span> <span class="nf">read</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">buf</span>: <span class="kp">&</span><span class="nc">mut</span><span class="w"> </span><span class="p">[</span><span class="kt">u8</span><span class="p">])</span><span class="w"> </span>-> <span class="nc">std</span>::<span class="n">io</span>::<span class="nb">Result</span><span class="o"><</span><span class="kt">usize</span><span class="o">></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Same as above ...</span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">impl</span><span class="w"> </span><span class="nb">Drop</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">NotifyFd</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">fn</span> <span class="nf">drop</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Same as above ...</span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="c1">// Needed for integration with Tokio</span>
<span class="k">impl</span><span class="w"> </span><span class="n">AsRawFd</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">NotifyFd</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">fn</span> <span class="nf">as_raw_fd</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-> <span class="nc">RawFd</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">fd</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">struct</span> <span class="nc">AsyncNotifyFd</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">inner</span>: <span class="nc">AsyncFd</span><span class="o"><</span><span class="n">NotifyFd</span><span class="o">></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="n">token</span>: <span class="nc">c_int</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">impl</span><span class="w"> </span><span class="n">AsyncNotifyFd</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">fn</span> <span class="nf">new</span><span class="p">(</span><span class="n">key</span>: <span class="kp">&</span><span class="kt">str</span><span class="p">)</span><span class="w"> </span>-> <span class="nb">Result</span><span class="o"><</span><span class="bp">Self</span><span class="p">,</span><span class="w"> </span><span class="nb">Box</span><span class="o"><</span><span class="k">dyn</span><span class="w"> </span><span class="n">Error</span><span class="o">>></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">nfd</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">NotifyFd</span>::<span class="n">new</span><span class="p">(</span><span class="n">key</span><span class="p">)</span><span class="o">?</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Suspend the events while we adjust the fd</span>
<span class="w"> </span><span class="k">unsafe</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">r</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">notify_suspend</span><span class="p">(</span><span class="n">nfd</span><span class="p">.</span><span class="n">token</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">r</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nb">Err</span><span class="p">(</span><span class="s">"notify_suspend failed"</span><span class="p">.</span><span class="n">into</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Set the file descriptor in non blocking mode</span>
<span class="w"> </span><span class="k">unsafe</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">flags</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">libc</span>::<span class="n">fcntl</span><span class="p">(</span><span class="n">nfd</span><span class="p">.</span><span class="n">fd</span><span class="p">,</span><span class="w"> </span><span class="n">libc</span>::<span class="n">F_GETFL</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">r</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">libc</span>::<span class="n">fcntl</span><span class="p">(</span><span class="n">nfd</span><span class="p">.</span><span class="n">fd</span><span class="p">,</span><span class="w"> </span><span class="n">libc</span>::<span class="n">F_SETFL</span><span class="p">,</span><span class="w"> </span><span class="n">flags</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">libc</span>::<span class="n">O_NONBLOCK</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">r</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nb">Err</span><span class="p">(</span><span class="s">"fcntl failed"</span><span class="p">.</span><span class="n">into</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Drain the file descriptor of all data before registering with Tokio</span>
<span class="w"> </span><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">buf</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="p">;</span><span class="w"> </span><span class="mi">4</span><span class="p">];</span><span class="w"></span>
<span class="w"> </span><span class="k">match</span><span class="w"> </span><span class="n">nfd</span><span class="p">.</span><span class="n">read_exact</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="n">buf</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">Ok</span><span class="p">(</span><span class="n">_</span><span class="p">)</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">continue</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nb">Err</span><span class="p">(</span><span class="n">e</span><span class="p">)</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">e</span><span class="p">.</span><span class="n">kind</span><span class="p">()</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">std</span>::<span class="n">io</span>::<span class="n">ErrorKind</span>::<span class="n">WouldBlock</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">break</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nb">Err</span><span class="p">(</span><span class="fm">format!</span><span class="p">(</span><span class="s">"unexpected read io error {}"</span><span class="p">,</span><span class="w"> </span><span class="n">e</span><span class="p">).</span><span class="n">into</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">nfd</span><span class="p">.</span><span class="n">token</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Register the file descriptor with tokio</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">afd</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">AsyncFd</span>::<span class="n">with_interest</span><span class="p">(</span><span class="n">nfd</span><span class="p">,</span><span class="w"> </span><span class="n">Interest</span>::<span class="n">READABLE</span><span class="p">)</span><span class="o">?</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Resume events</span>
<span class="w"> </span><span class="k">unsafe</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">r</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">notify_resume</span><span class="p">(</span><span class="n">t</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">r</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nb">Err</span><span class="p">(</span><span class="s">"notify_resume failed"</span><span class="p">.</span><span class="n">into</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nb">Ok</span><span class="p">(</span><span class="bp">Self</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">inner</span>: <span class="nc">afd</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">token</span>: <span class="nc">t</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">});</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">impl</span><span class="w"> </span><span class="n">AsyncRead</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">AsyncNotifyFd</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">fn</span> <span class="nf">poll_read</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span>: <span class="nc">Pin</span><span class="o"><&</span><span class="k">mut</span><span class="w"> </span><span class="bp">Self</span><span class="o">></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">cx</span>: <span class="kp">&</span><span class="nc">mut</span><span class="w"> </span><span class="n">Context</span><span class="o"><'</span><span class="nb">_</span><span class="o">></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">buf</span>: <span class="kp">&</span><span class="nc">mut</span><span class="w"> </span><span class="n">ReadBuf</span><span class="o"><'</span><span class="nb">_</span><span class="o">></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">)</span><span class="w"> </span>-> <span class="nc">Poll</span><span class="o"><</span><span class="n">std</span>::<span class="n">io</span>::<span class="nb">Result</span><span class="o"><</span><span class="p">()</span><span class="o">>></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">guard</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ready</span><span class="o">!</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">inner</span><span class="p">.</span><span class="n">poll_read_ready_mut</span><span class="p">(</span><span class="n">cx</span><span class="p">))</span><span class="o">?</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">r</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">guard</span><span class="p">.</span><span class="n">try_io</span><span class="p">(</span><span class="o">|</span><span class="n">x</span><span class="o">|</span><span class="w"> </span><span class="n">x</span><span class="p">.</span><span class="n">get_mut</span><span class="p">().</span><span class="n">read</span><span class="p">(</span><span class="n">buf</span><span class="p">.</span><span class="n">initialize_unfilled</span><span class="p">()));</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">r</span><span class="p">.</span><span class="n">is_ok</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">Poll</span>::<span class="n">Ready</span><span class="p">(</span><span class="n">r</span><span class="p">.</span><span class="n">unwrap</span><span class="p">().</span><span class="n">map</span><span class="p">(</span><span class="o">|</span><span class="n">r</span><span class="o">|</span><span class="w"> </span><span class="n">buf</span><span class="p">.</span><span class="n">advance</span><span class="p">(</span><span class="n">r</span><span class="p">)));</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="cp">#[tokio::main]</span><span class="w"></span>
<span class="k">async</span><span class="w"> </span><span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span>-> <span class="nb">Result</span><span class="o"><</span><span class="p">(),</span><span class="w"> </span><span class="nb">Box</span><span class="o"><</span><span class="k">dyn</span><span class="w"> </span><span class="n">Error</span><span class="o">>></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">manager</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">battery</span>::<span class="n">Manager</span>::<span class="n">new</span><span class="p">()</span><span class="o">?</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">battery</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">manager</span><span class="p">.</span><span class="n">batteries</span><span class="p">()</span><span class="o">?</span><span class="p">.</span><span class="n">next</span><span class="p">().</span><span class="n">ok_or</span><span class="p">(</span><span class="s">"no battery found"</span><span class="p">)</span><span class="o">??</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">nfd</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">AsyncNotifyFd</span>::<span class="n">new</span><span class="p">(</span><span class="n">KEY</span><span class="p">)</span><span class="o">?</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">buf</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="p">;</span><span class="w"> </span><span class="mi">4</span><span class="p">];</span><span class="w"></span>
<span class="w"> </span><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"percent charge: {:?}"</span><span class="p">,</span><span class="w"> </span><span class="n">battery</span><span class="p">.</span><span class="n">state_of_charge</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="n">nfd</span><span class="p">.</span><span class="n">read_exact</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="n">buf</span><span class="p">).</span><span class="k">await</span><span class="p">.</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">v</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">c_int</span>::<span class="n">from_be_bytes</span><span class="p">(</span><span class="n">buf</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">v</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">nfd</span><span class="p">.</span><span class="n">token</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">manager</span><span class="p">.</span><span class="n">refresh</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="n">battery</span><span class="p">)</span><span class="o">?</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nb">Err</span><span class="p">(</span><span class="s">"Unknown token in file descriptor!"</span><span class="p">.</span><span class="n">into</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">extern</span><span class="w"> </span><span class="s">"C"</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">notify_register_file_descriptor</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="n">name</span>: <span class="o">*</span><span class="k">const</span><span class="w"> </span><span class="n">c_char</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">notify_fd</span>: <span class="o">*</span><span class="k">mut</span><span class="w"> </span><span class="n">c_int</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">flags</span>: <span class="nc">c_int</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">out_token</span>: <span class="o">*</span><span class="k">mut</span><span class="w"> </span><span class="n">c_int</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">)</span><span class="w"> </span>-> <span class="kt">u32</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">notify_cancel</span><span class="p">(</span><span class="n">token</span>: <span class="nc">c_int</span><span class="p">)</span><span class="w"> </span>-> <span class="kt">u32</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Added to allow safely setting the fd to non blocking mode</span>
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">notify_suspend</span><span class="p">(</span><span class="n">token</span>: ::<span class="n">std</span>::<span class="n">os</span>::<span class="n">raw</span>::<span class="n">c_int</span><span class="p">)</span><span class="w"> </span>-> <span class="kt">u32</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">notify_resume</span><span class="p">(</span><span class="n">token</span>: ::<span class="n">std</span>::<span class="n">os</span>::<span class="n">raw</span>::<span class="n">c_int</span><span class="p">)</span><span class="w"> </span>-> <span class="kt">u32</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<figcaption><p>Asynchronously waiting for events from <code>notify.h</code> and
refreshing battery state</p></figcaption></figure>
<p>The above code produces the same output as the blocking
implementation, with the added advantage that we can concurrently do
other work while waiting on the file descriptor to be readable. For
example, this code can <code>select!</code> on other futures while
waiting for the next event.</p>
<h2 id="conclusion">Conclusion<a class="pilcrow" href="#conclusion"></a></h2>
<p>The <code>notify.h</code> API in macOS allows programs to be
signalled when events occur over a file descriptor. It’s easy to safely
use <code>notify.h</code> in Rust and integrate with Tokio for
asynchronous handling of events. If you ever want to poll for battery
state on macOS consider using <code>notify.h</code> to signal when the
battery state should be refreshed instead of polling very
frequently.</p>
<section id="footnotes" class="footnotes footnotes-end-of-document" role="doc-endnotes">
<hr>
<ol>
<li id="fn1"><p>Obviously Rust code<a href="#fnref1" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
<li id="fn2"><p>Core Foundation has an equivalent API in <a href="https://developer.apple.com/documentation/corefoundation/cfnotificationcenter-rkv">CFNotification</a>
but it requires using a <code>CFRunLoop</code> which makes the ‘async’
part of this a little harder.<a href="#fnref2" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
<li id="fn3"><p>This event is published every time the network changes,
for example the active WiFi network.<a href="#fnref3" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
<li id="fn4"><p>Searching for strings that start with
<code>com.apple.system.</code> in header files in the macOS SDK will
show all of the public notify keys.<a href="#fnref4" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
<li id="fn5"><p>There are a few additional functions such as having a <a href="https://clang.llvm.org/docs/BlockLanguageSpec.html">block</a> run
for every event or have events sent via a match port, but using those
from Rust is more difficult.<a href="#fnref5" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
<li id="fn6"><p>I wasn’t able to find definitive documentation if
draining the file descriptor is necessary but it seems like the safest
thing to do.<a href="#fnref6" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
</ol>
</section>
tag:zameermanji.com,2021-08-01:/blog/2021/8/1/counting-open-file-descriptors-on-macos/Counting open file descriptors on macOS2021-08-01T00:00:00Z2021-08-01T00:00:00Z<p>The typical way to count the number of open file descriptors by
current process on macOS is to check the contents of the
<code>/dev/fd</code> directory. This directory is not documented by
Apple, but the code for populating this directory is in the <a href="https://github.com/apple/darwin-xnu/blob/2ff845c2e033bd0ff64b5b6aa6063a1f8f65aa32/bsd/miscfs/devfs/devfs_fdesc_support.c#L157">XNU
sources</a> on GitHub. The code is borrowed from FreeBSD and this
directory is documented in the FreeBSD documentation. The
<code>fdescfs(5)</code> <a href="https://www.freebsd.org/cgi/man.cgi?query=fdescfs&sektion=5&apropos=0&manpath=FreeBSD+13.0-RELEASE+and+Ports">man
page</a> contains two facts below.</p>
<blockquote>
<p>The file system’s contents appear as a list of numbered files which
correspond to the open files of the process reading the directory. The
files /dev/fd/0 through /dev/fd/# refer to file descriptors which can be
accessed through the file system.</p>
</blockquote>
<blockquote>
<p>Note: /dev/fd/0, /dev/fd/1 and /dev/fd/2 files are created by default
when devfs alone is mounted. fdescfs creates entries for all file
descriptors opened by the process.</p>
</blockquote>
<p>Based on the above, it seems counting the number of entries under
<code>/dev/fd</code> would return the number of open file descriptors
that reference files on the file system. By default a process would have
exactly three file descriptors open, for stdin, stdout and stderr.</p>
<p>Surprisingly a the below program doesn’t return 3.</p>
<figure class="code-block rust"><div class="highlight"><pre><span></span><code><span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">num_fds</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">std</span>::<span class="n">fs</span>::<span class="n">read_dir</span><span class="p">(</span><span class="s">"/dev/fd"</span><span class="p">).</span><span class="n">unwrap</span><span class="p">().</span><span class="n">count</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"Number of open fds: {}"</span><span class="p">,</span><span class="w"> </span><span class="n">num_fds</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<figcaption><p>A naive Rust program to count open fds</p></figcaption></figure>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>./target/debug/counting-fds
<span class="go">Number of open fds: 4</span>
</code></pre></div>
<figcaption><p>Output of the naive Rust program</p></figcaption></figure>
<p>The problem here is that in order to read the contents of
<code>/dev/fs</code> the program has to allocate another file
descriptor. This additional file descriptor causes this method to always
return one more than the expected number of file descriptors. This
limitation combined with <code>/dev/fs</code> will not show other kinds
of file descriptors like sockets, means this method is limited and
inaccurate.</p>
<h2 id="a-better-approach">A Better Approach<a class="pilcrow" href="#a-better-approach"></a></h2>
<p>A better approach on macOS is to use the completely undocumented
<code>libproc.h</code> header in the macOS SDK. Within this header there
is the <code>proc_pidinfo</code> function with the following
signature.</p>
<figure class="code-block C"><div class="highlight"><pre><span></span><code><span class="kt">int</span><span class="w"> </span><span class="nf">proc_pidinfo</span><span class="p">(</span><span class="kt">int</span><span class="w"> </span><span class="n">pid</span><span class="p">,</span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">flavor</span><span class="p">,</span><span class="w"> </span><span class="kt">uint64_t</span><span class="w"> </span><span class="n">arg</span><span class="p">,</span><span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="o">*</span><span class="n">buffer</span><span class="p">,</span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">buffersize</span><span class="p">)</span><span class="w"> </span><span class="n">__OSX_AVAILABLE_STARTING</span><span class="p">(</span><span class="n">__MAC_10_5</span><span class="p">,</span><span class="w"> </span><span class="n">__IPHONE_2_0</span><span class="p">);</span><span class="w"></span>
</code></pre></div>
<figcaption><p>Copied from <code>libproc.h</code></p></figcaption></figure>
<p>Although undocumented, this is the same function that <a href="https://github.com/lsof-org/lsof/blob/234ac9ca6082084640cb1b762f6721eccc993c79/dialects/darwin/libproc/dproc.c#L496">lsof
uses</a>. The <code>pid</code> argument is self explanatory. The
<code>flavor</code> argument comes from the <code>sys/proc_info.h</code>
header. The header lists many possible values but
<code>PROC_PIDLISTFDS</code> appears to be the value needed for counting
file descriptors.</p>
<figure class="code-block C"><div class="highlight"><pre><span></span><code><span class="cm">/* Flavors for proc_pidinfo() */</span><span class="w"></span>
<span class="cp">#define PROC_PIDLISTFDS 1</span>
<span class="cp">#define PROC_PIDLISTFD_SIZE (sizeof(struct proc_fdinfo))</span>
</code></pre></div>
<figcaption><p>Copied from <code>sys/proc_info.h</code></p></figcaption></figure>
<p>In our case <code>PROC_PIDLISTFDS</code> is the desired flavor and
the <code>proc_fdinfo</code> struct shows some promising fields.</p>
<figure class="code-block C"><div class="highlight"><pre><span></span><code><span class="cm">/* defns of process file desc type */</span><span class="w"></span>
<span class="cp">#define PROX_FDTYPE_ATALK 0</span>
<span class="cp">#define PROX_FDTYPE_VNODE 1</span>
<span class="cp">#define PROX_FDTYPE_SOCKET 2</span>
<span class="cp">#define PROX_FDTYPE_PSHM 3</span>
<span class="cp">#define PROX_FDTYPE_PSEM 4</span>
<span class="cp">#define PROX_FDTYPE_KQUEUE 5</span>
<span class="cp">#define PROX_FDTYPE_PIPE 6</span>
<span class="cp">#define PROX_FDTYPE_FSEVENTS 7</span>
<span class="cp">#define PROX_FDTYPE_NETPOLICY 9</span>
<span class="k">struct</span><span class="w"> </span><span class="nc">proc_fdinfo</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kt">int32_t</span><span class="w"> </span><span class="n">proc_fd</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kt">uint32_t</span><span class="w"> </span><span class="n">proc_fdtype</span><span class="p">;</span><span class="w"></span>
<span class="p">};</span><span class="w"></span>
</code></pre></div>
<figcaption><p>Copied from <code>sys/proc_info.h</code></p></figcaption></figure>
<p>Already this approach appears to be better for two reasons. First,
<code>proc_pidinfo</code> accepts an arbitrary pid, allowing for
inspection of an arbitrary process. Two, all kinds of file descriptors
can be counted and inspected. The header indicates that file descriptors
related to files, sockets, pipes, kqueue, <a href="https://en.wikipedia.org/wiki/AppleTalk">AppleTalk</a> and more
will be returned.</p>
<p>The <code>buffer</code> and <code>buffersize</code> and the return
value are all undocumented. However the source code of
<code>proc_pidinfo</code> function <a href="https://github.com/Apple-FOSS-Mirror/Libc/blob/2ca2ae74647714acfc18674c3114b1a5d3325d7d/darwin/libproc.c#L93">is
released</a> by Apple. So it’s possible to understand these arguments by
looking at the source.</p>
<figure class="code-block C"><div class="highlight"><pre><span></span><code><span class="kt">int</span><span class="w"></span>
<span class="nf">proc_pidinfo</span><span class="p">(</span><span class="kt">int</span><span class="w"> </span><span class="n">pid</span><span class="p">,</span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">flavor</span><span class="p">,</span><span class="w"> </span><span class="kt">uint64_t</span><span class="w"> </span><span class="n">arg</span><span class="p">,</span><span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="o">*</span><span class="n">buffer</span><span class="p">,</span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">buffersize</span><span class="p">)</span><span class="w"></span>
<span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">retval</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">((</span><span class="n">retval</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">__proc_info</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span><span class="w"> </span><span class="n">pid</span><span class="p">,</span><span class="w"> </span><span class="n">flavor</span><span class="p">,</span><span class="w"> </span><span class="n">arg</span><span class="p">,</span><span class="w"> </span><span class="n">buffer</span><span class="p">,</span><span class="w"> </span><span class="n">buffersize</span><span class="p">))</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">-1</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="p">(</span><span class="mi">0</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="p">(</span><span class="n">retval</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<figcaption><p>Copied from <code>libproc.c</code> from Apple’s libc sources</p></figcaption></figure>
<p>Where <code>__proc_info</code> appears to be a <a href="https://github.com/apple/darwin-xnu/blob/2ff845c2e033bd0ff64b5b6aa6063a1f8f65aa32/bsd/kern/syscalls.master#L512">kernel
system call</a>. Inspecting the XNU sources eventually leads to the
complete implementation of the <code>PROC_PIDLISTFDS</code> flavor <a href="https://github.com/apple/darwin-xnu/blob/2ff845c2e033bd0ff64b5b6aa6063a1f8f65aa32/bsd/kern/proc_info.c#L486">in
a function</a> called <code>proc_pidfdlist</code>.</p>
<figure class="code-block C"><div class="highlight"><pre><span></span><code><span class="kt">int</span><span class="w"></span>
<span class="nf">proc_pidfdlist</span><span class="p">(</span><span class="n">proc_t</span><span class="w"> </span><span class="n">p</span><span class="p">,</span><span class="w"> </span><span class="n">user_addr_t</span><span class="w"> </span><span class="n">buffer</span><span class="p">,</span><span class="w"> </span><span class="kt">uint32_t</span><span class="w"> </span><span class="n">buffersize</span><span class="p">,</span><span class="w"> </span><span class="kt">int32_t</span><span class="w"> </span><span class="o">*</span><span class="n">retval</span><span class="p">)</span><span class="w"></span>
<span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kt">uint32_t</span><span class="w"> </span><span class="n">numfds</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kt">uint32_t</span><span class="w"> </span><span class="n">needfds</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">kbuf</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kt">uint32_t</span><span class="w"> </span><span class="n">count</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">error</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">p</span><span class="o">-></span><span class="n">p_fd</span><span class="o">-></span><span class="n">fd_nfiles</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">0</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">numfds</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="kt">uint32_t</span><span class="p">)</span><span class="n">p</span><span class="o">-></span><span class="n">p_fd</span><span class="o">-></span><span class="n">fd_nfiles</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">buffer</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="p">(</span><span class="n">user_addr_t</span><span class="p">)</span><span class="w"> </span><span class="mi">0</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">numfds</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="mi">20</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="o">*</span><span class="n">retval</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="n">numfds</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">sizeof</span><span class="p">(</span><span class="k">struct</span><span class="w"> </span><span class="nc">proc_fdinfo</span><span class="p">));</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="cm">/* buffersize is big enough atleast for one struct */</span><span class="w"></span>
<span class="w"> </span><span class="n">needfds</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">buffersize</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="k">sizeof</span><span class="p">(</span><span class="k">struct</span><span class="w"> </span><span class="nc">proc_fdinfo</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">numfds</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="n">needfds</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">numfds</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">needfds</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="n">kbuf</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">kheap_alloc</span><span class="p">(</span><span class="n">KHEAP_TEMP</span><span class="p">,</span><span class="w"> </span><span class="n">numfds</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">sizeof</span><span class="p">(</span><span class="k">struct</span><span class="w"> </span><span class="nc">proc_fdinfo</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="n">Z_WAITOK</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">Z_ZERO</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">kbuf</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="nb">NULL</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">ENOMEM</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="cm">/* cannot overflow due to count <= numfds */</span><span class="w"></span>
<span class="w"> </span><span class="n">count</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="kt">uint32_t</span><span class="p">)</span><span class="n">proc_fdlist_internal</span><span class="p">(</span><span class="n">p</span><span class="p">,</span><span class="w"> </span><span class="p">(</span><span class="k">struct</span><span class="w"> </span><span class="nc">proc_fdinfo</span><span class="w"> </span><span class="o">*</span><span class="p">)</span><span class="n">kbuf</span><span class="p">,</span><span class="w"> </span><span class="p">(</span><span class="kt">size_t</span><span class="p">)</span><span class="n">numfds</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">error</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">copyout</span><span class="p">(</span><span class="n">kbuf</span><span class="p">,</span><span class="w"> </span><span class="n">buffer</span><span class="p">,</span><span class="w"> </span><span class="n">count</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">sizeof</span><span class="p">(</span><span class="k">struct</span><span class="w"> </span><span class="nc">proc_fdinfo</span><span class="p">));</span><span class="w"></span>
<span class="w"> </span><span class="n">kheap_free</span><span class="p">(</span><span class="n">KHEAP_TEMP</span><span class="p">,</span><span class="w"> </span><span class="n">kbuf</span><span class="p">,</span><span class="w"> </span><span class="n">numfds</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">sizeof</span><span class="p">(</span><span class="k">struct</span><span class="w"> </span><span class="nc">proc_fdinfo</span><span class="p">));</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">error</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">0</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="o">*</span><span class="n">retval</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">count</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">sizeof</span><span class="p">(</span><span class="k">struct</span><span class="w"> </span><span class="nc">proc_fdinfo</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">error</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<figcaption><p>Copied from <code>proc_info.c</code> in XNU kernel sources.</p></figcaption></figure>
<p>Based on this code it’s clear that <code>buffer</code> is supposed to
be filled with <code>proc_fdinfo</code> structs that are returned from
the call and <code>buffersize</code> is the size of this buffer in
bytes. The return value is the number of entries successfully written to
the buffer. If the function is called with <code>NULL</code> for the
buffer, it returns the number of file descriptors plus 20 as a suggested
buffer size. <code>arg</code> appears to be completely unused.</p>
<p>Given all of this, writing a program to use <code>proc_pidinfo</code>
to accurately count the number of open fds in the current process is
straight forward.</p>
<figure class="code-block rust"><div class="highlight"><pre><span></span><code><span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">error</span>::<span class="n">Error</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">os</span>::<span class="n">raw</span>::<span class="p">{</span><span class="n">c_int</span><span class="p">,</span><span class="w"> </span><span class="n">c_void</span><span class="p">};</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">ptr</span>::<span class="n">null_mut</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">convert</span>::<span class="n">TryInto</span><span class="p">;</span><span class="w"></span>
<span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">num_fds</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">count_open_fds</span><span class="p">().</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"Number of open fds: {}"</span><span class="p">,</span><span class="w"> </span><span class="n">num_fds</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">count_open_fds</span><span class="p">()</span><span class="w"> </span>-> <span class="nb">Result</span><span class="o"><</span><span class="kt">usize</span><span class="p">,</span><span class="w"> </span><span class="nb">Box</span><span class="o"><</span><span class="k">dyn</span><span class="w"> </span><span class="n">Error</span><span class="o">>></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">pid</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">std</span>::<span class="n">process</span>::<span class="n">id</span><span class="p">()</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">c_int</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">fds_flavor</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">c_int</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">buffer_size_bytes</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">unsafe</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">proc_pidinfo</span><span class="p">(</span><span class="n">pid</span><span class="p">,</span><span class="w"> </span><span class="n">fds_flavor</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">null_mut</span><span class="p">(),</span><span class="w"> </span><span class="mi">0</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">};</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">buffer_size_bytes</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nb">Err</span><span class="p">(</span><span class="s">"proc_pidinfo failed"</span><span class="p">.</span><span class="n">into</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">fds_buffer_length</span><span class="w"> </span>: <span class="kt">usize</span> <span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="n">buffer_size_bytes</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="kt">usize</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="n">std</span>::<span class="n">mem</span>::<span class="n">size_of</span>::<span class="o"><</span><span class="n">proc_fd_info</span><span class="o">></span><span class="p">()).</span><span class="n">try_into</span><span class="p">()</span><span class="o">?</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">buf</span>: <span class="nb">Vec</span><span class="o"><</span><span class="n">proc_fd_info</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="fm">vec!</span><span class="p">[</span><span class="n">proc_fd_info</span>::<span class="n">new</span><span class="p">();</span><span class="w"> </span><span class="n">fds_buffer_length</span><span class="p">];</span><span class="w"></span>
<span class="w"> </span><span class="n">buf</span><span class="p">.</span><span class="n">shrink_to_fit</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">actual_buffer_size_bytes</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">unsafe</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">proc_pidinfo</span><span class="p">(</span><span class="n">pid</span><span class="p">,</span><span class="w"> </span><span class="n">fds_flavor</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">buf</span><span class="p">.</span><span class="n">as_mut_ptr</span><span class="p">()</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="o">*</span><span class="k">mut</span><span class="w"> </span><span class="n">c_void</span><span class="p">,</span><span class="w"> </span><span class="n">buffer_size_bytes</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">};</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">actual_buffer_size_bytes</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nb">Err</span><span class="p">(</span><span class="s">"proc_pidinfo failed"</span><span class="p">.</span><span class="n">into</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">actual_buffer_size_bytes</span><span class="w"> </span><span class="o">>=</span><span class="w"> </span><span class="n">buffer_size_bytes</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nb">Err</span><span class="p">(</span><span class="s">"allocated buffer too small"</span><span class="p">.</span><span class="n">into</span><span class="p">())</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="n">buf</span><span class="p">.</span><span class="n">truncate</span><span class="p">(</span><span class="n">actual_buffer_size_bytes</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="kt">usize</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="n">std</span>::<span class="n">mem</span>::<span class="n">size_of</span>::<span class="o"><</span><span class="n">proc_fd_info</span><span class="o">></span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nb">Ok</span><span class="p">(</span><span class="n">buf</span><span class="p">.</span><span class="n">len</span><span class="p">());</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="c1">// Copying the related definitions from the headers</span>
<span class="cp">#[repr(C)]</span><span class="w"></span>
<span class="cp">#[derive(Copy, Clone)]</span><span class="w"></span>
<span class="k">struct</span> <span class="nc">proc_fd_info</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="n">proc_fd</span>: <span class="kt">i32</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="n">proc_fd_type</span>: <span class="kt">u32</span>
<span class="p">}</span><span class="w"></span>
<span class="k">impl</span><span class="w"> </span><span class="n">proc_fd_info</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">fn</span> <span class="nf">new</span><span class="p">()</span><span class="w"> </span>-> <span class="nc">proc_fd_info</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="bp">Self</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">proc_fd</span>: <span class="mi">0</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">proc_fd_type</span>: <span class="mi">0</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">extern</span><span class="w"> </span><span class="s">"C"</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">fn</span> <span class="nf">proc_pidinfo</span><span class="p">(</span><span class="n">pid</span>: <span class="nc">c_int</span><span class="p">,</span><span class="w"> </span><span class="n">flavor</span>: <span class="nc">c_int</span><span class="p">,</span><span class="w"> </span><span class="n">arg</span>: <span class="kt">u64</span><span class="p">,</span><span class="w"> </span><span class="n">buffer</span>: <span class="o">*</span><span class="k">mut</span><span class="w"> </span><span class="n">c_void</span><span class="p">,</span><span class="w"> </span><span class="n">size</span>: <span class="nc">c_int</span><span class="p">)</span><span class="w"> </span>-> <span class="nc">c_int</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<figcaption><p>A Rust program that uses <code>proc_pidinfo</code> to count the
number of open fds</p></figcaption></figure>
<p>Running this program now returns the expected output.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>./target/debug/counting-fds
<span class="go">Number of open fds: 3</span>
</code></pre></div>
</figure>
<h2 id="conclusion">Conclusion<a class="pilcrow" href="#conclusion"></a></h2>
<p>Using the <code>proc_pidinfo</code> function is a better way of
counting the number of open file descriptors for the current process on
macOS. <code>proc_pidinfo</code> allows for inspecting an arbitrary
process and allows for accurate results. In addition, the results can be
filtered on type of file descriptor such as sockets or pipes.</p>
tag:zameermanji.com,2021-07-19:/blog/2021/7/19/trying-and-failing-to-leak-memory-with-rust-and-foundation/Trying and failing to leak memory with Rust and Foundation2021-07-19T00:00:00Z2021-07-19T00:00:00Z<p>One of the oldest frameworks on macOS is <a href="https://developer.apple.com/documentation/foundation">Foundation</a>.
It’s impossible to interact with an Apple provided API that is not built
upon Foundation in some way. This API is only exposed via Objective-C
which makes calling it from Rust a complicated endeavour. Most of this
complexity stems from managing memory and ensuring that memory does not
leak or using previously freed memory.</p>
<p>Objective-C has a complicated history of memory management. Initially
Objective-C <em>only</em> had manual reference counting. It was expected
that programmers use a combination of <code>retain</code> and
<code>release</code> to adjust the reference count, and when the last
<code>release</code> on the object was run, the Objective-C runtime
would free the object. The runtime also contains an <a href="https://developer.apple.com/documentation/foundation/nsautoreleasepool">Autorelease
Pool</a> where objects could be added to the pool with
<code>autorelease</code>. Eventually <code>drain</code> could be called
on the pool and it would call <code>release</code> on everything added
to the pool. This provides a tool that allows some APIs to allocate an
object and then immediately call <code>autorelease</code> on it. The
caller can choose to call <code>retain</code> it or if the use is
temporary, do nothing and expect the object to be freed when the pool is
drained as some point in the future.</p>
<p>Complicating matters is a brief use of <a href="https://developer.apple.com/documentation/foundation/nsgarbagecollector">Garbage
Collection</a> with Objective-C on macOS, first available on macOS 10.5
and then later deprecated on 10.8. The goal here was to remove manual
the use of <code>release</code> <code>retain</code> and
<code>autorelease</code> in application code. However it was deprecated
and replaced with <a href="https://developer.apple.com/library/archive/releasenotes/ObjectiveC/RN-TransitioningToARC/Introduction/Introduction.html#//apple_ref/doc/uid/TP40011226">Automatic
Reference Counting</a> where the compiler would infer ownership of
objects<a href="#fn1" class="footnote-ref" id="fnref1" role="doc-noteref"><sup>1</sup></a> and <a href="https://clang.llvm.org/docs/AutomaticReferenceCounting.html">insert</a>
<code>retain</code> and <code>release</code> statements as needed. ARC
also introduced special syntax in Objective-C for managing Autorelease
Pools, which would ensure that the pool was drained at the end of a
scope instead of requiring the programmer to call
<code>drain</code>.</p>
<p>Now that ARC is the preferred way of interacting with Foundation APIs
in Objective-C, it makes calling Foundation APIs from Rust difficult.
The Rust side has to carefully manage calls to <code>release</code> and
<code>retain</code> to emulate ARC.</p>
<p>Further Apple’s <a href="https://developer.apple.com/documentation/foundation/nsautoreleasepool">documentation</a>
says the Rust side has to manage an <code>autorelease</code> pool and
call <code>drain</code> when using Foundation outside of AppKit or
Objective-C and it might not be safe to send objects added to a pool
across threads.</p>
<blockquote>
<p>Threads</p>
<p>If you are making Cocoa calls outside of the Application Kit’s main
thread—for example if you create a Foundation-only application or if you
detach a thread—you need to create your own autorelease pool.</p>
<p>If your application or thread is long-lived and potentially generates
a lot of autoreleased objects, you should periodically drain and create
autorelease pools (like the Application Kit does on the main thread);
otherwise, autoreleased objects accumulate and your memory footprint
grows. If, however, your detached thread does not make Cocoa calls, you
do not need to create an autorelease pool.</p>
</blockquote>
<p>With all of this in mind, I would expect a very naive Rust program
which converts a Rust <code>String</code> to <code>NSString</code> to
leak and observe the leaks using the built in <code>leaks</code> tool
from XCode. However, a naive program doesn’t leak as Apple’s
documentation implies.</p>
<h2 id="a-program-that-should-leak-but-doesnt">A program that should
leak but doesn’t<a class="pilcrow" href="#a-program-that-should-leak-but-doesnt"></a></h2>
<p>Lets create a Rust program called <code>a-leaky-bucket</code> which
tries to leak memory.</p>
<figure class="code-block shell-session"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>cargo new --bin a-leaky-bucket
<span class="go"> Created binary (application) `a-leaky-bucket` package</span>
</code></pre></div>
</figure>
<p>From my <a href="/blog/2021/7/13/using-bindgen-with-system-frameworks-on-macos/">previous
post</a> on bindgen it’s pretty easy to create some bindings to
<code>NSString</code>. For Objective-C related code, it’s important to
note that <code>bindgen</code> expects the application to depend on the
<a href="https://crates.io/crates/objc">objc</a> crate which provides
macros that the generated code will use.</p>
<p>Generating bindings is pretty straight forward with a
<code>wrapper.h</code> that references <code>NSString.h</code>.</p>
<figure class="code-block c"><div class="highlight"><pre><span></span><code><span class="cp">#include</span><span class="w"> </span><span class="cpf"><Foundation/NSString.h></span><span class="cp"></span>
</code></pre></div>
<figcaption><p>The <code>wrapper.h</code> to give <code>bindgen</code></p></figcaption></figure>
<p>Feeding the <code>wrapper.h</code> to <code>bindgen</code> with
support for Objective-C gives us the bindings. Due to some bugs in the
<code>bindgen</code> heuristics, it’s important to pass
<code>--no-derive-copy</code> and <code>--no-derive-debug</code>
otherwise the generated bindings will not compile.</p>
<figure class="code-block shell-session"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>bindgen --no-derive-copy --no-derive-debug wrapper.h -- -isysroot<span class="k">$(</span>xcrun --sdk macosx --show-sdk-path<span class="k">)</span> -x objective-c > ./src/nsstring.rs
</code></pre></div>
<figcaption><p>Generating the bindings to <code>NSString</code></p></figcaption></figure>
<p>To use these bindings the <code>objc</code> crate needs to be added
to the <code>Cargo.toml</code>.</p>
<figure class="code-block toml"><div class="highlight"><pre><span></span><code><span class="k">[package]</span><span class="w"></span>
<span class="n">name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"a-leaky-bucket"</span><span class="w"></span>
<span class="n">version</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"0.1.0"</span><span class="w"></span>
<span class="n">edition</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"2018"</span><span class="w"></span>
<span class="k">[dependencies]</span><span class="w"></span>
<span class="n">objc</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">'0.2.7'</span><span class="w"></span>
</code></pre></div>
<figcaption><p><code>Cargo.toml</code></p></figcaption></figure>
<p>Then <code>main.rs</code> can use these bindings to convert a Rust
<code>String</code> to <code>NSString</code>.</p>
<figure class="code-block rust"><div class="highlight"><pre><span></span><code><span class="cp">#[macro_use]</span><span class="w"></span>
<span class="k">extern</span><span class="w"> </span><span class="k">crate</span><span class="w"> </span><span class="n">objc</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">ffi</span>::<span class="p">{</span><span class="n">CString</span><span class="p">,</span><span class="w"> </span><span class="n">CStr</span><span class="p">};</span><span class="w"></span>
<span class="cp">#[allow(dead_code)]</span><span class="w"></span>
<span class="cp">#[allow(non_camel_case_types)]</span><span class="w"></span>
<span class="cp">#[allow(non_snake_case)]</span><span class="w"></span>
<span class="cp">#[allow(non_upper_case_globals)]</span><span class="w"></span>
<span class="k">mod</span> <span class="nn">nsstring</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">nsstring</span>::<span class="n">NSString</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">nsstring</span>::<span class="n">NSString_NSStringExtensionMethods</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">nsstring</span>::<span class="n">NSUTF8StringEncoding</span><span class="p">;</span><span class="w"></span>
<span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">leak</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="k">unsafe</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">cstr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">CStr</span>::<span class="n">from_ptr</span><span class="p">(</span><span class="n">s</span><span class="p">.</span><span class="n">cStringUsingEncoding_</span><span class="p">(</span><span class="n">NSUTF8StringEncoding</span><span class="p">)).</span><span class="n">to_str</span><span class="p">().</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"{}"</span><span class="p">,</span><span class="w"> </span><span class="n">cstr</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">fn</span> <span class="nf">leak</span><span class="p">()</span><span class="w"> </span>-> <span class="nc">NSString</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">CString</span>::<span class="n">new</span><span class="p">(</span><span class="s">"a leaky string"</span><span class="p">).</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="k">unsafe</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">NSString</span><span class="p">(</span><span class="n">NSString</span>::<span class="n">stringWithUTF8String_</span><span class="p">(</span><span class="n">i</span><span class="p">.</span><span class="n">into_boxed_c_str</span><span class="p">().</span><span class="n">as_ptr</span><span class="p">()))</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="cp">#[link(name = </span><span class="s">"Foundation"</span><span class="cp">, kind = </span><span class="s">"framework"</span><span class="cp">)]</span><span class="w"></span>
<span class="k">extern</span><span class="w"> </span><span class="s">"C"</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<figcaption><p><code>main.rs</code></p></figcaption></figure>
<p>Running the code results in “a leaky string” printed as expected.</p>
<figure class="code-block shell-session"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>./target/debug/a-leaky-bucket
<span class="go">a leaky string</span>
</code></pre></div>
</figure>
<p>However the <code>leaks</code> tool from XCode does not show any
leaks in the code despite the code not making a single
<code>release</code> call after the <code>NSString</code> was
allocated.</p>
<figure class="code-block shell-session"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>leaks --atExit -- ./target/debug/a-leaky-bucket
<span class="go">a leaky string</span>
<span class="go">Process: a-leaky-bucket [47670]</span>
<span class="go">Path: /Users/USER/*/a-leaky-bucket</span>
<span class="go">Load Address: 0x109093000</span>
<span class="go">Identifier: a-leaky-bucket</span>
<span class="go">Version: ???</span>
<span class="go">Code Type: X86-64</span>
<span class="go">Platform: macOS</span>
<span class="go">Parent Process: leaks [47669]</span>
<span class="go">Date/Time: 2021-07-18 22:14:24.959 -0400</span>
<span class="go">Launch Time: 2021-07-18 22:14:24.358 -0400</span>
<span class="go">OS Version: macOS 11.4 (20F71)</span>
<span class="go">Report Version: 7</span>
<span class="go">Analysis Tool: /Applications/Xcode.app/Contents/Developer/usr/bin/leaks</span>
<span class="go">Analysis Tool Version: Xcode 12.5.1 (12E507)</span>
<span class="go">Physical footprint: 660K</span>
<span class="go">Physical footprint (peak): 660K</span>
<span class="go">----</span>
<span class="go">leaks Report Version: 4.0</span>
<span class="go">Process 47670: 503 nodes malloced for 35 KB</span>
<span class="go">Process 47670: 0 leaks for 0 total leaked bytes.</span>
</code></pre></div>
</figure>
<h2 id="looking-under-the-hood">Looking under the hood<a class="pilcrow" href="#looking-under-the-hood"></a></h2>
<p>Trying to understand this behaviour requires taking a look into
Foundation. With <code>rust-lldb</code> it’s pretty straight forward to
set some breakpoints and try to understand what is calling
<code>release</code> eventually on the created <code>NSString</code>.
<code>rust-lldb</code> is distributed along with <code>rustc</code> and
<code>cargo</code> so it should be available. We can set a breakpoint at
<code>main</code> and once it is hit, break at any Objective-C runtime
function with the word <code>release</code> in it.</p>
<figure class="code-block shell-session"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>rust-lldb ./target/debug/a-leaky-bucket
<span class="gp gp-VirtualEnv">(lldb)</span> <span class="go">command script import "/Users/zmanji/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/etc/lldb_lookup.py"</span>
<span class="gp gp-VirtualEnv">(lldb)</span> <span class="go">command source -s 0 '/Users/zmanji/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/etc/lldb_commands'</span>
<span class="go">Executing commands in '/Users/zmanji/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/etc/lldb_commands'.</span>
<span class="gp gp-VirtualEnv">(lldb)</span> <span class="go">type synthetic add -l lldb_lookup.synthetic_lookup -x ".*" --category Rust</span>
<span class="gp gp-VirtualEnv">(lldb)</span> <span class="go">type summary add -F lldb_lookup.summary_lookup -e -x -h "^(alloc::([a-z_]+::)+)String$" --category Rust</span>
<span class="gp gp-VirtualEnv">(lldb)</span> <span class="go">type summary add -F lldb_lookup.summary_lookup -e -x -h "^&str$" --category Rust</span>
<span class="gp gp-VirtualEnv">(lldb)</span> <span class="go">type summary add -F lldb_lookup.summary_lookup -e -x -h "^&\\[.+\\]$" --category Rust</span>
<span class="gp gp-VirtualEnv">(lldb)</span> <span class="go">type summary add -F lldb_lookup.summary_lookup -e -x -h "^(std::ffi::([a-z_]+::)+)OsString$" --category Rust</span>
<span class="gp gp-VirtualEnv">(lldb)</span> <span class="go">type summary add -F lldb_lookup.summary_lookup -e -x -h "^(alloc::([a-z_]+::)+)Vec<.+>$" --category Rust</span>
<span class="gp gp-VirtualEnv">(lldb)</span> <span class="go">type summary add -F lldb_lookup.summary_lookup -e -x -h "^(alloc::([a-z_]+::)+)VecDeque<.+>$" --category Rust</span>
<span class="gp gp-VirtualEnv">(lldb)</span> <span class="go">type summary add -F lldb_lookup.summary_lookup -e -x -h "^(alloc::([a-z_]+::)+)BTreeSet<.+>$" --category Rust</span>
<span class="gp gp-VirtualEnv">(lldb)</span> <span class="go">type summary add -F lldb_lookup.summary_lookup -e -x -h "^(alloc::([a-z_]+::)+)BTreeMap<.+>$" --category Rust</span>
<span class="gp gp-VirtualEnv">(lldb)</span> <span class="go">type summary add -F lldb_lookup.summary_lookup -e -x -h "^(std::collections::([a-z_]+::)+)HashMap<.+>$" --category Rust</span>
<span class="gp gp-VirtualEnv">(lldb)</span> <span class="go">type summary add -F lldb_lookup.summary_lookup -e -x -h "^(std::collections::([a-z_]+::)+)HashSet<.+>$" --category Rust</span>
<span class="gp gp-VirtualEnv">(lldb)</span> <span class="go">type summary add -F lldb_lookup.summary_lookup -e -x -h "^(alloc::([a-z_]+::)+)Rc<.+>$" --category Rust</span>
<span class="gp gp-VirtualEnv">(lldb)</span> <span class="go">type summary add -F lldb_lookup.summary_lookup -e -x -h "^(alloc::([a-z_]+::)+)Arc<.+>$" --category Rust</span>
<span class="gp gp-VirtualEnv">(lldb)</span> <span class="go">type summary add -F lldb_lookup.summary_lookup -e -x -h "^(core::([a-z_]+::)+)Cell<.+>$" --category Rust</span>
<span class="gp gp-VirtualEnv">(lldb)</span> <span class="go">type summary add -F lldb_lookup.summary_lookup -e -x -h "^(core::([a-z_]+::)+)Ref<.+>$" --category Rust</span>
<span class="gp gp-VirtualEnv">(lldb)</span> <span class="go">type summary add -F lldb_lookup.summary_lookup -e -x -h "^(core::([a-z_]+::)+)RefMut<.+>$" --category Rust</span>
<span class="gp gp-VirtualEnv">(lldb)</span> <span class="go">type summary add -F lldb_lookup.summary_lookup -e -x -h "^(core::([a-z_]+::)+)RefCell<.+>$" --category Rust</span>
<span class="gp gp-VirtualEnv">(lldb)</span> <span class="go">type category enable Rust</span>
<span class="gp gp-VirtualEnv">(lldb)</span> <span class="go">target create "./target/debug/a-leaky-bucket"</span>
<span class="go">Current executable set to '/Users/zmanji/code/a-leaky-bucket/target/debug/a-leaky-bucket' (x86_64).</span>
<span class="gp gp-VirtualEnv">(lldb)</span> <span class="go">break set --name main</span>
<span class="go">Breakpoint 1: 11 locations.</span>
<span class="gp gp-VirtualEnv">(lldb)</span> <span class="go">r</span>
<span class="go">Process 56685 launched: '/Users/zmanji/code/a-leaky-bucket/target/debug/a-leaky-bucket' (x86_64)</span>
<span class="go">Process 56685 stopped</span>
<span class="go">* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.2</span>
<span class="go"> frame #0: 0x0000000100001cc0 a-leaky-bucket`main</span>
<span class="go">a-leaky-bucket`main:</span>
<span class="go">-> 0x100001cc0 <+0>: pushq %rbp</span>
<span class="go"> 0x100001cc1 <+1>: movq %rsp, %rbp</span>
<span class="go"> 0x100001cc4 <+4>: movq %rsi, %rdx</span>
<span class="go"> 0x100001cc7 <+7>: movslq %edi, %rsi</span>
<span class="go">Target 0: (a-leaky-bucket) stopped.</span>
<span class="gp gp-VirtualEnv">(lldb)</span> <span class="go">c</span>
<span class="go">Process 56685 resuming</span>
<span class="go">Process 56685 stopped</span>
<span class="go">* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1</span>
<span class="go"> frame #0: 0x0000000100001b1b a-leaky-bucket`a_leaky_bucket::main::h00757089d677f6a2 at main.rs:18:13</span>
<span class="go"> 15 use crate::nsstring::PNSObject;</span>
<span class="go"> 16</span>
<span class="go"> 17 fn main() {</span>
<span class="go">-> 18 let s = leak();</span>
<span class="go"> 19 unsafe {</span>
<span class="go"> 20 let cstr = CStr::from_ptr(s.cStringUsingEncoding_(NSUTF8StringEncoding)).to_str().unwrap();</span>
<span class="go"> 21 println!("{}", cstr);</span>
<span class="go">Target 0: (a-leaky-bucket) stopped.</span>
<span class="gp gp-VirtualEnv">(lldb)</span> <span class="go">break set -r .*release.* -s libobjc.A.dylib</span>
<span class="go">Breakpoint 2: 46 locations.</span>
</code></pre></div>
<figcaption><p>Using <code>rust-lldb</code> to prepare breakpoints in the
<code>Objective-C</code> runtime</p></figcaption></figure>
<p>With the breakpoints set some curious pieces of code were hit. First
there is proof that the <code>stringWithUTF8String</code> method calls
<code>autorelease</code> on the returned <code>NSString</code>.</p>
<figure class="code-block shell-session"><div class="highlight"><pre><span></span><code><span class="gp gp-VirtualEnv">(lldb)</span> <span class="go">c</span>
<span class="go">Process 56685 resuming</span>
<span class="go">Process 56685 stopped</span>
<span class="go">* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 2.37</span>
<span class="go"> frame #0: 0x00007fff2020e5e0 libobjc.A.dylib`objc_autorelease</span>
<span class="go">libobjc.A.dylib`objc_autorelease:</span>
<span class="go">-> 0x7fff2020e5e0 <+0>: testq %rdi, %rdi</span>
<span class="go"> 0x7fff2020e5e3 <+3>: je 0x7fff2020e659 ; <+121></span>
<span class="go"> 0x7fff2020e5e5 <+5>: movl %edi, %eax</span>
<span class="go"> 0x7fff2020e5e7 <+7>: andl $0x1, %eax</span>
<span class="go">Target 0: (a-leaky-bucket) stopped.</span>
<span class="gp gp-VirtualEnv">(lldb)</span> <span class="go">bt</span>
<span class="go">error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0</span>
<span class="go">* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 2.37</span>
<span class="go"> * frame #0: 0x00007fff2020e5e0 libobjc.A.dylib`objc_autorelease</span>
<span class="go"> frame #1: 0x00000001000019a5 a-leaky-bucket`_$LT$$LP$A$C$$RP$$u20$as$u20$objc..message..MessageArguments$GT$::invoke::h330b94c840fc95e2(imp=(libobjc.A.dylib`objc_msgSend), obj=0x00007fff800dc8d0, sel=Sel @ 0x00007ffeefbfee40, (null)=(*const i8) @ 0x00007ffeefbfee70) at mod.rs:128:17</span>
<span class="go"> frame #2: 0x0000000100001884 a-leaky-bucket`objc::message::platform::send_unverified::hdd48c548e61778ec(obj=0x00007fff800dc8d0, sel=Sel @ 0x00007ffeefbfeee0, args=(*const i8) @ 0x00007ffeefbfef08) at mod.rs:27:9</span>
<span class="go"> frame #3: 0x0000000100000fe5 a-leaky-bucket`a_leaky_bucket::nsstring::NSString_NSStringExtensionMethods::stringWithUTF8String_::h13adb976a6c4f01a [inlined] objc::message::send_message::hb975109fc5b4cbdd(obj=0x00007fff800dc8d0, sel=Sel @ 0x00007ffeefbff128, args=(*const i8) @ 0x00007ffeefbff148) at mod.rs:178:5</span>
<span class="go"> frame #4: 0x0000000100000fcd a-leaky-bucket`a_leaky_bucket::nsstring::NSString_NSStringExtensionMethods::stringWithUTF8String_::h13adb976a6c4f01a(nullTerminatedCString="a leaky string") at nsstring.rs:7034</span>
<span class="go"> frame #5: 0x0000000100001c74 a-leaky-bucket`a_leaky_bucket::leak::h9a1e6714fee9d36c at main.rs:28:17</span>
<span class="go"> frame #6: 0x0000000100001b20 a-leaky-bucket`a_leaky_bucket::main::h00757089d677f6a2 at main.rs:18:13</span>
<span class="go"> frame #7: 0x000000010000156e a-leaky-bucket`core::ops::function::FnOnce::call_once::haeff37c118186a6c((null)=(a-leaky-bucket`a_leaky_bucket::main::h00757089d677f6a2 at main.rs:17), (null)=<unavailable>) at function.rs:227:5</span>
<span class="go"> frame #8: 0x0000000100001351 a-leaky-bucket`std::sys_common::backtrace::__rust_begin_short_backtrace::hfd92fb564fd3b2e4(f=(a-leaky-bucket`a_leaky_bucket::main::h00757089d677f6a2 at main.rs:17)) at backtrace.rs:125:18</span>
<span class="go"> frame #9: 0x0000000100001924 a-leaky-bucket`std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::h72100a97160f9dee at rt.rs:66:18</span>
<span class="go"> frame #10: 0x0000000100020fe4 a-leaky-bucket`std::rt::lang_start_internal::h0c37a46739a0311d [inlined] core::ops::function::impls::_$LT$impl$u20$core..ops..function..FnOnce$LT$A$GT$$u20$for$u20$$RF$F$GT$::call_once::h3b22ce68aa2879c2 at function.rs:259:13 [opt]</span>
<span class="go"> frame #11: 0x0000000100020fdd a-leaky-bucket`std::rt::lang_start_internal::h0c37a46739a0311d [inlined] std::panicking::try::do_call::h51a4853c94b1bdea at panicking.rs:379 [opt]</span>
<span class="go"> frame #12: 0x0000000100020fdd a-leaky-bucket`std::rt::lang_start_internal::h0c37a46739a0311d [inlined] std::panicking::try::ha45bc5eab1f103eb at panicking.rs:343 [opt]</span>
<span class="go"> frame #13: 0x0000000100020fdd a-leaky-bucket`std::rt::lang_start_internal::h0c37a46739a0311d [inlined] std::panic::catch_unwind::h191bc002afc126a7 at panic.rs:431 [opt]</span>
<span class="go"> frame #14: 0x0000000100020fdd a-leaky-bucket`std::rt::lang_start_internal::h0c37a46739a0311d at rt.rs:51 [opt]</span>
<span class="go"> frame #15: 0x00000001000018fe a-leaky-bucket`std::rt::lang_start::h62e28d4b373c8b88(main=(a-leaky-bucket`a_leaky_bucket::main::h00757089d677f6a2 at main.rs:17), argc=1, argv=0x00007ffeefbff418) at rt.rs:65:5</span>
<span class="go"> frame #16: 0x0000000100001cd6 a-leaky-bucket`main + 22</span>
<span class="go"> frame #17: 0x00007fff20386f5d libdyld.dylib`start + 1</span>
<span class="go"> frame #18: 0x00007fff20386f5d libdyld.dylib`start + 1</span>
</code></pre></div>
<figcaption><p>The first breakpoint hit in the Objective-C runtime.</p></figcaption></figure>
<p>The backtrace shows that the <code>autorelease</code> method was
called from inside <code>stringWithUTF8String</code> method. I would
expect this to fail or leak memory since there is no Autorelease Pool
created. However continuing a few times results in a backtrace deep in
the Objective-C runtime.</p>
<figure class="code-block shell-session"><div class="highlight"><pre><span></span><code><span class="go">* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 2.13</span>
<span class="go"> * frame #0: 0x00007fff20228562 libobjc.A.dylib`AutoreleasePoolPage::autoreleaseNoPage(objc_object*)</span>
<span class="go"> frame #1: 0x00007fff2020d49c libobjc.A.dylib`objc_object::rootAutorelease2() + 32</span>
<span class="go"> frame #2: 0x00000001000019a5 a-leaky-bucket`_$LT$$LP$A$C$$RP$$u20$as$u20$objc..message..MessageArguments$GT$::invoke::h330b94c840fc95e2(imp=(libobjc.A.dylib`objc_msgSend), obj=0x00007fff800dc8d0, sel=Sel @ 0x00007ffeefbfee40, (null)=(*const i8) @ 0x00007ffeefbfee70) at mod.rs:128:17</span>
<span class="go"> frame #3: 0x0000000100001884 a-leaky-bucket`objc::message::platform::send_unverified::hdd48c548e61778ec(obj=0x00007fff800dc8d0, sel=Sel @ 0x00007ffeefbfeee0, args=(*const i8) @ 0x00007ffeefbfef08) at mod.rs:27:9</span>
<span class="go"> frame #4: 0x0000000100000fe5 a-leaky-bucket`a_leaky_bucket::nsstring::NSString_NSStringExtensionMethods::stringWithUTF8String_::h13adb976a6c4f01a [inlined] objc::message::send_message::hb975109fc5b4cbdd(obj=0x00007fff800dc8d0, sel=Sel @ 0x00007ffeefbff128, args=(*const i8) @ 0x00007ffeefbff148) at mod.rs:178:5</span>
<span class="go"> frame #5: 0x0000000100000fcd a-leaky-bucket`a_leaky_bucket::nsstring::NSString_NSStringExtensionMethods::stringWithUTF8String_::h13adb976a6c4f01a(nullTerminatedCString="a leaky string") at nsstring.rs:7034</span>
<span class="go"> frame #6: 0x0000000100001c74 a-leaky-bucket`a_leaky_bucket::leak::h9a1e6714fee9d36c at main.rs:28:17</span>
<span class="go"> frame #7: 0x0000000100001b20 a-leaky-bucket`a_leaky_bucket::main::h00757089d677f6a2 at main.rs:18:13</span>
<span class="go"> frame #8: 0x000000010000156e a-leaky-bucket`core::ops::function::FnOnce::call_once::haeff37c118186a6c((null)=(a-leaky-bucket`a_leaky_bucket::main::h00757089d677f6a2 at main.rs:17), (null)=<unavailable>) at function.rs:227:5</span>
<span class="go"> frame #9: 0x0000000100001351 a-leaky-bucket`std::sys_common::backtrace::__rust_begin_short_backtrace::hfd92fb564fd3b2e4(f=(a-leaky-bucket`a_leaky_bucket::main::h00757089d677f6a2 at main.rs:17)) at backtrace.rs:125:18</span>
<span class="go"> frame #10: 0x0000000100001924 a-leaky-bucket`std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::h72100a97160f9dee at rt.rs:66:18</span>
<span class="go"> frame #11: 0x0000000100020fe4 a-leaky-bucket`std::rt::lang_start_internal::h0c37a46739a0311d [inlined] core::ops::function::impls::_$LT$impl$u20$core..ops..function..FnOnce$LT$A$GT$$u20$for$u20$$RF$F$GT$::call_once::h3b22ce68aa2879c2 at function.rs:259:13 [opt]</span>
<span class="go"> frame #12: 0x0000000100020fdd a-leaky-bucket`std::rt::lang_start_internal::h0c37a46739a0311d [inlined] std::panicking::try::do_call::h51a4853c94b1bdea at panicking.rs:379 [opt]</span>
<span class="go"> frame #13: 0x0000000100020fdd a-leaky-bucket`std::rt::lang_start_internal::h0c37a46739a0311d [inlined] std::panicking::try::ha45bc5eab1f103eb at panicking.rs:343 [opt]</span>
<span class="go"> frame #14: 0x0000000100020fdd a-leaky-bucket`std::rt::lang_start_internal::h0c37a46739a0311d [inlined] std::panic::catch_unwind::h191bc002afc126a7 at panic.rs:431 [opt]</span>
<span class="go"> frame #15: 0x0000000100020fdd a-leaky-bucket`std::rt::lang_start_internal::h0c37a46739a0311d at rt.rs:51 [opt]</span>
<span class="go"> frame #16: 0x00000001000018fe a-leaky-bucket`std::rt::lang_start::h62e28d4b373c8b88(main=(a-leaky-bucket`a_leaky_bucket::main::h00757089d677f6a2 at main.rs:17), argc=1, argv=0x00007ffeefbff418) at rt.rs:65:5</span>
<span class="go"> frame #17: 0x0000000100001cd6 a-leaky-bucket`main + 22</span>
<span class="go"> frame #18: 0x00007fff20386f5d libdyld.dylib`start + 1</span>
<span class="go"> frame #19: 0x00007fff20386f5d libdyld.dylib`start + 1</span>
</code></pre></div>
</figure>
<p>The <code>AutoreleasePoolPage</code> seems to be a C++ class which is
the implementation of the autorelease functionality in Objective-C.
Since the code does not allocate an Autorelease Pool, this code should
not be running.</p>
<p>Fortunately Apple <a href="https://opensource.apple.com/source/objc4/objc4-818.2/">publishes
the source</a> of the Objective-C runtime. Searching for
<code>AutoreleasePoolPage</code> in the source code reveals the
implementation in <code>NSObject.mm</code>. Within this implementation
there are two interesting snippets.</p>
<figure class="code-block cpp"><div class="highlight"><pre><span></span><code><span class="k">static</span><span class="w"> </span><span class="kr">inline</span><span class="w"> </span><span class="n">id</span><span class="w"> </span><span class="o">*</span><span class="nf">autoreleaseFast</span><span class="p">(</span><span class="n">id</span><span class="w"> </span><span class="n">obj</span><span class="p">)</span><span class="w"></span>
<span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">AutoreleasePoolPage</span><span class="w"> </span><span class="o">*</span><span class="n">page</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">hotPage</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">page</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="o">!</span><span class="n">page</span><span class="o">-></span><span class="n">full</span><span class="p">())</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">page</span><span class="o">-></span><span class="n">add</span><span class="p">(</span><span class="n">obj</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">page</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">autoreleaseFullPage</span><span class="p">(</span><span class="n">obj</span><span class="p">,</span><span class="w"> </span><span class="n">page</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">autoreleaseNoPage</span><span class="p">(</span><span class="n">obj</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="c1">// Also further down</span>
<span class="k">static</span><span class="w"> </span><span class="n">__attribute__</span><span class="p">((</span><span class="n">noinline</span><span class="p">))</span><span class="w"></span>
<span class="n">id</span><span class="w"> </span><span class="o">*</span><span class="n">autoreleaseNoPage</span><span class="p">(</span><span class="n">id</span><span class="w"> </span><span class="n">obj</span><span class="p">)</span><span class="w"></span>
<span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// "No page" could mean no pool has been pushed</span>
<span class="w"> </span><span class="c1">// or an empty placeholder pool has been pushed and has no contents yet</span>
<span class="w"> </span><span class="n">ASSERT</span><span class="p">(</span><span class="o">!</span><span class="n">hotPage</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="kt">bool</span><span class="w"> </span><span class="n">pushExtraBoundary</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">false</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">haveEmptyPoolPlaceholder</span><span class="p">())</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// We are pushing a second pool over the empty placeholder pool</span>
<span class="w"> </span><span class="c1">// or pushing the first object into the empty placeholder pool.</span>
<span class="w"> </span><span class="c1">// Before doing that, push a pool boundary on behalf of the pool</span>
<span class="w"> </span><span class="c1">// that is currently represented by the empty placeholder.</span>
<span class="w"> </span><span class="n">pushExtraBoundary</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">true</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">obj</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="n">POOL_BOUNDARY</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="n">DebugMissingPools</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// We are pushing an object with no pool in place,</span>
<span class="w"> </span><span class="c1">// and no-pool debugging was requested by environment.</span>
<span class="w"> </span><span class="n">_objc_inform</span><span class="p">(</span><span class="s">"MISSING POOLS: (%p) Object %p of class %s "</span><span class="w"></span>
<span class="w"> </span><span class="s">"autoreleased with no pool in place - "</span><span class="w"></span>
<span class="w"> </span><span class="s">"just leaking - break on "</span><span class="w"></span>
<span class="w"> </span><span class="s">"objc_autoreleaseNoPool() to debug"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">objc_thread_self</span><span class="p">(),</span><span class="w"> </span><span class="p">(</span><span class="kt">void</span><span class="o">*</span><span class="p">)</span><span class="n">obj</span><span class="p">,</span><span class="w"> </span><span class="n">object_getClassName</span><span class="p">(</span><span class="n">obj</span><span class="p">));</span><span class="w"></span>
<span class="w"> </span><span class="n">objc_autoreleaseNoPool</span><span class="p">(</span><span class="n">obj</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">nil</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">obj</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">POOL_BOUNDARY</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="o">!</span><span class="n">DebugPoolAllocation</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// We are pushing a pool with no pool in place,</span>
<span class="w"> </span><span class="c1">// and alloc-per-pool debugging was not requested.</span>
<span class="w"> </span><span class="c1">// Install and return the empty pool placeholder.</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">setEmptyPoolPlaceholder</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="c1">// We are pushing an object or a non-placeholder'd pool.</span>
<span class="w"> </span><span class="c1">// Install the first page.</span>
<span class="w"> </span><span class="n">AutoreleasePoolPage</span><span class="w"> </span><span class="o">*</span><span class="n">page</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">AutoreleasePoolPage</span><span class="p">(</span><span class="n">nil</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">setHotPage</span><span class="p">(</span><span class="n">page</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Push a boundary on behalf of the previously-placeholder'd pool.</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">pushExtraBoundary</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">page</span><span class="o">-></span><span class="n">add</span><span class="p">(</span><span class="n">POOL_BOUNDARY</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Push the requested object or pool.</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">page</span><span class="o">-></span><span class="n">add</span><span class="p">(</span><span class="n">obj</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<figcaption><p>A snippet from <code>NSObject.mm</code> in the Autorelease Pool
implementation</p></figcaption></figure>
<p>It seems that when <code>autorelease</code> is called for the first
time and no “page” has been allocated before, a page will be allocated
so the object can be added to the page. Since other parts of the code
make references to thread local storage, and Apple’s documentation
implies Autorelease Pools are thread specific, I assume the pool will be
drained when the <code>main</code> thread dies, which explains why
<code>leaks</code> does not report any leaks.</p>
<h2 id="triggering-the-leak-as-expected">Triggering the leak as
expected<a class="pilcrow" href="#triggering-the-leak-as-expected"></a></h2>
<p>In the snippet above, there is a hint indicating we can prevent
allocating a pool on demand.</p>
<figure class="code-block cpp"><div class="highlight"><pre><span></span><code><span class="k">else</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">obj</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="n">POOL_BOUNDARY</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="n">DebugMissingPools</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// We are pushing an object with no pool in place,</span>
<span class="w"> </span><span class="c1">// and no-pool debugging was requested by environment.</span>
<span class="w"> </span><span class="n">_objc_inform</span><span class="p">(</span><span class="s">"MISSING POOLS: (%p) Object %p of class %s "</span><span class="w"></span>
<span class="w"> </span><span class="s">"autoreleased with no pool in place - "</span><span class="w"></span>
<span class="w"> </span><span class="s">"just leaking - break on "</span><span class="w"></span>
<span class="w"> </span><span class="s">"objc_autoreleaseNoPool() to debug"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">objc_thread_self</span><span class="p">(),</span><span class="w"> </span><span class="p">(</span><span class="kt">void</span><span class="o">*</span><span class="p">)</span><span class="n">obj</span><span class="p">,</span><span class="w"> </span><span class="n">object_getClassName</span><span class="p">(</span><span class="n">obj</span><span class="p">));</span><span class="w"></span>
<span class="w"> </span><span class="n">objc_autoreleaseNoPool</span><span class="p">(</span><span class="n">obj</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">nil</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
</figure>
<p>The <code>DebugMissingPools</code> is an Objective-C runtime
configuration flag that is controlled by the
<code>OBJC_DEBUG_MISSING_POOLS</code> environment variable.</p>
<p>Running <code>a-leaky-bucket</code> with this variable set shows the
warning message printed out along side with our expected output.</p>
<figure class="code-block shell-session"><div class="highlight"><pre><span></span><code><span class="gp">$ </span><span class="nv">OBJC_DEBUG_MISSING_POOLS</span><span class="o">=</span>YES ./target/debug/a-leaky-bucket
<span class="go">objc[60670]: MISSING POOLS: (0x10c64de00) Object 0x7f974940a470 of class __NSCFString autoreleased with no pool in place - just leaking - break on objc_autoreleaseNoPool() to debug</span>
<span class="go">a leaky string</span>
</code></pre></div>
</figure>
<p>Running our program with <code>leaks</code> and this environment
variable set shows the leak as expected.</p>
<figure class="code-block shell-session"><div class="highlight"><pre><span></span><code><span class="go">OBJC_DEBUG_MISSING_POOLS=YES leaks --atExit -- ./target/debug/a-leaky-bucket</span>
<span class="go">objc[61039]: MISSING POOLS: (0x1037b7e00) Object 0x7fa0ed40a470 of class __NSCFString autoreleased with no pool in place - just leaking - break on objc_autoreleaseNoPool() to debug</span>
<span class="go">a leaky string</span>
<span class="go">Process: a-leaky-bucket [61039]</span>
<span class="go">Path: /Users/USER/*/a-leaky-bucket</span>
<span class="go">Load Address: 0x102f82000</span>
<span class="go">Identifier: a-leaky-bucket</span>
<span class="go">Version: ???</span>
<span class="go">Code Type: X86-64</span>
<span class="go">Platform: macOS</span>
<span class="go">Parent Process: leaks [61038]</span>
<span class="go">Date/Time: 2021-07-19 20:32:57.550 -0400</span>
<span class="go">Launch Time: 2021-07-19 20:32:56.905 -0400</span>
<span class="go">OS Version: macOS 11.4 (20F71)</span>
<span class="go">Report Version: 7</span>
<span class="go">Analysis Tool: /Applications/Xcode.app/Contents/Developer/usr/bin/leaks</span>
<span class="go">Analysis Tool Version: Xcode 12.5.1 (12E507)</span>
<span class="go">Physical footprint: 656K</span>
<span class="go">Physical footprint (peak): 660K</span>
<span class="go">----</span>
<span class="go">leaks Report Version: 4.0</span>
<span class="go">Process 61039: 502 nodes malloced for 31 KB</span>
<span class="go">Process 61039: 1 leak for 32 total leaked bytes.</span>
<span class="go"> 1 (32 bytes) ROOT LEAK: <CFString 0x7fa0ed40a470> [32] length: 14 "a leaky string"</span>
</code></pre></div>
</figure>
<h2 id="conclusion">Conclusion<a class="pilcrow" href="#conclusion"></a></h2>
<p>Despite Apple’s documentation, it is not required to allocate an
Autorelease Pool when calling Foundation APIs to prevent memory leaks.
However, failure to do so creates a thread local Autorelease Pool,
preventing release of the objects in the pool until the thread exits.
When calling Foundation APIs from Rust, careful care has to be taken to
ensure a pool is allocated and drained appropriately, otherwise memory
will not be freed until the thread exits.</p>
<p>Also the <code>OBJC_DEBUG_MISSING_POOLS</code> environment variable
will prevent automatically creating pools, allowing for detection of
missing Autorelease Pools via <code>leaks</code>.</p>
<section id="footnotes" class="footnotes footnotes-end-of-document" role="doc-endnotes">
<hr>
<ol>
<li id="fn1"><p>Apple uses the term “strong” and “weak” instead of owned
and borrowed.<a href="#fnref1" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
</ol>
</section>
tag:zameermanji.com,2021-07-13:/blog/2021/7/13/using-bindgen-with-system-frameworks-on-macos/Using <code>bindgen</code> with System Frameworks on macOS2021-07-13T00:00:00Z2021-07-13T00:00:00Z<p>Recently I was using <a href="https://developer.apple.com/documentation/iokit">IOKit</a> in Rust
and I found that existing FFI bindings on crates.io were not sufficient
for my use case. I had two options:</p>
<ul>
<li>Create FFI bindings by hand.</li>
<li>Use <a href="https://github.com/rust-lang/rust-bindgen">bindgen</a>
to generate FFI bindings from header files.</li>
</ul>
<p>Since IOKit has a large and complex API surface, I felt that creating
them by hand would be too cumbersome, so I ended up looking at
<code>bindgen</code>.</p>
<p>The premise of <code>bindgen</code> is pretty simple, given a C
header file, it will parse the header file and output Rust functions and
structures that allow for FFI. Underneath the hood, <code>bingen</code>
relies on <code>libclang</code> to parse the headers and then given the
parsed results, it produces Rust code for FFI.</p>
<p>However, on macOS <code>bindgen</code> fails out of the box to
generate bindings for headers that are located in the macOS SDK and this
is because Apple’s provided <code>libclang</code> does not have the same
logic as Apple’s provided <code>clang</code> on where to find Framework
headers.</p>
<p>A simple example can show this failure. A <code>wrapper.h</code> file
which references a macOS SDK provided header like
<code>CoreFoundation.h</code> will fail with <code>bindgen</code> even
though <code>clang</code> can process it successfully.</p>
<figure class="code-block shell-session"><div class="highlight"><pre><span></span><code><span class="gp">$ </span><span class="nb">echo</span> <span class="s1">'#include <CoreFoundation/CoreFoundation.h>'</span> > wrapper.h
<span class="gp">$ </span><span class="c1"># Ask Apple's clang to expand the header</span>
<span class="gp">$ </span>clang -E wrapper.h > /dev/null
<span class="gp">$ </span><span class="nb">echo</span> <span class="nv">$?</span>
<span class="go">0</span>
</code></pre></div>
<figcaption><p>Successfully processing the <code>wrapper.h</code> file with
<code>clang</code></p></figcaption></figure>
<p>However passing the same <code>wrapper.h</code> to
<code>bindgen</code> results in an error.</p>
<figure class="code-block shell-session"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>bindgen wrapper.h
<span class="go">wrapper.h:1:10: fatal error: 'CoreFoundation/CoreFoundation.h' file not found</span>
<span class="go">wrapper.h:1:10: note: did not find header 'CoreFoundation.h' in framework 'CoreFoundation' (loaded from '/System/Library/Frameworks')</span>
<span class="go">wrapper.h:1:10: fatal error: 'CoreFoundation/CoreFoundation.h' file not found, err: true</span>
<span class="go">thread 'main' panicked at 'Unable to generate bindings: ()', /Users/zmanji/.cargo/registry/src/github.com-1ecc6299db9ec823/bindgen-0.58.1/src/main.rs:54:36</span>
<span class="go">note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace</span>
</code></pre></div>
<figcaption><p>Failing to process the <code>wrapper.h</code> with
<code>bindgen</code></p></figcaption></figure>
<p>The <code>libclang</code> used by <code>bindgen</code> is searching
in <code>/System/Library/Frameworks</code> which does not contain
headers. Ever since macOS 10.14 Apple stopped placing headers in
<code>/usr/include</code> and <code>/System/Library/Frameworks</code>.
Instead the headers are located in the SDK directories. Apple’s
<code>clang</code> knows to check in these directories.</p>
<figure class="code-block shell-session"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>clang -H wrapper.h
<span class="go">. /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/CoreFoundation.framework/Headers/CoreFoundation.h</span>
<span class="go">...</span>
</code></pre></div>
</figure>
<p>The fix is to direct <code>bindgen</code> and <code>libclang</code>
to the SDK directory which has the headers. It might be temping to
hard-code the <code>Xcode.app</code> directory however headers can
<em>also</em> be installed by the <a href="https://developer.apple.com/library/archive/technotes/tn2339/_index.html#//apple_ref/doc/uid/DTS40014588-CH1-WHAT_IS_THE_COMMAND_LINE_TOOLS_PACKAGE_">Command
Line Tools for Xcode</a> which would be located in the
<code>/Library/Developer/CommandLineTools</code> directory.</p>
<p>The fix would be <em>not</em> to hard code these directories but
instead query the SDK path with the <code>xcrun</code> tool. From the
man page</p>
<blockquote>
<p>xcrun provides a means to locate or invoke developer tools from the
command-line, without requiring users to modify Makefiles or otherwise
take inconvenient measures to support multiple Xcode tool chains.</p>
</blockquote>
<p>Using <code>xcrun</code> the SDK path can be obtained not only for
the macOS SDK but for the iOS SDK and other Apple platforms.</p>
<figure class="code-block shell-session"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>xcrun --sdk macosx --show-sdk-path
<span class="go">/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk</span>
<span class="gp">$ </span>xcrun --sdk iphoneos --show-sdk-path
<span class="go">/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk</span>
</code></pre></div>
</figure>
<p>The path can be passed to <code>bindgen</code> to ensure it can find
the headers.</p>
<figure class="code-block shell-session"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>bindgen input.h -- -isysroot<span class="k">$(</span>xcrun --sdk macosx --show-sdk-path<span class="k">)</span>
<span class="go">/* automatically generated by rust-bindgen 0.58.1 */</span>
<span class="go">...</span>
</code></pre></div>
</figure>
<p>To conclude, <code>bindgen</code> on macOS does not work out of the
box for System Frameworks. The only way for <code>bindgen</code> to
process the framework headers is to be told where the macOS SDK path is
and a way to do that is to use <code>xcrun</code>. Using
<code>xcrun</code> will work regardless if the OS has a full
<code>Xcode.app</code> installation or just the Command Line Tools for
XCode installed.</p>
tag:zameermanji.com,2021-07-07:/blog/2021/7/7/getting-standard-macos-directories/Getting standard macOS directories2021-07-07T00:00:00Z2021-07-07T00:00:00Z<p>macOS applications are supposed to store certain kinds of data in
specific folders. From Apple’s <a href="https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/FileSystemProgrammingGuide/FileSystemOverview/FileSystemOverview.html">File
System Programming Guide</a> the guide says:</p>
<blockquote>
<p>Put data cache files in the Library/Caches/ directory.</p>
</blockquote>
<blockquote>
<p>Put app-created support files in the Library/Application Support/
directory.</p>
</blockquote>
<p>It’s possible to hardcode these paths in an application but macOS
provides APIs to programmatically discover these directories. Using
these APIs can ensure an application is storing data in the “right
place” no matter what.</p>
<p>For non Objective-C/Swift applications there are two APIs available
from macOS. This post will show how to use them with Rust, but it’s
applicable for any application linking against the macOS SDK.</p>
<h2 id="nssearchpathfordirectoriesindomains"><code>NSSearchPathForDirectoriesInDomains</code><a class="pilcrow" href="#nssearchpathfordirectoriesindomains"></a></h2>
<p>The <code>NSSearchPathForDirectoriesInDomains</code> is located in
the <code>Foundation</code> framework. Typically <code>Foundation</code>
contains Objective-C APIs but this function is unique. It’s declared in
<code>Foundation.framework/Versions/C/Headers/NSPathUtilities.h</code>
The declaration is:</p>
<figure class="code-block obj-c"><div class="highlight"><pre><span></span><code><span class="n">FOUNDATION_EXPORT</span><span class="w"> </span><span class="bp">NSArray</span><span class="o"><</span><span class="bp">NSString</span><span class="w"> </span><span class="o">*></span><span class="w"> </span><span class="o">*</span><span class="n">NSSearchPathForDirectoriesInDomains</span><span class="p">(</span><span class="n">NSSearchPathDirectory</span><span class="w"> </span><span class="n">directory</span><span class="p">,</span><span class="w"> </span><span class="n">NSSearchPathDomainMask</span><span class="w"> </span><span class="n">domainMask</span><span class="p">,</span><span class="w"> </span><span class="kt">BOOL</span><span class="w"> </span><span class="n">expandTilde</span><span class="p">);</span><span class="w"></span>
</code></pre></div>
<figcaption><p>Copied from <code>NSPathUtilities.h</code></p></figcaption></figure>
<p>It’s a C function, but it returns Objective-C objects. Normally this
would be very difficult to use without the Objective-C runtime, but
macOS APIs have “toll free bridging” <a href="#fn1" class="footnote-ref" id="fnref1" role="doc-noteref"><sup>1</sup></a> between select Core
Foundation and Foundation objects including <code>NSArray</code> and
<code>NSString</code>. This means an application can simply cast the
return type to a <code>CFArray</code> of <code>CFString</code> instead.
So long as the caller deallocates the returned array, there should be no
issues from this approach.</p>
<p>It’s important to note that the <code>NSSearchPathDirectory</code>
and <code>NSSearchPathDomainMask</code> arguments are enums using
<code>NSUInteger</code>.</p>
<figure class="code-block obj-c"><div class="highlight"><pre><span></span><code><span class="k">typedef</span><span class="w"> </span><span class="n">NS_ENUM</span><span class="p">(</span><span class="n">NSUInteger</span><span class="p">,</span><span class="w"> </span><span class="n">NSSearchPathDirectory</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">NSApplicationDirectory</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="c1">// supported applications (Applications)</span>
<span class="w"> </span><span class="n">NSDemoApplicationDirectory</span><span class="p">,</span><span class="w"> </span><span class="c1">// unsupported applications, demonstration versions (Demos)</span>
<span class="w"> </span><span class="n">NSDeveloperApplicationDirectory</span><span class="p">,</span><span class="w"> </span><span class="c1">// developer applications (Developer/Applications). DEPRECATED - there is no one single Developer directory.</span>
<span class="w"> </span><span class="n">NSAdminApplicationDirectory</span><span class="p">,</span><span class="w"> </span><span class="c1">// system and network administration applications (Administration)</span>
<span class="w"> </span><span class="n">NSLibraryDirectory</span><span class="p">,</span><span class="w"> </span><span class="c1">// various documentation, support, and configuration files, resources (Library)</span>
<span class="w"> </span><span class="n">NSDeveloperDirectory</span><span class="p">,</span><span class="w"> </span><span class="c1">// developer resources (Developer) DEPRECATED - there is no one single Developer directory.</span>
<span class="w"> </span><span class="n">NSUserDirectory</span><span class="p">,</span><span class="w"> </span><span class="c1">// user home directories (Users)</span>
<span class="w"> </span><span class="n">NSDocumentationDirectory</span><span class="p">,</span><span class="w"> </span><span class="c1">// documentation (Documentation)</span>
<span class="w"> </span><span class="n">NSDocumentDirectory</span><span class="p">,</span><span class="w"> </span><span class="c1">// documents (Documents)</span>
<span class="w"> </span><span class="n">NSCoreServiceDirectory</span><span class="p">,</span><span class="w"> </span><span class="c1">// location of CoreServices directory (System/Library/CoreServices)</span>
<span class="w"> </span><span class="n">NSAutosavedInformationDirectory</span><span class="w"> </span><span class="n">API_AVAILABLE</span><span class="p">(</span><span class="n">macos</span><span class="p">(</span><span class="mf">10.6</span><span class="p">),</span><span class="w"> </span><span class="n">ios</span><span class="p">(</span><span class="mf">4.0</span><span class="p">),</span><span class="w"> </span><span class="n">watchos</span><span class="p">(</span><span class="mf">2.0</span><span class="p">),</span><span class="w"> </span><span class="n">tvos</span><span class="p">(</span><span class="mf">9.0</span><span class="p">))</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">11</span><span class="p">,</span><span class="w"> </span><span class="c1">// location of autosaved documents (Documents/Autosaved)</span>
<span class="w"> </span><span class="n">NSDesktopDirectory</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">12</span><span class="p">,</span><span class="w"> </span><span class="c1">// location of user's desktop</span>
<span class="w"> </span><span class="n">NSCachesDirectory</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">13</span><span class="p">,</span><span class="w"> </span><span class="c1">// location of discardable cache files (Library/Caches)</span>
<span class="w"> </span><span class="n">NSApplicationSupportDirectory</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">14</span><span class="p">,</span><span class="w"> </span><span class="c1">// location of application support files (plug-ins, etc) (Library/Application Support)</span>
<span class="w"> </span><span class="n">NSDownloadsDirectory</span><span class="w"> </span><span class="n">API_AVAILABLE</span><span class="p">(</span><span class="n">macos</span><span class="p">(</span><span class="mf">10.5</span><span class="p">),</span><span class="w"> </span><span class="n">ios</span><span class="p">(</span><span class="mf">2.0</span><span class="p">),</span><span class="w"> </span><span class="n">watchos</span><span class="p">(</span><span class="mf">2.0</span><span class="p">),</span><span class="w"> </span><span class="n">tvos</span><span class="p">(</span><span class="mf">9.0</span><span class="p">))</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">15</span><span class="p">,</span><span class="w"> </span><span class="c1">// location of the user's "Downloads" directory</span>
<span class="w"> </span><span class="n">NSInputMethodsDirectory</span><span class="w"> </span><span class="n">API_AVAILABLE</span><span class="p">(</span><span class="n">macos</span><span class="p">(</span><span class="mf">10.6</span><span class="p">),</span><span class="w"> </span><span class="n">ios</span><span class="p">(</span><span class="mf">4.0</span><span class="p">),</span><span class="w"> </span><span class="n">watchos</span><span class="p">(</span><span class="mf">2.0</span><span class="p">),</span><span class="w"> </span><span class="n">tvos</span><span class="p">(</span><span class="mf">9.0</span><span class="p">))</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">16</span><span class="p">,</span><span class="w"> </span><span class="c1">// input methods (Library/Input Methods)</span>
<span class="w"> </span><span class="n">NSMoviesDirectory</span><span class="w"> </span><span class="n">API_AVAILABLE</span><span class="p">(</span><span class="n">macos</span><span class="p">(</span><span class="mf">10.6</span><span class="p">),</span><span class="w"> </span><span class="n">ios</span><span class="p">(</span><span class="mf">4.0</span><span class="p">),</span><span class="w"> </span><span class="n">watchos</span><span class="p">(</span><span class="mf">2.0</span><span class="p">),</span><span class="w"> </span><span class="n">tvos</span><span class="p">(</span><span class="mf">9.0</span><span class="p">))</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">17</span><span class="p">,</span><span class="w"> </span><span class="c1">// location of user's Movies directory (~/Movies)</span>
<span class="w"> </span><span class="n">NSMusicDirectory</span><span class="w"> </span><span class="n">API_AVAILABLE</span><span class="p">(</span><span class="n">macos</span><span class="p">(</span><span class="mf">10.6</span><span class="p">),</span><span class="w"> </span><span class="n">ios</span><span class="p">(</span><span class="mf">4.0</span><span class="p">),</span><span class="w"> </span><span class="n">watchos</span><span class="p">(</span><span class="mf">2.0</span><span class="p">),</span><span class="w"> </span><span class="n">tvos</span><span class="p">(</span><span class="mf">9.0</span><span class="p">))</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">18</span><span class="p">,</span><span class="w"> </span><span class="c1">// location of user's Music directory (~/Music)</span>
<span class="w"> </span><span class="n">NSPicturesDirectory</span><span class="w"> </span><span class="n">API_AVAILABLE</span><span class="p">(</span><span class="n">macos</span><span class="p">(</span><span class="mf">10.6</span><span class="p">),</span><span class="w"> </span><span class="n">ios</span><span class="p">(</span><span class="mf">4.0</span><span class="p">),</span><span class="w"> </span><span class="n">watchos</span><span class="p">(</span><span class="mf">2.0</span><span class="p">),</span><span class="w"> </span><span class="n">tvos</span><span class="p">(</span><span class="mf">9.0</span><span class="p">))</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">19</span><span class="p">,</span><span class="w"> </span><span class="c1">// location of user's Pictures directory (~/Pictures)</span>
<span class="w"> </span><span class="n">NSPrinterDescriptionDirectory</span><span class="w"> </span><span class="n">API_AVAILABLE</span><span class="p">(</span><span class="n">macos</span><span class="p">(</span><span class="mf">10.6</span><span class="p">),</span><span class="w"> </span><span class="n">ios</span><span class="p">(</span><span class="mf">4.0</span><span class="p">),</span><span class="w"> </span><span class="n">watchos</span><span class="p">(</span><span class="mf">2.0</span><span class="p">),</span><span class="w"> </span><span class="n">tvos</span><span class="p">(</span><span class="mf">9.0</span><span class="p">))</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">20</span><span class="p">,</span><span class="w"> </span><span class="c1">// location of system's PPDs directory (Library/Printers/PPDs)</span>
<span class="w"> </span><span class="n">NSSharedPublicDirectory</span><span class="w"> </span><span class="n">API_AVAILABLE</span><span class="p">(</span><span class="n">macos</span><span class="p">(</span><span class="mf">10.6</span><span class="p">),</span><span class="w"> </span><span class="n">ios</span><span class="p">(</span><span class="mf">4.0</span><span class="p">),</span><span class="w"> </span><span class="n">watchos</span><span class="p">(</span><span class="mf">2.0</span><span class="p">),</span><span class="w"> </span><span class="n">tvos</span><span class="p">(</span><span class="mf">9.0</span><span class="p">))</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">21</span><span class="p">,</span><span class="w"> </span><span class="c1">// location of user's Public sharing directory (~/Public)</span>
<span class="w"> </span><span class="n">NSPreferencePanesDirectory</span><span class="w"> </span><span class="n">API_AVAILABLE</span><span class="p">(</span><span class="n">macos</span><span class="p">(</span><span class="mf">10.6</span><span class="p">),</span><span class="w"> </span><span class="n">ios</span><span class="p">(</span><span class="mf">4.0</span><span class="p">),</span><span class="w"> </span><span class="n">watchos</span><span class="p">(</span><span class="mf">2.0</span><span class="p">),</span><span class="w"> </span><span class="n">tvos</span><span class="p">(</span><span class="mf">9.0</span><span class="p">))</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">22</span><span class="p">,</span><span class="w"> </span><span class="c1">// location of the PreferencePanes directory for use with System Preferences (Library/PreferencePanes)</span>
<span class="w"> </span><span class="n">NSApplicationScriptsDirectory</span><span class="w"> </span><span class="n">API_AVAILABLE</span><span class="p">(</span><span class="n">macos</span><span class="p">(</span><span class="mf">10.8</span><span class="p">))</span><span class="w"> </span><span class="bp">API_UNAVAILABLE</span><span class="p">(</span><span class="n">ios</span><span class="p">,</span><span class="w"> </span><span class="n">watchos</span><span class="p">,</span><span class="w"> </span><span class="n">tvos</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">23</span><span class="p">,</span><span class="w"> </span><span class="c1">// location of the user scripts folder for the calling application (~/Library/Application Scripts/code-signing-id)</span>
<span class="w"> </span><span class="n">NSItemReplacementDirectory</span><span class="w"> </span><span class="n">API_AVAILABLE</span><span class="p">(</span><span class="n">macos</span><span class="p">(</span><span class="mf">10.6</span><span class="p">),</span><span class="w"> </span><span class="n">ios</span><span class="p">(</span><span class="mf">4.0</span><span class="p">),</span><span class="w"> </span><span class="n">watchos</span><span class="p">(</span><span class="mf">2.0</span><span class="p">),</span><span class="w"> </span><span class="n">tvos</span><span class="p">(</span><span class="mf">9.0</span><span class="p">))</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">99</span><span class="p">,</span><span class="w"> </span><span class="c1">// For use with NSFileManager's URLForDirectory:inDomain:appropriateForURL:create:error:</span>
<span class="w"> </span><span class="n">NSAllApplicationsDirectory</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">100</span><span class="p">,</span><span class="w"> </span><span class="c1">// all directories where applications can occur</span>
<span class="w"> </span><span class="n">NSAllLibrariesDirectory</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">101</span><span class="p">,</span><span class="w"> </span><span class="c1">// all directories where resources can occur</span>
<span class="w"> </span><span class="n">NSTrashDirectory</span><span class="w"> </span><span class="n">API_AVAILABLE</span><span class="p">(</span><span class="n">macos</span><span class="p">(</span><span class="mf">10.8</span><span class="p">),</span><span class="w"> </span><span class="n">ios</span><span class="p">(</span><span class="mf">11.0</span><span class="p">))</span><span class="w"> </span><span class="bp">API_UNAVAILABLE</span><span class="p">(</span><span class="n">watchos</span><span class="p">,</span><span class="w"> </span><span class="n">tvos</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">102</span><span class="w"> </span><span class="c1">// location of Trash directory</span>
<span class="p">};</span><span class="w"></span>
<span class="k">typedef</span><span class="w"> </span><span class="n">NS_OPTIONS</span><span class="p">(</span><span class="n">NSUInteger</span><span class="p">,</span><span class="w"> </span><span class="n">NSSearchPathDomainMask</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">NSUserDomainMask</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="c1">// user's home directory --- place to install user's personal items (~)</span>
<span class="w"> </span><span class="n">NSLocalDomainMask</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">2</span><span class="p">,</span><span class="w"> </span><span class="c1">// local to the current machine --- place to install items available to everyone on this machine (/Library)</span>
<span class="w"> </span><span class="n">NSNetworkDomainMask</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">4</span><span class="p">,</span><span class="w"> </span><span class="c1">// publically available location in the local area network --- place to install items available on the network (/Network)</span>
<span class="w"> </span><span class="n">NSSystemDomainMask</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">8</span><span class="p">,</span><span class="w"> </span><span class="c1">// provided by Apple, unmodifiable (/System)</span>
<span class="w"> </span><span class="n">NSAllDomainsMask</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mh">0x0ffff</span><span class="w"> </span><span class="c1">// all domains: all of the above and future items</span>
<span class="p">};</span><span class="w"></span>
</code></pre></div>
<figcaption><p>Copied from <code>NSPathUtilities.h</code></p></figcaption></figure>
<p>Using the <a href="https://crates.io/crates/core-foundation"><code>core-foundation</code>
crate</a> a program to print out the Application Support Directory is
below.</p>
<figure class="code-block rust"><div class="highlight"><pre><span></span><code><span class="k">use</span><span class="w"> </span><span class="n">core_foundation</span>::<span class="n">base</span>::<span class="n">TCFType</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">core_foundation</span>::<span class="n">array</span>::<span class="p">{</span><span class="n">CFArrayRef</span><span class="p">,</span><span class="w"> </span><span class="n">CFArray</span><span class="p">};</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">core_foundation</span>::<span class="n">string</span>::<span class="p">{</span><span class="n">CFString</span><span class="p">};</span><span class="w"></span>
<span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">results</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">unsafe</span><span class="w"> </span><span class="p">{</span><span class="n">NSSearchPathForDirectoriesInDomains</span><span class="p">(</span><span class="mi">14</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w"> </span><span class="p">};</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">results</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">unsafe</span><span class="w"> </span><span class="p">{</span><span class="n">CFArray</span>::<span class="o"><</span><span class="n">CFString</span><span class="o">></span>::<span class="n">wrap_under_get_rule</span><span class="p">(</span><span class="n">results</span><span class="p">)};</span><span class="w"></span>
<span class="w"> </span><span class="n">results</span><span class="p">.</span><span class="n">iter</span><span class="p">().</span><span class="n">for_each</span><span class="p">(</span><span class="o">|</span><span class="n">x</span><span class="o">|</span><span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"{}"</span><span class="p">,</span><span class="w"> </span><span class="o">*</span><span class="n">x</span><span class="p">));</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="cp">#[link(name = </span><span class="s">"Foundation"</span><span class="cp">, kind = </span><span class="s">"framework"</span><span class="cp">)]</span><span class="w"></span>
<span class="k">extern</span><span class="w"> </span><span class="s">"C"</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">NSSearchPathForDirectoriesInDomains</span><span class="p">(</span><span class="n">directory</span>: <span class="kt">u64</span><span class="p">,</span><span class="w"> </span><span class="n">domain_mask</span>: <span class="kt">u64</span><span class="p">,</span><span class="w"> </span><span class="n">expand_tilde</span>: <span class="kt">i8</span><span class="p">)</span><span class="w"> </span>-> <span class="nc">CFArrayRef</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<figcaption><p>Using the <code>NSSearchPathForDirectoriesInDomains</code> API</p></figcaption></figure>
<p>The above program prints out the following on my machine:</p>
<figure class="code-block text"><div class="highlight"><pre><span></span><code>/Users/zmanji/Library/Application Support
</code></pre></div>
</figure>
<h2 id="sysdir"><code>sysdir</code><a class="pilcrow" href="#sysdir"></a></h2>
<p>The second option is the <code>sysdir</code> API, it appears to be
less documented and less used, but it’s located at
<code>/usr/include/sysdir.h</code> and comes with a manpage
<code>sysdir(3)</code>.</p>
<p>The declaration is:</p>
<figure class="code-block c"><div class="highlight"><pre><span></span><code><span class="k">typedef</span><span class="w"> </span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">sysdir_search_path_enumeration_state</span><span class="p">;</span><span class="w"></span>
<span class="n">__BEGIN_DECLS</span><span class="w"></span>
<span class="k">extern</span><span class="w"> </span><span class="n">sysdir_search_path_enumeration_state</span><span class="w"></span>
<span class="n">sysdir_start_search_path_enumeration</span><span class="p">(</span><span class="n">sysdir_search_path_directory_t</span><span class="w"> </span><span class="n">dir</span><span class="p">,</span><span class="w"> </span><span class="n">sysdir_search_path_domain_mask_t</span><span class="w"> </span><span class="n">domainMask</span><span class="p">)</span><span class="w"> </span><span class="n">__API_AVAILABLE</span><span class="p">(</span><span class="n">macosx</span><span class="p">(</span><span class="mf">10.12</span><span class="p">),</span><span class="w"> </span><span class="n">ios</span><span class="p">(</span><span class="mf">10.0</span><span class="p">),</span><span class="w"> </span><span class="n">watchos</span><span class="p">(</span><span class="mf">3.0</span><span class="p">),</span><span class="w"> </span><span class="n">tvos</span><span class="p">(</span><span class="mf">10.0</span><span class="p">));</span><span class="w"></span>
<span class="k">extern</span><span class="w"> </span><span class="n">sysdir_search_path_enumeration_state</span><span class="w"></span>
<span class="nf">sysdir_get_next_search_path_enumeration</span><span class="p">(</span><span class="n">sysdir_search_path_enumeration_state</span><span class="w"> </span><span class="n">state</span><span class="p">,</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="n">path</span><span class="p">)</span><span class="w"> </span><span class="n">__API_AVAILABLE</span><span class="p">(</span><span class="n">macosx</span><span class="p">(</span><span class="mf">10.12</span><span class="p">),</span><span class="w"> </span><span class="n">ios</span><span class="p">(</span><span class="mf">10.0</span><span class="p">),</span><span class="w"> </span><span class="n">watchos</span><span class="p">(</span><span class="mf">3.0</span><span class="p">),</span><span class="w"> </span><span class="n">tvos</span><span class="p">(</span><span class="mf">10.0</span><span class="p">));</span><span class="w"></span>
<span class="n">__END_DECLS</span><span class="w"></span>
</code></pre></div>
<figcaption><p>Copied from <code>sysdir.h</code></p></figcaption></figure>
<p><code>sysdir_search_path_directory_t</code> and
<code>sysdir_search_path_domain_mask_t</code> are enums identical to the
<code>NSSearchPathForDirectoriesInDomains</code> API:</p>
<figure class="code-block c"><div class="highlight"><pre><span></span><code><span class="c1">// Available OSX 10.12, iOS 10.0, WatchOS 3.0 and TVOS 10.0. Not all enum identifiers return a useful path on all platforms.</span>
<span class="n">OS_ENUM</span><span class="p">(</span><span class="n">sysdir_search_path_directory</span><span class="p">,</span><span class="w"> </span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">int</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">SYSDIR_DIRECTORY_APPLICATION</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="c1">// supported applications (Applications)</span>
<span class="w"> </span><span class="n">SYSDIR_DIRECTORY_DEMO_APPLICATION</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">2</span><span class="p">,</span><span class="w"> </span><span class="c1">// unsupported applications, demonstration versions (Applications/Demos)</span>
<span class="w"> </span><span class="n">SYSDIR_DIRECTORY_DEVELOPER_APPLICATION</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">3</span><span class="p">,</span><span class="w"> </span><span class="c1">// developer applications (Developer/Applications) Soft deprecated as of __MAC_10_5 - there is no one single Developer directory</span>
<span class="w"> </span><span class="n">SYSDIR_DIRECTORY_ADMIN_APPLICATION</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">4</span><span class="p">,</span><span class="w"> </span><span class="c1">// system and network administration applications (Applications/Utilities)</span>
<span class="w"> </span><span class="n">SYSDIR_DIRECTORY_LIBRARY</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">5</span><span class="p">,</span><span class="w"> </span><span class="c1">// various user-visible documentation, support, and configuration files, resources (Library)</span>
<span class="w"> </span><span class="n">SYSDIR_DIRECTORY_DEVELOPER</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">6</span><span class="p">,</span><span class="w"> </span><span class="c1">// developer resources (Developer) Soft deprecated as of __MAC_10_5 - there is no one single Developer directory</span>
<span class="w"> </span><span class="n">SYSDIR_DIRECTORY_USER</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">7</span><span class="p">,</span><span class="w"> </span><span class="c1">// user home directories (Users)</span>
<span class="w"> </span><span class="n">SYSDIR_DIRECTORY_DOCUMENTATION</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">8</span><span class="p">,</span><span class="w"> </span><span class="c1">// documentation (Library/Documentation)</span>
<span class="w"> </span><span class="n">SYSDIR_DIRECTORY_DOCUMENT</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">9</span><span class="p">,</span><span class="w"> </span><span class="c1">// documents (Documents)</span>
<span class="w"> </span><span class="n">SYSDIR_DIRECTORY_CORESERVICE</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">10</span><span class="p">,</span><span class="w"> </span><span class="c1">// location of core services (Library/CoreServices)</span>
<span class="w"> </span><span class="n">SYSDIR_DIRECTORY_AUTOSAVED_INFORMATION</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">11</span><span class="p">,</span><span class="w"> </span><span class="c1">// location of user's directory for use with autosaving (Library/Autosave Information)</span>
<span class="w"> </span><span class="n">SYSDIR_DIRECTORY_DESKTOP</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">12</span><span class="p">,</span><span class="w"> </span><span class="c1">// location of user's Desktop (Desktop)</span>
<span class="w"> </span><span class="n">SYSDIR_DIRECTORY_CACHES</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">13</span><span class="p">,</span><span class="w"> </span><span class="c1">// location of discardable cache files (Library/Caches)</span>
<span class="w"> </span><span class="n">SYSDIR_DIRECTORY_APPLICATION_SUPPORT</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">14</span><span class="p">,</span><span class="w"> </span><span class="c1">// location of application support files (plug-ins, etc) (Library/Application Support)</span>
<span class="w"> </span><span class="n">SYSDIR_DIRECTORY_DOWNLOADS</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">15</span><span class="p">,</span><span class="w"> </span><span class="c1">// location of user's Downloads directory (Downloads)</span>
<span class="w"> </span><span class="n">SYSDIR_DIRECTORY_INPUT_METHODS</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">16</span><span class="p">,</span><span class="w"> </span><span class="c1">// input methods (Library/Input Methods)</span>
<span class="w"> </span><span class="n">SYSDIR_DIRECTORY_MOVIES</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">17</span><span class="p">,</span><span class="w"> </span><span class="c1">// location of user's Movies directory (Movies)</span>
<span class="w"> </span><span class="n">SYSDIR_DIRECTORY_MUSIC</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">18</span><span class="p">,</span><span class="w"> </span><span class="c1">// location of user's Music directory (Music)</span>
<span class="w"> </span><span class="n">SYSDIR_DIRECTORY_PICTURES</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">19</span><span class="p">,</span><span class="w"> </span><span class="c1">// location of user's Pictures directory (Pictures)</span>
<span class="w"> </span><span class="n">SYSDIR_DIRECTORY_PRINTER_DESCRIPTION</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">20</span><span class="p">,</span><span class="w"> </span><span class="c1">// location of system's PPDs directory (Library/Printers/PPDs)</span>
<span class="w"> </span><span class="n">SYSDIR_DIRECTORY_SHARED_PUBLIC</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">21</span><span class="p">,</span><span class="w"> </span><span class="c1">// location of user's Public sharing directory (Public)</span>
<span class="w"> </span><span class="n">SYSDIR_DIRECTORY_PREFERENCE_PANES</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">22</span><span class="p">,</span><span class="w"> </span><span class="c1">// location of the PreferencePanes directory for use with System Preferences (Library/PreferencePanes)</span>
<span class="w"> </span><span class="n">SYSDIR_DIRECTORY_ALL_APPLICATIONS</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">100</span><span class="p">,</span><span class="w"> </span><span class="c1">// all directories where applications can occur (Applications, Applications/Utilities, Developer/Applications, ...)</span>
<span class="w"> </span><span class="n">SYSDIR_DIRECTORY_ALL_LIBRARIES</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">101</span><span class="p">,</span><span class="w"> </span><span class="c1">// all directories where resources can occur (Library, Developer)</span>
<span class="p">);</span><span class="w"></span>
<span class="c1">// Available OSX 10.12, iOS 10.0, WatchOS 3.0 and TVOS 10.0. Not all enum identifiers are useful on all platforms.</span>
<span class="n">OS_OPTIONS</span><span class="p">(</span><span class="n">sysdir_search_path_domain_mask</span><span class="p">,</span><span class="w"> </span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">int</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">SYSDIR_DOMAIN_MASK_USER</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="w"> </span><span class="mi">1UL</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">),</span><span class="w"> </span><span class="c1">// user's home directory --- place to install user's personal items (~)</span>
<span class="w"> </span><span class="n">SYSDIR_DOMAIN_MASK_LOCAL</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="w"> </span><span class="mi">1UL</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">),</span><span class="w"> </span><span class="c1">// local to the current machine --- place to install items available to everyone on this machine</span>
<span class="w"> </span><span class="n">SYSDIR_DOMAIN_MASK_NETWORK</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="w"> </span><span class="mi">1UL</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">),</span><span class="w"> </span><span class="c1">// publically available location in the local area network --- place to install items available on the network (/Network)</span>
<span class="w"> </span><span class="n">SYSDIR_DOMAIN_MASK_SYSTEM</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="w"> </span><span class="mi">1UL</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="p">),</span><span class="w"> </span><span class="c1">// provided by Apple</span>
<span class="w"> </span><span class="n">SYSDIR_DOMAIN_MASK_ALL</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mh">0x0ffff</span><span class="p">,</span><span class="w"> </span><span class="c1">// all domains: all of the above and more, future items</span>
<span class="p">);</span><span class="w"></span>
</code></pre></div>
<figcaption><p>Copied from <code>sysdir.h</code></p></figcaption></figure>
<p>Using this API via FFI is awkward compared to the
<code>NSSearchPathForDirectoriesInDomains</code> API because the
application has to manually iterate over the results.</p>
<figure class="code-block rust"><div class="highlight"><pre><span></span><code><span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">os</span>::<span class="n">raw</span>::<span class="n">c_char</span><span class="p">;</span><span class="w"></span>
<span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">result</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">unsafe</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">sysdir_start_search_path_enumeration</span><span class="p">(</span><span class="mi">14</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w"> </span><span class="p">};</span><span class="w"></span>
<span class="w"> </span><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">v</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="fm">vec!</span><span class="p">[</span><span class="mi">0</span><span class="p">;</span><span class="w"> </span><span class="mi">1024</span><span class="p">];</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">path</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">v</span><span class="p">.</span><span class="n">as_mut_ptr</span><span class="p">()</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="o">*</span><span class="k">mut</span><span class="w"> </span><span class="n">c_char</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">result</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">unsafe</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">sysdir_get_next_search_path_enumeration</span><span class="p">(</span><span class="n">result</span><span class="p">,</span><span class="w"> </span><span class="n">path</span><span class="p">)</span><span class="w"> </span><span class="p">};</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">result</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">break</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">s</span>: <span class="nb">String</span> <span class="o">=</span><span class="w"> </span><span class="nb">String</span>::<span class="n">from_utf8</span><span class="p">(</span><span class="n">v</span><span class="p">).</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"{}"</span><span class="p">,</span><span class="w"> </span><span class="n">s</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">extern</span><span class="w"> </span><span class="s">"C"</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">sysdir_start_search_path_enumeration</span><span class="p">(</span><span class="n">directory</span>: <span class="kt">u32</span><span class="p">,</span><span class="w"> </span><span class="n">domain_mask</span>: <span class="kt">u32</span><span class="p">)</span><span class="w"> </span>-> <span class="kt">u32</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">sysdir_get_next_search_path_enumeration</span><span class="p">(</span><span class="n">state</span>: <span class="kt">u32</span><span class="p">,</span><span class="w"> </span><span class="n">path</span>: <span class="o">*</span><span class="k">mut</span><span class="w"> </span><span class="n">c_char</span><span class="p">)</span><span class="w"> </span>-> <span class="kt">u32</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<figcaption><p>Using the <code>sysdir</code> API in Rust</p></figcaption></figure>
<p>This approach does not require any external crates or linking against
any frameworks. The <code>sysdir</code> API is located in
<code>libSystem</code> and is available to any application in macOS. The
above program prints out the following on my machine:</p>
<figure class="code-block text"><div class="highlight"><pre><span></span><code>~/Library/Application Support
</code></pre></div>
</figure>
<p>It’s important to note that this API will always return
<code>~</code> to represent the user’s home directory which might make
the return value hard to use with other APIs.</p>
<h2 id="conclusion">Conclusion<a class="pilcrow" href="#conclusion"></a></h2>
<p>There are two APIs available to get standard application directories
on macOS for non Objective-C/Swift applications. <code>sysdir</code> is
available in <code>libSystem</code> but does not expand <code>~</code>
in user specific paths. Alternatively,
<code>NSSearchPathForDirectoriesInDomains</code> is available in
<code>Foundation.framework</code> and returns a <code>CFArray</code> and
can expand <code>~</code> in user specific paths.</p>
<section id="footnotes" class="footnotes footnotes-end-of-document" role="doc-endnotes">
<hr>
<ol>
<li id="fn1"><p>See <a href="https://developer.apple.com/library/archive/documentation/General/Conceptual/CocoaEncyclopedia/Toll-FreeBridgin/Toll-FreeBridgin.html">this
Apple document</a> for all of the details.<a href="#fnref1" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
</ol>
</section>
tag:zameermanji.com,2021-07-04:/blog/2021/7/4/creating-a-virtualenv-from-a-pex/Creating a Virtualenv from a PEX2021-07-04T00:00:00Z2021-07-04T00:00:00Z<p>Since <a href="https://github.com/pantsbuild/pex/releases/tag/v2.1.25">PEX
v2.1.25</a> PEX has supported a <code>--venv</code> flag when creating a
PEX. The flag causes the created PEX to extract itself into a virtual
environment on startup. This has two benefits:</p>
<ul>
<li>If a PEX is built for many platforms, then the work to determine
which wheels to use is done only once when creating the virtualenv.</li>
<li>There is no overhead from unzipping the PEX on startup.</li>
</ul>
<p>The speed improvements are significant for tools that one might want
to run repeatedly such as <code>black</code> or <code>mypy</code>.</p>
<p>Here is a benchmark of running <code>black</code> from a traditional
PEX vs one built with the <code>--venv</code> flag.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>pex <span class="s1">'black==20.8b1'</span> --script black -o black.pex
<span class="gp">$ </span>pex <span class="s1">'black==20.8b1'</span> --script black --venv -o black-venv.pex
<span class="gp">$ </span>hyperfine -w2 <span class="s1">'./black.pex --help'</span> <span class="s1">'./black-venv.pex --help'</span>
<span class="go">Benchmark #1: ./black.pex --help</span>
<span class="go"> Time (mean ± σ): 1.046 s ± 0.009 s [User: 866.0 ms, System: 160.4 ms]</span>
<span class="go"> Range (min … max): 1.037 s … 1.068 s 10 runs</span>
<span class="go">Benchmark #2: ./black-venv.pex --help</span>
<span class="go"> Time (mean ± σ): 360.7 ms ± 4.0 ms [User: 254.0 ms, System: 88.0 ms]</span>
<span class="go"> Range (min … max): 356.5 ms … 367.4 ms 10 runs</span>
<span class="go">Summary</span>
<span class="go"> './black-venv.pex --help' ran</span>
<span class="go"> 2.90 ± 0.04 times faster than './black.pex --help'</span>
</code></pre></div>
</figure>
<p>The PEX built with the <code>--venv</code> flag is almost <em>3x</em>
faster to execute.</p>
<h2 id="where-is-the-virtualenv-">Where is the Virtualenv ?<a class="pilcrow" href="#where-is-the-virtualenv-"></a></h2>
<p>By default the PEX will create a virtualenv in the
<code>~/.pex/venvs</code> directory similar to where PEX keeps its
various caches. The full file path is determined by the hash of the PEX
as well as the hash of the environment used to select the Python
interpreter. Like everything in <code>~/.pex</code> this can be safely
deleted without issue and the next run of the PEX will recreate the
folder.</p>
<h2 id="alternative-location-for-the-virtualenv">Alternative Location
for the Virtualenv<a class="pilcrow" href="#alternative-location-for-the-virtualenv"></a></h2>
<p>The default location of <code>~/.pex/venvs</code> is hard to use
since the full path of the virtualenv is composed of opaque file hashes
and makes it hard to use the created virtualenv with other tools.
Alternatively the virtualenv can be created in an arbitrary location
using the ‘PEX Tools’ functionality.</p>
<p>When a PEX is created with the <code>--venv</code> flag, the created
PEX has some extra PEX related code that can run when the
<code>PEX_TOOLS=1</code> environment variable is set.</p>
<p>The PEX can be created in the <code>./venv</code> directory by
invoking the PEX like so:</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span><span class="nv">PEX_TOOLS</span><span class="o">=</span><span class="m">1</span> ./black-venv.pex venv ./venv
</code></pre></div>
</figure>
<h2 id="whats-in-the-virtualenv">What’s in the Virtualenv?<a class="pilcrow" href="#whats-in-the-virtualenv"></a></h2>
<p>The created virtualenv has everything a regular virtualenv would have
plus a few PEX specific extras:</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>ls
<span class="go">PEX-INFO __main__.py bin include lib pex pyvenv.cfg</span>
</code></pre></div>
</figure>
<p>Inside the <code>bin</code> directory we have the activation scripts
as well as all of the console entry points from all of the wheels as
well as our wheel console scripts.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>ls bin
<span class="go">Activate.ps1 activate.csh black blackd python3</span>
<span class="go">activate activate.fish black-primer python python3.9</span>
</code></pre></div>
</figure>
<p>It’s important to note that <code>setuptools</code> and
<code>pip</code> are not installed into this virtual environment which
makes them immutable.</p>
<p>We can directly execute a console script in the <code>./bin</code>
directory for an even larger performance boost than executing the PEX
built with <code>--venv</code>.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>hyperfine -w2 <span class="s1">'./venv/bin/black --help'</span> <span class="s1">'./black-venv.pex --help'</span> <span class="s1">'./black.pex --help'</span>
<span class="go">Benchmark #1: ./venv/bin/black --help</span>
<span class="go"> Time (mean ± σ): 187.8 ms ± 4.6 ms [User: 151.0 ms, System: 32.7 ms]</span>
<span class="go"> Range (min … max): 183.0 ms … 197.8 ms 15 runs</span>
<span class="go">Benchmark #2: ./black-venv.pex --help</span>
<span class="go"> Time (mean ± σ): 359.5 ms ± 5.6 ms [User: 252.3 ms, System: 88.3 ms]</span>
<span class="go"> Range (min … max): 351.9 ms … 367.4 ms 10 runs</span>
<span class="go">Benchmark #3: ./black.pex --help</span>
<span class="go"> Time (mean ± σ): 1.050 s ± 0.011 s [User: 864.8 ms, System: 164.8 ms]</span>
<span class="go"> Range (min … max): 1.033 s … 1.064 s 10 runs</span>
<span class="go">Summary</span>
<span class="go"> './venv/bin/black --help' ran</span>
<span class="go"> 1.91 ± 0.06 times faster than './black-venv.pex --help'</span>
<span class="go"> 5.59 ± 0.15 times faster than './black.pex --help'</span>
</code></pre></div>
</figure>
<p>There is also a <code>PEX-INFO</code> file in the root which is
copied from the PEX that created the virtualenv.</p>
<figure class="code-block json"><div class="highlight"><pre><span></span><code><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nt">"always_write_cache"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"build_properties"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nt">"class"</span><span class="p">:</span><span class="w"> </span><span class="s2">"CPython"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"pex_version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2.1.42"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"platform"</span><span class="p">:</span><span class="w"> </span><span class="s2">"macosx_11_0_x86_64"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"version"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"></span>
<span class="w"> </span><span class="mi">3</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="mi">9</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="mi">4</span><span class="w"></span>
<span class="w"> </span><span class="p">]</span><span class="w"></span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="nt">"code_hash"</span><span class="p">:</span><span class="w"> </span><span class="s2">"da39a3ee5e6b4b0d3255bfef95601890afd80709"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"distributions"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nt">"appdirs-1.4.4-py2.py3-none-any.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"b219560d4b28b017da90ce2c5484a2b6c62dfc52"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"black-20.8b1-py3-none-any.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"efc41b1c13052ffc6f2e9c5144aedefeba0c4d41"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"click-8.0.1-py3-none-any.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"60cbe72bab38df1f22792bc2d8b6b3a072d02dbf"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"mypy_extensions-0.4.3-py2.py3-none-any.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"41374e5ada663fa33b6fc37950c079faba1a8f32"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"pathspec-0.8.1-py2.py3-none-any.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"eb9e7328e69e63265e32837e52d572b28103f41a"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"regex-2021.7.1-cp39-cp39-macosx_10_9_x86_64.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"356520db0bd32672b69f021d1e5f55f2a55b747e</span>
<span class="s2">"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"toml-0.10.2-py2.py3-none-any.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"941913d720ad4816a848c11a218c9110f1978120"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"typed_ast-1.4.3-cp39-cp39-macosx_10_9_x86_64.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ff61103c4d1a9e3da01759d8a400ef02103f1d0</span>
<span class="s2">8"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"typing_extensions-3.10.0.0-py3-none-any.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2d21ea9841868bec222ae7bffdf3212ac3ad9c3e"</span><span class="w"></span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="nt">"emit_warnings"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"entry_point"</span><span class="p">:</span><span class="w"> </span><span class="s2">"black:patched_main"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"ignore_errors"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"inherit_path"</span><span class="p">:</span><span class="w"> </span><span class="s2">"false"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"interpreter_constraints"</span><span class="p">:</span><span class="w"> </span><span class="p">[],</span><span class="w"></span>
<span class="w"> </span><span class="nt">"pex_hash"</span><span class="p">:</span><span class="w"> </span><span class="s2">"3bdf45072c53c0a39ca9398703cfccb01fb1ca4a"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"pex_path"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"pex_root"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/Users/zmanji/.pex"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"requirements"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"></span>
<span class="w"> </span><span class="s2">"black==20.8b1"</span><span class="w"></span>
<span class="w"> </span><span class="p">],</span><span class="w"></span>
<span class="w"> </span><span class="nt">"strip_pex_env"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"unzip"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"venv"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"venv_bin_path"</span><span class="p">:</span><span class="w"> </span><span class="s2">"false"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"venv_copies"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"zip_safe"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
</figure>
<p>Finally there is also a <code>__main__.py</code> at the root which
makes the entire directory executable by Python. Pointing Python to this
directory is equivalent to pointing Python to the PEX file itself except
you get the speed benefits of the virtualenv.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>python ./venv - --help
<span class="go">Re-execing from /Users/zmanji/.pyenv/versions/3.9.4/bin/python</span>
<span class="go">Usage: python -m venv [OPTIONS] [SRC]...</span>
<span class="go"> The uncompromising code formatter.</span>
<span class="go">...</span>
</code></pre></div>
</figure>
<h2 id="conclusion">Conclusion<a class="pilcrow" href="#conclusion"></a></h2>
<p>Creating a PEX with the <code>--venv</code> flag allows for a
virtualenv to be implicitly or explicitly created from the PEX. Creating
a virtualenv has less execution overhead than a plain PEX and the
resulting virtualenv is immutable.</p>
tag:zameermanji.com,2021-06-25:/blog/2021/6/25/packaging-multi-platform-python-applications/Packaging Multi Platform Python Applications2021-06-25T00:00:00Z2021-06-25T00:00:00Z<p>When creating a Python program that many dependencies, it can become
frustrating or complicated to deploy this program to multiple
environments. Target environments might have different operating
systems, CPU architectures or Python interpreters. Traditional
deployment solutions such as <a href="https://github.com/spotify/dh-virtualenv">dh-virtualenv</a> would
require creating multiple packages in order to deploy to a variety of
environments. Creating a single executable that can be used across these
environments can be very helpful.</p>
<p>To get a single executable that we can deploy to a large variety of
environments, we can use <a href="https://github.com/pantsbuild/pex">PEX</a> to solve this
problem.</p>
<p>This blog post will explain how we can use PEX’s
<code>--platform</code> flag to create a single executable that can run
across different Python interpreter versions and different operating
systems.</p>
<h2 id="why-pex-">Why PEX ?<a class="pilcrow" href="#why-pex-"></a></h2>
<p>There are a multitude of ways to package and distribute an
application written in Python. However PEX is unique compared to other
alternatives because it supports the following:</p>
<ul>
<li>Targeting Multiple Operating Systems: macOS and Linux.</li>
<li>Targeting Multiple Python Interpreters: PyPy, CPython, etc.</li>
<li>Producing a <em>single</em> artifact that works across all of the
above.</li>
<li>Based on Python wheels, allowing for any kind of native
dependency.</li>
</ul>
<p>In general, if your application and all of the dependencies have
wheels for all of your target environments, then it’s trivial to produce
a PEX that allows for your application to work there.</p>
<h2 id="why-is-this-useful">Why is this useful?<a class="pilcrow" href="#why-is-this-useful"></a></h2>
<p>Imagine the case where you have a tool that uses <a href="https://pypi.org/project/pygit2/">pygit2</a> to interact with a
git repository. It might be useful to use this tool as apart of your
build your development process. However you might need to target the
following environments with your tool:</p>
<ul>
<li>macOS for local development.</li>
<li>Debian stable bare metal machines.</li>
<li>Ubuntu LTS based Docker images.</li>
<li>Alpine based Docker image.</li>
</ul>
<p>Within the above, you would need to support many CPython versions,
since the CPython that ships with Debian Buster is <code>3.7.3</code>,
the version that ships with Ubuntu <code>20.04</code> is
<code>3.8.5</code> and Alpine Docker images are currently based on
<code>3.9.5</code>.</p>
<p>Single Linux distribution tools like <a href="https://github.com/spotify/dh-virtualenv">dh-virtualenv</a> are
not helpful in this scenario since we would still need to figure out how
package our application for macOS and Alpine. A single PEX file that we
could deploy to all of the above environments would greatly simplify
distribution.</p>
<h2 id="an-example">An Example<a class="pilcrow" href="#an-example"></a></h2>
<p>We can create a Python application call <code>git-inspect</code> that
uses <code>libgit2</code> to print out the commit id of HEAD in the
repository.</p>
<figure class="code-block cfg"><div class="highlight"><pre><span></span><code><span class="k">[metadata]</span><span class="w"></span>
<span class="na">name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">git-inspect</span><span class="w"></span>
<span class="k">[options]</span><span class="w"></span>
<span class="na">py_modules</span><span class="w"> </span><span class="o">=</span><span class="w"></span>
<span class="w"> </span><span class="na">git_inspect</span><span class="w"></span>
<span class="na">install_requires</span><span class="w"> </span><span class="o">=</span><span class="w"></span>
<span class="w"> </span><span class="na">pygit2</span><span class="w"> </span><span class="o">=</span><span class="s">= 1.6.1</span><span class="w"></span>
<span class="k">[options.entry_points]</span><span class="w"></span>
<span class="na">console_scripts</span><span class="w"> </span><span class="o">=</span><span class="w"></span>
<span class="w"> </span><span class="na">git-inspect</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">git_inspect:main</span><span class="w"></span>
</code></pre></div>
<figcaption><p><code>setup.cfg</code></p></figcaption></figure>
<figure class="code-block python"><div class="highlight"><pre><span></span><code><span class="ch">#!/usr/bin/env python3</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">import</span> <span class="nn">pygit2</span>
<span class="k">def</span> <span class="nf">main</span><span class="p">():</span>
<span class="n">repo</span> <span class="o">=</span> <span class="n">pygit2</span><span class="o">.</span><span class="n">Repository</span><span class="p">(</span><span class="n">path</span><span class="o">=</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
<span class="nb">print</span><span class="p">(</span><span class="n">repo</span><span class="o">.</span><span class="n">head</span><span class="o">.</span><span class="n">peel</span><span class="p">())</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">"__main__"</span><span class="p">:</span>
<span class="n">main</span><span class="p">()</span>
</code></pre></div>
<figcaption><p><code>git_inspect.py</code></p></figcaption></figure>
<p>We can also use the <code>pip-compile</code> command from the <a href="https://github.com/jazzband/pip-tools">pip-tools</a> project to
give us the full dependency list for <code>pygit2 == 1.6.1</code>.</p>
<figure class="code-block text"><div class="highlight"><pre><span></span><code>cffi==1.14.5
# via pygit2
pycparser==2.20
# via cffi
pygit2==1.6.1
# via git-inspect (setup.py)
</code></pre></div>
<figcaption><p><code>requirements.txt</code></p></figcaption></figure>
<p>Using PEX we can now build a PEX for our local environment:</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>pex --script git-inspect <span class="se">\</span>
-r requirements.txt . <span class="se">\</span>
--python-shebang <span class="s1">'#!/usr/bin/env python3'</span> <span class="se">\</span>
-o dist/out.pex
</code></pre></div>
</figure>
<p>Running it produces a PEX file which runs fine on my MacBook:</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>./dist/out.pex ~/code/pex
<span class="go"><pygit2.Object{commit:075da872288e8aa5405a454069cf633d578e578d}></span>
</code></pre></div>
</figure>
<p>However running this on a Debian 10 host results in an error:</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>./out.pex
<span class="go">Traceback (most recent call last):</span>
<span class="go"> File "/home/zmanji/out.pex/.bootstrap/pex/pex.py", line 483, in execute</span>
<span class="go"> File "/home/zmanji/out.pex/.bootstrap/pex/pex.py", line 139, in activate</span>
<span class="go"> File "/home/zmanji/out.pex/.bootstrap/pex/pex.py", line 126, in _activate</span>
<span class="go"> File "/home/zmanji/out.pex/.bootstrap/pex/environment.py", line 428, in activate</span>
<span class="go"> File "/home/zmanji/out.pex/.bootstrap/pex/environment.py", line 784, in _activate</span>
<span class="go"> File "/home/zmanji/out.pex/.bootstrap/pex/environment.py", line 608, in resolve</span>
<span class="go"> File "/home/zmanji/out.pex/.bootstrap/pex/environment.py", line 629, in resolve_dists</span>
<span class="go"> File "/home/zmanji/out.pex/.bootstrap/pex/environment.py", line 573, in _root_requirements_iter</span>
<span class="go">pex.environment.ResolveError: A distribution for cffi could not be resolved in this environment.Found 1 distribution for cffi that do not apply:</span>
<span class="go">1.) The wheel tags for cffi 1.14.5 are cp39-cp39-macosx_10_9_x86_64 which do not match the supported tags of DistributionTarget(interpreter=PythonInterpreter('/usr/bin/python3.7', PythonIdentity('/usr/bin/python3', 'cp37', 'cp37m', 'manylinux_2_28_x86_64', (3, 7, 3)))):</span>
<span class="go">...</span>
</code></pre></div>
</figure>
<p>This is because the PEX doesn’t contain wheels for Linux just for
macOS. The wheels in the PEX can be seen in the <code>PEX-INFO</code>
file.</p>
<figure class="code-block json"><div class="highlight"><pre><span></span><code><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nt">"always_write_cache"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"build_properties"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nt">"class"</span><span class="p">:</span><span class="w"> </span><span class="s2">"CPython"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"pex_version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2.1.42"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"platform"</span><span class="p">:</span><span class="w"> </span><span class="s2">"macosx_11_0_x86_64"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"version"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"></span>
<span class="w"> </span><span class="mi">3</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="mi">9</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="mi">4</span><span class="w"></span>
<span class="w"> </span><span class="p">]</span><span class="w"></span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="nt">"code_hash"</span><span class="p">:</span><span class="w"> </span><span class="s2">"da39a3ee5e6b4b0d3255bfef95601890afd80709"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"distributions"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nt">"cffi-1.14.5-cp39-cp39-macosx_10_9_x86_64.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"89776038a89a4dfc0f24b0b274656ae011709cff"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"git_inspect-0.0.0-py3-none-any.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"bd703f052a1b9eff5d194d197f6d9a3120c4695d"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"pycparser-2.20-py2.py3-none-any.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"c1eaa0ece2823d3947bb56caed5b9fa8973c5e11"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"pygit2-1.6.1-cp39-cp39-macosx_10_9_x86_64.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"a88d821366e60b44bc524cdaa8a9f2718ca510d1"</span><span class="w"></span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="nt">"emit_warnings"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"entry_point"</span><span class="p">:</span><span class="w"> </span><span class="s2">"git_inspect:main"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"ignore_errors"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"inherit_path"</span><span class="p">:</span><span class="w"> </span><span class="s2">"false"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"interpreter_constraints"</span><span class="p">:</span><span class="w"> </span><span class="p">[],</span><span class="w"></span>
<span class="w"> </span><span class="nt">"pex_hash"</span><span class="p">:</span><span class="w"> </span><span class="s2">"7c5e0b6634b3959dd4267227a0dbd3cbefa3fd1a"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"pex_path"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"requirements"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"></span>
<span class="w"> </span><span class="s2">"cffi==1.14.5"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="s2">"git-inspect==0.0.0"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="s2">"pycparser==2.20"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="s2">"pygit2==1.6.1"</span><span class="w"></span>
<span class="w"> </span><span class="p">],</span><span class="w"></span>
<span class="w"> </span><span class="nt">"strip_pex_env"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"unzip"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"venv"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"venv_bin_path"</span><span class="p">:</span><span class="w"> </span><span class="s2">"False"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"venv_copies"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"zip_safe"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<figcaption><p><code>PEX-INFO</code></p></figcaption></figure>
<p>To get a PEX file that can run on macOS and Linux the
<code>--platform</code> argument to <code>pex</code> can be used to get
wheels required for every target platform. The <code>--platform</code>
argument accepts <a href="https://packaging.python.org/specifications/platform-compatibility-tags/">Python
platform tags</a>. By default PEX defaults to the platform of the
interpreter that is running which in my case was
<code>cp39-cp39-macosx_10_9_x86_64</code>. <code>pex</code> can be given
many <code>--platform</code> arguments, to target CPython 3.6, 3.7, 3.8,
3.9 for macOS and Linux we can do:</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>pex --script git-inspect <span class="se">\</span>
-r requirements.txt . <span class="se">\</span>
--python-shebang <span class="s1">'#!/usr/bin/env python3'</span> <span class="se">\</span>
--platform macosx_11_0_x86_64-cp-39-cp39 <span class="se">\</span>
--platform macosx_11_0_x86_64-cp-38-cp38 <span class="se">\</span>
--platform macosx_11_0_x86_64-cp-37-cp37m <span class="se">\</span>
--platform macosx_11_0_x86_64-cp-36-cp36m <span class="se">\</span>
--platform manylinux2014_x86_64-cp-39-cp39 <span class="se">\</span>
--platform manylinux2014_x86_64-cp-38-cp38 <span class="se">\</span>
--platform manylinux2014_x86_64-cp-37-cp37m <span class="se">\</span>
--platform manylinux2014_x86_64-cp-36-cp36m <span class="se">\</span>
-o dist/out.pex
</code></pre></div>
</figure>
<p>The PEX still works fine on my MacBook:</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>./git_inspect.py ~/code/pex
<span class="go"><pygit2.Object{commit:075da872288e8aa5405a454069cf633d578e578d}></span>
</code></pre></div>
</figure>
<p>Running this on a Debian 10 host also works:</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>./out.pex ./pex
<span class="go"><pygit2.Object{commit:075da872288e8aa5405a454069cf633d578e578d}></span>
</code></pre></div>
</figure>
<p>This works despite the fact that the Debian 10 host only has Python
3.7 and my MacBook has Python 3.9. The same PEX will work on any Mac or
Linux host so long as it has CPython 3.6-3.9 installed.</p>
<p>The <code>PEX-INFO</code> file shows that all of the wheels are
present for the target platforms:</p>
<figure class="code-block json"><div class="highlight"><pre><span></span><code><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nt">"always_write_cache"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"build_properties"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nt">"class"</span><span class="p">:</span><span class="w"> </span><span class="s2">"CPython"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"pex_version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2.1.42"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"platform"</span><span class="p">:</span><span class="w"> </span><span class="s2">"macosx_11_0_x86_64"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"version"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"></span>
<span class="w"> </span><span class="mi">3</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="mi">9</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="mi">4</span><span class="w"></span>
<span class="w"> </span><span class="p">]</span><span class="w"></span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="nt">"code_hash"</span><span class="p">:</span><span class="w"> </span><span class="s2">"da39a3ee5e6b4b0d3255bfef95601890afd80709"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"distributions"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nt">"cached_property-1.5.2-py2.py3-none-any.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"96efe61d013fd9050cb22369407ea028e9cc9b2c"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"cffi-1.14.5-cp36-cp36m-macosx_10_9_x86_64.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"55e5a5ff22a16068ae44212c0a88185b2e8c3f87"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"cffi-1.14.5-cp36-cp36m-manylinux1_x86_64.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2f061f0abdc3125831602a4a9cd43c1d47088143"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"cffi-1.14.5-cp37-cp37m-macosx_10_9_x86_64.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"7721d95e00506cb199417ad52f33bb1e915c08b6"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"cffi-1.14.5-cp37-cp37m-manylinux1_x86_64.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"7f2c48d8739abcd023275df9dc7e467b5515ccc6"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"cffi-1.14.5-cp38-cp38-macosx_10_9_x86_64.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"64f6888ba9f8be79681ce952633e559175211ddd"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"cffi-1.14.5-cp38-cp38-manylinux1_x86_64.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"7439b0775123c4cba0394c5a77aa2e3fce55c5a9"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"cffi-1.14.5-cp39-cp39-macosx_10_9_x86_64.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"89776038a89a4dfc0f24b0b274656ae011709cff"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"cffi-1.14.5-cp39-cp39-manylinux1_x86_64.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"80738ab38437ace103272942853e352d3f412274"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"git_inspect-0.0.0-py3-none-any.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"bd703f052a1b9eff5d194d197f6d9a3120c4695d"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"pycparser-2.20-py2.py3-none-any.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"c1eaa0ece2823d3947bb56caed5b9fa8973c5e11"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"pygit2-1.6.1-cp36-cp36m-macosx_10_9_x86_64.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"0c8ed85f42aeaf377d99fefae68dfc184a93e37c"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"pygit2-1.6.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"6fcb1d960dc5a008d802babf9a774d1e21655851"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"pygit2-1.6.1-cp37-cp37m-macosx_10_9_x86_64.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"47ec25fa3b4654a612e841f9e9454b97dd07258e"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"pygit2-1.6.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1a370f1d0fc4ae433597becd8c7a08f5f5a3c9e8"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"pygit2-1.6.1-cp38-cp38-macosx_10_9_x86_64.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1f7e923e35f87c36bbae1615e02d30b6713f1701"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"pygit2-1.6.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"d14961c955a7e104c2734a9e1e8606355092fe09"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"pygit2-1.6.1-cp39-cp39-macosx_10_9_x86_64.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"a88d821366e60b44bc524cdaa8a9f2718ca510d1"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"pygit2-1.6.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"0c4f1503bd8cf631712c47879cd25642b37c8d98"</span><span class="w"></span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="nt">"emit_warnings"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"entry_point"</span><span class="p">:</span><span class="w"> </span><span class="s2">"git_inspect:main"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"ignore_errors"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"inherit_path"</span><span class="p">:</span><span class="w"> </span><span class="s2">"false"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"interpreter_constraints"</span><span class="p">:</span><span class="w"> </span><span class="p">[],</span><span class="w"></span>
<span class="w"> </span><span class="nt">"pex_hash"</span><span class="p">:</span><span class="w"> </span><span class="s2">"eab984c7ae563881fdc03c3baae2f64b16883631"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"pex_path"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"requirements"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"></span>
<span class="w"> </span><span class="s2">"cffi==1.14.5"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="s2">"git-inspect==0.0.0"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="s2">"pycparser==2.20"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="s2">"pygit2==1.6.1"</span><span class="w"></span>
<span class="w"> </span><span class="p">],</span><span class="w"></span>
<span class="w"> </span><span class="nt">"strip_pex_env"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"unzip"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"venv"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"venv_bin_path"</span><span class="p">:</span><span class="w"> </span><span class="s2">"False"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"venv_copies"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"zip_safe"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
</figure>
<h2 id="a-note-on-cpython-abis-and-pex-size">A Note on CPython ABIs and
PEX size<a class="pilcrow" href="#a-note-on-cpython-abis-and-pex-size"></a></h2>
<p>One issue with the approach above is the sheer size of the output
PEX.</p>
<p>The PEX just for my MacBook is 1.5 MB whereas the PEX targeting all
of our environments is 24 MB. This is because some of our dependencies,
such as <code>cffi</code> and <code>pygit2</code> do not target the <a href="https://docs.python.org/3/c-api/stable.html">stable CPython
ABI</a> but instead the CPython version specific ABI. This results in
requiring a wheel for every target platform which results in a
significantly larger PEX. Dependencies that target the stable ABI could
have a single wheel that satisfies all platforms, resulting in smaller
PEX files.</p>
tag:zameermanji.com,2021-06-17:/blog/2021/6/17/embedding-a-rust-binary-in-another-rust-binary/Embedding a Rust binary in another Rust binary2021-06-17T00:00:00Z2021-06-17T00:00:00Z<p>For reasons<a href="#fn1" class="footnote-ref" id="fnref1" role="doc-noteref"><sup>1</sup></a> I need to embed a Rust binary in
another Rust binary, however out of the box Cargo doesn’t support this
easily. <a href="https://github.com/rust-lang/cargo/issues/9096">RFC
3028</a> will fix this but as of Rust <code>1.52</code> it is not
implemented. This blog post will go though an example, show where Cargo
does not work today and how to overcome this with a custom script.</p>
<h2 id="an-example">An Example<a class="pilcrow" href="#an-example"></a></h2>
<p>A basic example would be having two crates, <code>inner</code> and
<code>outer</code> in the same Cargo workspace. Both are binary crates,
but we want to embed the entire <code>inner</code> binary into the
<code>outer</code> binary. An example for this would be that when
<code>outer</code> starts up, it needs to extract the binary in a
special location and ask another application to launch
<code>inner</code>.</p>
<h2 id="a-naive-approach">A naive approach<a class="pilcrow" href="#a-naive-approach"></a></h2>
<p>A Cargo workspace could look like:</p>
<figure class="code-block text"><div class="highlight"><pre><span></span><code>.
├── Cargo.lock
├── Cargo.toml
├── inner
│ ├── Cargo.toml
│ └── src
│ └── main.rs
└── outer
├── Cargo.toml
└── src
└── main.rs
</code></pre></div>
<figcaption><p>the workspace</p></figcaption></figure>
<p>At the root we have our <code>Cargo.toml</code> which declares the
two crates:</p>
<figure class="code-block toml"><div class="highlight"><pre><span></span><code><span class="k">[workspace]</span><span class="w"></span>
<span class="n">members</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="w"></span>
<span class="w"> </span><span class="s">"inner"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="s">"outer"</span><span class="p">,</span><span class="w"></span>
<span class="p">]</span><span class="w"></span>
</code></pre></div>
<figcaption><p><code>./Cargo.toml</code></p></figcaption></figure>
<p>In our <code>inner</code> crates’ <code>Cargo.toml</code> we
have:</p>
<figure class="code-block toml"><div class="highlight"><pre><span></span><code><span class="k">[package]</span><span class="w"></span>
<span class="n">name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"inner"</span><span class="w"></span>
<span class="n">version</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"0.1.0"</span><span class="w"></span>
<span class="n">edition</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"2018"</span><span class="w"></span>
</code></pre></div>
<figcaption><p><code>./inner/Cargo.toml</code></p></figcaption></figure>
<p>And the entire code for the crate is:</p>
<figure class="code-block rust"><div class="highlight"><pre><span></span><code><span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"Hello, from inner!"</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<figcaption><p><code>inner/src/main.rs</code></p></figcaption></figure>
<p>Our <code>outer</code> crate could look like</p>
<figure class="code-block toml"><div class="highlight"><pre><span></span><code><span class="k">[package]</span><span class="w"></span>
<span class="n">name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"outer"</span><span class="w"></span>
<span class="n">version</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"0.1.0"</span><span class="w"></span>
<span class="n">edition</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"2018"</span><span class="w"></span>
<span class="k">[dependencies]</span><span class="w"></span>
<span class="n">inner</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span><span class="n">path</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"../inner"</span><span class="p">}</span><span class="w"></span>
</code></pre></div>
<figcaption><p><code>./outer/Cargo.toml</code></p></figcaption></figure>
<figure class="code-block rust"><div class="highlight"><pre><span></span><code><span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">inner_binary_bytes</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="fm">include_bytes!</span><span class="p">(</span><span class="s">"../../target/debug/inner"</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"The inner binary size: {}"</span><span class="p">,</span><span class="w"> </span><span class="n">inner_binary_bytes</span><span class="p">.</span><span class="n">len</span><span class="p">());</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<figcaption><p><code>./outer/src/main.rs</code></p></figcaption></figure>
<p>In the above approach, we declare a dependency from
<code>outer</code> to <code>inner</code> and hard code the location of
the debug binary.</p>
<p>From a clean workspace a build doesn’t work</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>cargo b
<span class="go">...</span>
<span class="go">error: couldn't read outer/src/../../target/debug/inner: No such file or directory (os error 2)</span>
<span class="go"> --> outer/src/main.rs:2:30</span>
<span class="go"> |</span>
<span class="go">2 | let inner_binary_bytes = include_bytes!("../../target/debug/inner");</span>
<span class="go"> | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^</span>
<span class="go"> |</span>
<span class="go"> = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)</span>
<span class="go">...</span>
</code></pre></div>
</figure>
<p>However a subsequent build does work:</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>cargo b
<span class="go">...</span>
<span class="go"> Finished dev [unoptimized + debuginfo] target(s) in 0.86s</span>
</code></pre></div>
</figure>
<p>And we can run <code>outer</code> with:</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>cargo run --bin outer
<span class="go"> Finished dev [unoptimized + debuginfo] target(s) in 0.00s</span>
<span class="go"> Running `target/debug/outer`</span>
<span class="go">The inner binary size: 442752</span>
</code></pre></div>
</figure>
<p>There are three problems here:</p>
<ul>
<li>Cargo does not respect the dependency from <code>outer</code> to
<code>inner</code> and proceeds to build them in parallel.</li>
<li>Subsequent builds ‘work’ but you might be getting a stale binary
embedded which can lead to a lot of confusion.</li>
<li>There is no easy way for <code>outer</code> to know were the desired
<code>inner</code> binary is. The above code assumes a debug build for
the current architecture but the path would be different if it were a
release build or a different architecture.</li>
</ul>
<h2 id="a-custom-build-script">A custom build script<a class="pilcrow" href="#a-custom-build-script"></a></h2>
<p>The only way I have found to do this is to have a custom
<code>build.rs</code> build script for <code>outer</code> which expects
the desired binary in a specific place, and another script to coordinate
the builds of <code>inner</code> and <code>outer</code>.</p>
<p>One benefit of this approach is that we can get rid of the dependency
between <code>inner</code> and <code>outer</code> which will prevent us
from accidentally referencing code in another binary.</p>
<p>The <code>build.rs</code> for <code>outer</code> looks like:</p>
<figure class="code-block rust"><div class="highlight"><pre><span></span><code><span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">result</span>::<span class="nb">Result</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">error</span>::<span class="n">Error</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">env</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">path</span>::<span class="n">Path</span><span class="p">;</span><span class="w"></span>
<span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span>-> <span class="nb">Result</span><span class="o"><</span><span class="p">(),</span><span class="w"> </span><span class="nb">Box</span><span class="o"><</span><span class="k">dyn</span><span class="w"> </span><span class="n">Error</span><span class="o">>></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">out_dir</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">env</span>::<span class="n">var_os</span><span class="p">(</span><span class="s">"OUT_DIR"</span><span class="p">).</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">out_dir</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Path</span>::<span class="n">new</span><span class="p">(</span><span class="o">&</span><span class="n">out_dir</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">std</span>::<span class="n">fs</span>::<span class="n">create_dir_all</span><span class="p">(</span><span class="o">&</span><span class="n">out_dir</span><span class="p">)</span><span class="o">?</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">inner_bin</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">get_env_var</span><span class="p">(</span><span class="s">"INNER_BIN"</span><span class="p">)</span><span class="o">?</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">inner_bin</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Path</span>::<span class="n">new</span><span class="p">(</span><span class="o">&</span><span class="n">inner_bin</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">target_bin</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">out_dir</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="s">"inner"</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">std</span>::<span class="n">fs</span>::<span class="n">copy</span><span class="p">(</span><span class="n">inner_bin</span><span class="p">,</span><span class="w"> </span><span class="n">target_bin</span><span class="p">)</span><span class="o">?</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="nb">Ok</span><span class="p">(())</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">fn</span> <span class="nf">get_env_var</span><span class="p">(</span><span class="n">var</span>: <span class="kp">&</span><span class="kt">str</span><span class="p">)</span><span class="w"> </span>-> <span class="nb">Result</span><span class="o"><</span><span class="nb">String</span><span class="p">,</span><span class="w"> </span><span class="n">env</span>::<span class="n">VarError</span><span class="o">></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"{}"</span><span class="p">,</span><span class="w"> </span><span class="s">"cargo:rerun-if-env-changed="</span><span class="p">.</span><span class="n">to_owned</span><span class="p">()</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">var</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">env</span>::<span class="n">var</span><span class="p">(</span><span class="n">var</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<figcaption><p><code>./outer/build.rs</code></p></figcaption></figure>
<p>We have to use environment variables because that’s the only way a
build script can get an argument.</p>
<p>Once this is done we can change <code>./outer/src/main.rs</code> to
read from <code>OUT_DIR</code></p>
<figure class="code-block rust"><div class="highlight"><pre><span></span><code><span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">inner_binary_bytes</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="fm">include_bytes!</span><span class="p">(</span><span class="fm">concat!</span><span class="p">(</span><span class="fm">env!</span><span class="p">(</span><span class="s">"OUT_DIR"</span><span class="p">),</span><span class="w"> </span><span class="s">"/inner"</span><span class="p">));</span><span class="w"></span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"The inner binary size: {}"</span><span class="p">,</span><span class="w"> </span><span class="n">inner_binary_bytes</span><span class="p">.</span><span class="n">len</span><span class="p">());</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<figcaption><p><code>outer/src/main/rs</code></p></figcaption></figure>
<p>After this is done we just need a script to drive cargo and set the
appropriate environment variables:</p>
<figure class="code-block bash"><div class="highlight"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="nb">set</span> -euo pipefail
<span class="nb">export</span> <span class="nv">INNER_BIN</span><span class="o">=</span><span class="k">$(</span>cargo build <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span> -p inner --message-format<span class="o">=</span>json <span class="p">|</span> jq -r <span class="s1">'select(.reason == "compiler-artifact") | .filenames[0]'</span><span class="k">)</span>
cargo build <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span> -p outer
</code></pre></div>
<figcaption><p><code>build.sh</code></p></figcaption></figure>
<p>With this script, we explicitly build <code>inner</code> first, grab
the output location from cargo and persist it in the environment
variable that the <code>outer/build.rs</code> consumes.</p>
<p>Invoking the script allows us to properly build <code>inner</code>
and then embed it in <code>outer</code>. The design of the shell script
allows us to also pass arguments to <code>cargo build</code> like
<code>--release</code> or a target architecture. This script also works
from a clean build and will always ensure we are embedding the latest
build of <code>inner</code>.</p>
<section id="footnotes" class="footnotes footnotes-end-of-document" role="doc-endnotes">
<hr>
<ol>
<li id="fn1"><p>Consider the case of trying to bundle a complex
application into a single binary. This application could extract it’s
own sub applications on startup to simplify distribution.<a href="#fnref1" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
</ol>
</section>
tag:zameermanji.com,2021-06-14:/blog/2021/6/14/building-wheels-with-pip-is-getting-slower/Building wheels with <code>pip</code> is getting slower2021-06-14T00:00:00Z2021-06-14T00:00:00Z<p>I was fiddling around with my <a href="https://github.com/zmanji/git-squash">git-squash</a> utility and
after making it Python 3 friendly, I observed building a wheel got
slower after adding a <code>pyproject.toml</code> to the project. With
<code>pip</code> <code>21.1.2</code> and Python <code>3.9.4</code> on
macOS I measured the time it took to build a wheel with and without a
<code>pyproject.toml</code> in the root. All of the results below are
from the <a href="https://github.com/sharkdp/hyperfine">hyperfine</a>
benchmarking tool.</p>
<p>Without a <code>pyproject.toml</code> file it takes about 3
seconds:</p>
<figure class="code-block text"><div class="highlight"><pre><span></span><code>Benchmark #1: pip wheel -w dist/ --only-binary ":all:" --no-deps .
Time (mean ± σ): 3.082 s ± 0.178 s [User: 1.228 s, System: 1.764 s]
Range (min … max): 2.909 s … 3.383 s 10 runs
</code></pre></div>
</figure>
<p>Adding the contents of a <code>pyproject.toml</code> file with the
defaults prescribed in <a href="https://www.python.org/dev/peps/pep-0518/">PEP 518</a> results in
a <code>pyproject.toml</code> file that looks like:</p>
<figure class="code-block toml"><div class="highlight"><pre><span></span><code><span class="k">[build-system]</span><span class="w"></span>
<span class="n">requires</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="s">"setuptools"</span><span class="p">,</span><span class="w"> </span><span class="s">"wheel"</span><span class="p">]</span><span class="w"></span>
</code></pre></div>
</figure>
<p>With this file in the root of the package it takes about 6 seconds to
build the package:</p>
<figure class="code-block text"><div class="highlight"><pre><span></span><code>Benchmark #1: pip wheel -w dist/ --only-binary ":all:" --no-deps .
Time (mean ± σ): 5.987 s ± 0.037 s [User: 3.807 s, System: 2.035 s]
Range (min … max): 5.939 s … 6.067 s 10 runs
</code></pre></div>
</figure>
<p>Considering <code>git-squash</code> is a package with exactly one
file, building a wheel should be very straight forward and fast. Almost
doubling the time is very surprising.</p>
<p>The results were so alarming that I ended up filing a GitHub issue
against <a href="https://github.com/pypa/pip/issues/10060">pip</a>. It
turns out <a href="https://github.com/pypa/pip/issues/7294">I am not</a>
the first person to observe that <a href="https://www.python.org/dev/peps/pep-0518/">PEP 518</a> style
builds with <code>pip</code> are a lot slower than the world before.</p>
<p>It’s really concerning because using a <code>pyproject.toml</code>
file is the future of Python packaging and allows the ecosystem to move
beyond <code>setuptools</code> as the default ‘backend’ for building
wheels and <code>pip</code> as the default ‘frontend’ for invoking
setuptools. Further this behaviour will become default for all builds <a href="https://github.com/pypa/pip/issues/9175">eventually</a>, so even
removing a <code>pyproject.toml</code> file from the root will have no
effect in the future.</p>
<p>The rest of this post will dive into some background, <em>why</em>
pip is slower and what can be done to speed up pip without giving up the
benefits of <code>pyproject.toml</code>.</p>
<h2 id="background">Background<a class="pilcrow" href="#background"></a></h2>
<p><code>pip</code> and <code>setuptools</code> are the de-facto Python
tools for packaging. <code>pip</code> implements dependency resolution
and can fetch wheels from PyPI or build wheels from source locally. The
latter step has a hard dependency on <code>setuptools</code>.</p>
<p>While <code>pip</code> and <code>setuptools</code> can be improved,
alternatives to both have emerged. Dependency resolution can be done
with <code>pipenv</code> and <code>poetry</code>, while wheels can be
built with <code>flit</code> or <code>enscons</code>. As the ecosystem
is growing, <a href="https://www.python.org/dev/peps/pep-0517/">PEP
517</a> and <a href="https://www.python.org/dev/peps/pep-0518/">PEP
518</a> dictate how these tools can interoperate.</p>
<p>Primarily these PEPs allow a package to have a
<code>pyproject.toml</code> file which lists what packages are needed to
build them. For example a <code>pyproject.toml</code> file with:</p>
<figure class="code-block toml"><div class="highlight"><pre><span></span><code><span class="k">[build-system]</span><span class="w"></span>
<span class="n">requires</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="s">"flit > 3"</span><span class="p">]</span><span class="w"></span>
</code></pre></div>
</figure>
<p>indicates that the package needs at least version 3 of
<code>flit</code> to build a wheel.</p>
<p>This simple declarative method allows for packages to be built by
their preferred mechanism of building wheels by any number of tools.
This is a good thing and will allow for competitor to <code>pip</code>
and <code>setuptools</code> to flourish, hopefully improving the entire
ecosystem.</p>
<h2 id="what-is-pip-doing">What is <code>pip</code> Doing?<a class="pilcrow" href="#what-is-pip-doing"></a></h2>
<p>With the above in mind we can take a look at what <code>pip</code> is
doing. Prior to the invention of <code>pyproject.toml</code>,
<code>pip</code> assumed that every package had to be built with
<code>setuptools</code> and packages had no way of indicating which
version of <code>setuptools</code>. This means that the user had to
install <em>some</em> version of <code>pip</code> and
<code>setuptools</code> adjacent to each other, and then ask
<code>pip</code> to build a wheel. <code>pip</code> would use whatever
version of <code>setuptools</code> was available and that was the best
we had.</p>
<p>This lead to a few problems which <code>pyproject.toml</code> fixed.
For example, if a package was using features introduced in newer
<code>setuptools</code> it could not indicate a minimum version of
<code>setuptools</code> required, meaning it was possible for the build
to fail. Another issue would be that it was possible to publish a
package that no one else could build. An example is the
<code>setup.py</code> invoked by <code>setuptools</code> could import
other packages, and unless the author documented what those packages
were and how to get them, no one else could build the package.</p>
<p><code>pip</code>’s implementation of PEP 518 solves the above
problems by creating an <em>empty</em> environment for <em>every</em>
package to turn into a wheel. Then it resolves the dependencies listed
in the <code>build-system</code> section of <code>pyproject.toml</code>
and installs them into the just created environment. Since the defaults
prescribed in PEP 518 are <code>setuptools</code> and <code>wheel</code>
without any version constraints, <code>pip</code> has to resolve those
dependencies from scratch by reaching out to PyPI and downloading the
latest versions of each. Even having <code>flit > 3</code> would
require <code>pip</code> to reach out to PyPI to see what the latest
<code>flit</code> release is.</p>
<p>We can see that having a <code>pyproject.toml</code> in the root of
the package and specifying <code>--no-index</code> to
<code>pip wheel</code> results in failure of building the wheel:</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>pip wheel -w dist/ --only-binary <span class="s2">":all:"</span> --no-deps --no-index .
<span class="go">...</span>
<span class="go"> ERROR: Could not find a version that satisfies the requirement setuptools (from versions: none</span>
<span class="go">)</span>
<span class="go"> ERROR: No matching distribution found for setuptools</span>
</code></pre></div>
</figure>
<p>This happens even though I already have <code>setuptools</code>
installed in my local environment. This also means that if I don’t have
internet access the build would fail as well.</p>
<h2 id="speeding-pip-up">Speeding <code>pip</code> up<a class="pilcrow" href="#speeding-pip-up"></a></h2>
<p>With PEP 518 solving a lot of issues, I think it’s best to speed
<code>pip</code> up instead. Since the additional time is spent creating
clean environments and re-resolving dependencies, we can reduce the work
required entirely by avoiding dependency resolution. This can be done by
downloading the wheels for the build requirements ahead of time and
pointing <code>pip</code> to this set of wheels while disabling
dependency resolution. Fortunately <code>pip</code> comes with all of
the pieces required to do this.</p>
<p>First we can download <code>setuptool</code> and <code>wheel</code>
wheels to a local directory.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>cat requirements-build.txt
<span class="go">wheel==0.36.2</span>
<span class="go">setuptools==57.0.0</span>
<span class="gp">$ </span>pip download -r requirements-build.txt -d dist/build-wheels --only-binary <span class="s1">':all:'</span>
<span class="go">...</span>
<span class="gp">$ </span>ls dist/build-wheels
<span class="go">setuptools-57.0.0-py3-none-any.whl wheel-0.36.2-py2.py3-none-any.whl</span>
</code></pre></div>
</figure>
<p>Then when we build the package with the <code>pyproject.toml</code>
that requires <code>setuptools</code> and <code>wheel</code> we instruct
<code>pip</code> to not reach out to PyPI and instead only look at the
<code>dist/build-wheel</code> directory for dependencies. This will
force <code>pip</code> to use the exact wheels we downloaded.</p>
<figure class="code-block console"><div class="highlight"><pre><span></span><code><span class="gp">$ </span>pip wheel -w dist/ --only-binary <span class="s2">":all:"</span> --no-deps --no-index --find-links ./dist/build-whe
<span class="go">els .</span>
<span class="go">...</span>
<span class="go">Successfully built git-squash</span>
</code></pre></div>
</figure>
<p>This brings down the build just above 5s. This is a little bit is
faster then before but not much.</p>
<figure class="code-block text"><div class="highlight"><pre><span></span><code>Benchmark #1: pip wheel -w dist/ --only-binary ":all:" --no-deps --no-index --find-links ./dist/
build-wheels .
Time (mean ± σ): 5.394 s ± 0.043 s [User: 3.345 s, System: 1.997 s]
Range (min … max): 5.335 s … 5.465 s 10 runs
</code></pre></div>
</figure>
<p>There is also an escape hatch to building an isolated environment
every time with the <code>--no-build-isolation</code> flag. Using it is
risky but combining this flag with cached wheels results in the best
results:</p>
<figure class="code-block text"><div class="highlight"><pre><span></span><code>Benchmark #1: pip wheel -w dist/ --only-binary ":all:" --no-deps --no-index --find-links ./dist/
build-wheels --no-build-isolation .
Time (mean ± σ): 2.868 s ± 0.071 s [User: 1.178 s, System: 1.653 s]
Range (min … max): 2.787 s … 3.034 s 10 runs
</code></pre></div>
</figure>
<p>Careful use of creating fresh virtual environments for builds by hand
and the above flag could speed up building wheels immensely.</p>