Jonathan Palardy's blog2024-02-11T23:38:46+00:00https://blog.jpalardy.com/Jonathan PalardyGleam Support in Vim with ALE2024-02-11T23:36:00+00:00https://blog.jpalardy.com/posts/gleam-support-in-vim-with-ale<h2 id="background">Background</h2>
<p>Over Christmas, I decided that it was a good time to try the <a href="https://exercism.org/tracks/gleam">Gleam track on
Exercism</a>. I prefer to start without doing
too much research. Once the Exercism executable creates its directories and
stub files, that’s when I start looking at tooling.</p>
<p>For me, it starts with Vim: no highlighting, but that wasn’t too surprising
– it’s early days for Gleam. It wasn’t hard to find <a href="https://github.com/gleam-lang/gleam.vim">Gleam syntax
highlighting</a>. It comes straight
from the <a href="https://github.com/gleam-lang">gleam-lang</a> account, so that looks pretty solid.</p>
<h2 id="can-we-do-better">Can we do better?</h2>
<p>Yes, with <a href="https://github.com/dense-analysis/ale">ALE</a>.</p>
<p>When I ran <code class="language-plaintext highlighter-rouge">:ALEInfo</code>, I could <em>NOT</em> find any Gleam support. I had to double-check
against the <a href="https://github.com/dense-analysis/ale/blob/master/supported-tools.md">list of supported tools</a>.</p>
<p>Usually, I’m not such an early adopter.<br />
Usually, ALE is ahead of my needs</p>
<p>Until now.</p>
<h2 id="gleam-tooling">Gleam Tooling</h2>
<p>Running <code class="language-plaintext highlighter-rouge">gleam --help</code> shows that it has both a <code class="language-plaintext highlighter-rouge">format</code> command and a <a href="https://en.wikipedia.org/wiki/Language_Server_Protocol">language server</a>!</p>
<p><a href="/assets/gleam-in-vim/gleam-help.png"><img src="/assets/gleam-in-vim/gleam-help.png" alt="gleam --help has LSP and format built-in" /></a></p>
<p>How hard could this be?</p>
<h2 id="ale-support-for-gleam">ALE Support for Gleam</h2>
<p>Good news:</p>
<ul>
<li>PR: <a href="https://github.com/dense-analysis/ale/pull/4696">LSP support</a></li>
<li>PR: <a href="https://github.com/dense-analysis/ale/pull/4710">format support</a></li>
</ul>
<p>and both have been merged ✅</p>
<p><a href="/assets/gleam-in-vim/gleam-support.png"><img src="/assets/gleam-in-vim/gleam-support.png" alt="gleam is now part of ALE" /></a></p>
<p>Make sure to update your ALE plugin to the latest version and enjoy 😄</p>
The Best Books I Read in 20232023-12-12T00:00:00+00:00https://blog.jpalardy.com/posts/best-books-i-read-2023<p>I have read many books in 2023; let’s forget most and talk about the good ones.</p>
<p>There is no particular order, but I broke down my recommendations by
categories: <a href="#technical">technical</a>, <a href="#non-fiction">non-fiction</a> and
<a href="#fiction">fiction</a>.</p>
<p>This is a yearly tradition! You can read my book reviews from previous years:<br />
<a href="https://blog.jpalardy.com/posts/best-books-i-read-2015/">2015</a> <a href="https://blog.jpalardy.com/posts/best-books-i-read-2016/">2016</a> <a href="https://blog.jpalardy.com/posts/best-books-i-read-2017/">2017</a> <a href="https://blog.jpalardy.com/posts/best-books-i-read-2018/">2018</a> <a href="https://blog.jpalardy.com/posts/best-books-i-read-2019/">2019</a> <a href="https://blog.jpalardy.com/posts/best-books-i-read-2020/">2020</a> <a href="https://blog.jpalardy.com/posts/best-books-i-read-2021/">2021</a> and <a href="https://blog.jpalardy.com/posts/best-books-i-read-2022/">2022</a>.</p>
<hr />
<h2 id="technical">Technical</h2>
<p>Surprisingly, I don’t have anything to recommend in this section for 2023.</p>
<!-- ------------------------------------------------- -->
<hr />
<h2 id="non-fiction">Non-Fiction</h2>
<h3 id="engineering-in-plain-sight">Engineering in Plain Sight</h3>
<p><a href="https://www.amazon.com/dp/171850232X/"><img class="book-cover" src="/assets/best-books-2023/171850232X.jpg" alt="Engineering in Plain Sight" /></a></p>
<p>This is a beautifully illustrated book, from Grady Hillhouse, the host
of the <a href="https://www.youtube.com/@PracticalEngineeringChannel">Practical Engineering</a> channel on YouTube.</p>
<p>The YouTube channel covers various topics on civil engineering and
infrastructure. The book offers a chance to dig deeper into specific areas of
interest: the electrical grid, communications, roads, bridges, sewers …</p>
<p>There are many illustrations, on every other page, and the explanations are
helpful without getting too long or technical.</p>
<p>I particularly enjoyed the parts about the power grid and
telecommunications because they are so easy to relate to. If you look outside,
or take a walk in your neighborhood, you’ll be able to see (and recognize!) the
infrastructure.</p>
<p>You may never look at a utility pole the same way. 😄</p>
<ul>
<li>book: <a href="https://www.amazon.com/dp/171850232X/">amazon</a></li>
<li>videos: <a href="https://www.youtube.com/@PracticalEngineeringChannel">youtube</a></li>
</ul>
<!-- ------------------------------------------------- -->
<h3 id="atomic-habits">Atomic Habits</h3>
<p><a href="https://www.amazon.com/dp/0735211299/"><img class="book-cover" src="/assets/best-books-2023/0735211299.jpg" alt="Atomic Habits" /></a></p>
<p>Many people recommended this book. I did not love it, but I have my reasons.</p>
<p><strong>Atomic Habits</strong> felt like a great summary of the other books that I had already read
on the same topic. I felt the same way when I finally read Dracula, since
I had watched so many vampire movies…</p>
<p>I can still recommend it: if you would prefer to read one (more definitive) book on the topic.</p>
<ul>
<li>book: <a href="https://www.amazon.com/dp/0735211299/">amazon</a></li>
<li>author: <a href="https://en.wikipedia.org/wiki/James_Clear">wikipedia</a></li>
</ul>
<!-- ------------------------------------------------- -->
<h3 id="the-ultimate-book-of-everyday-knots">The Ultimate Book of Everyday Knots</h3>
<p><a href="https://www.amazon.com/dp/1616085606/"><img class="book-cover" src="/assets/best-books-2023/1616085606.jpg" alt="The Ultimate Book of Everyday Knots" /></a></p>
<p>Knots were the rabbit hole that I fell into this year.</p>
<p>It all started relatively innocently with a YouTube recommendation
for a video: <a href="https://www.youtube.com/watch?v=5lLPAHK_k6I">The 10 BEST Knots in Life</a>.
It’s 16-minute long, but it doesn’t waste any time.</p>
<p>If you take nothing else away from this post, go watch that video and learn a knot.</p>
<p>After I watched this, I started to wonder:</p>
<ul>
<li>how many useful knots are there?</li>
<li>which ones should I learn?</li>
<li>what are some good resources?</li>
</ul>
<p>Based on a comment from Reddit – <em>“why would you learn knots from a book?!”</em>
– I ended up spending some time on <a href="https://www.animatedknots.com/">Animated Knots</a>, which is an excellent resource;
although it is more of a reference.</p>
<p>I also tried a few books, and <strong>The Ultimate Book of Everyday Knots</strong> was really good. The pictures are clear, with
different colored ropes and step-by-step instructions.</p>
<p>Knots are kind of fun. And useful if you know a few good ones.</p>
<ul>
<li>book: <a href="https://www.amazon.com/dp/1616085606/">amazon</a></li>
</ul>
<!-- ------------------------------------------------- -->
<h3 id="a-city-on-mars">A City on Mars</h3>
<p><a href="https://www.amazon.com/dp/1984881728/"><img class="book-cover" src="/assets/best-books-2023/1984881728.jpg" alt="A City on Mars" /></a></p>
<p>I knew <a href="https://en.wikipedia.org/wiki/Kelly_Weinersmith">the</a> <a href="https://en.wikipedia.org/wiki/Zach_Weinersmith">Weinersmiths</a> were
writing some kind of book about space and stuff: I was already sold. I have
read and can recommend most of their books.</p>
<p>I would consider myself a pro-space. I read and enjoyed <a href="https://blog.jpalardy.com/posts/best-books-i-read-2015/#the-case-for-mars">The Case for Mars</a> amongst others.
I wanted to get excited about space! (don’t we all?)</p>
<p>But the book starts, in a way, with an apology. They wanted to write
about space and colonies … but the more research they did, the more problems
they found.</p>
<p>So, yes, it’s a bit of a downer … or you could take it as a roadmap to all
the work that we have ahead of us. I also think that it’s an important book to <em>advance</em>
the discussion, if we’re serious about the topic. Unbridled enthusiasm can only
take you so far.</p>
<p><strong>A City on Mars</strong> is well-researched, funny, with the right amount of
on-topic discussion and side stories.</p>
<ul>
<li>book: <a href="https://www.amazon.com/dp/1984881728/">amazon</a></li>
<li>authors: <a href="https://en.wikipedia.org/wiki/Zach_Weinersmith">wikipedia</a> and <a href="https://en.wikipedia.org/wiki/Kelly_Weinersmith">wikipedia</a></li>
</ul>
<hr />
<h2 id="fiction">Fiction</h2>
<h3 id="wool">Wool</h3>
<p><a href="https://www.amazon.com/dp/B0BKR8VBYV/"><img class="book-cover" src="/assets/best-books-2023/B0BKR8VBYV.jpg" alt="Wool" /></a></p>
<p>How good is Wool?</p>
<p>It’s not bad.</p>
<p>I knew it was on Apple TV, and as is often the case, I decided to read the books
first before tackling their screen adaptation.</p>
<p>I thought the first book of the <a href="https://en.wikipedia.org/wiki/Silo_(series)">Silo series</a>
was interesting; a murder story with plenty of mysteries to uncover:</p>
<ul>
<li>why do they live underground?</li>
<li>what really happened?</li>
<li>who’s behind all this?</li>
</ul>
<p>But even the first book wasn’t perfect. I could feel some rough edges; a
certain lack of polish that one could expect from a new writer (or a lack of
editor).</p>
<p>As for the other books … I cannot recommend them. <a href="https://www.amazon.com/dp/B0BKR6LF72/">Shift</a> (book 2)
moves away from the main story, trying to build enough world to hold things
together. And by <a href="https://www.amazon.com/dp/B0BKRBWZWF/">Dust</a> (book 3), you can feel how the writer wants to get this
over with.</p>
<p>As for the TV show, I think they did a pretty good job. I don’t agree with all
the choices they made, but it’s generally improving the story and pacing things
better. My hope is that future seasons take what’s good from the books, and
continue to fix the rest.</p>
<ul>
<li>book: <a href="https://www.amazon.com/dp/B0BKR8VBYV/">amazon</a></li>
<li>author: <a href="https://en.wikipedia.org/wiki/Hugh_Howey">wikipedia</a></li>
</ul>
<!-- ------------------------------------------------- -->
<h3 id="dragons-egg">Dragon’s Egg</h3>
<p><a href="https://www.amazon.com/dp/B077D2KP72/"><img class="book-cover" src="/assets/best-books-2023/B077D2KP72.jpg" alt="Dragon's Egg" /></a></p>
<p>This book came up on 3 different occasions, during the last year. It seemed
to inspire a certain nostalgia.</p>
<p>This isn’t fantasy: it’s neither about dragons nor their eggs. First published in
1980, this is a science fiction story in the
<a href="https://en.wikipedia.org/wiki/First_contact_(science_fiction)">first contact</a> theme.</p>
<p>Humans have discovered life on a neutron star. You can try to imagine how different,
how alien, it would be compared to human life.</p>
<p>The whole book covers this <em>clash</em> of civilizations. It is both a fleeting moment and the story of everything 😄</p>
<p>I recommend you don’t research the book too much before you try it: enjoy the journey.</p>
<ul>
<li>book: <a href="https://www.amazon.com/dp/B077D2KP72/">amazon</a></li>
<li>book: <a href="https://en.wikipedia.org/wiki/Dragon%27s_Egg">wikipedia</a></li>
<li>author: <a href="https://en.wikipedia.org/wiki/Robert_L._Forward">wikipedia</a></li>
</ul>
<hr />
<h2 id="closing-thoughts">Closing Thoughts</h2>
<p>If you have liked this blog post, please write your own. I would love to read your book
reviews and recommendations.</p>
psql Tips and Tricks2023-10-29T20:36:00+00:00https://blog.jpalardy.com/posts/psql-tips-and-tricks<p>Quick links:</p>
<ul>
<li><a href="#better-defaults">better defaults</a></li>
<li><a href="#variables">variables</a></li>
<li><a href="#temporary-tables">temporary tables</a></li>
</ul>
<h2 id="why-psql">Why psql?</h2>
<p><a href="https://www.postgresql.org/docs/current/app-psql.html">psql</a> is a terminal-based front-end to PostgreSQL. It looks like this:</p>
<p><a href="/assets/psql-tips-tricks/psql-ss.png"><img src="/assets/psql-tips-tricks/psql-ss.png" alt="screenshot of psql" /></a></p>
<p>Why you would use <code class="language-plaintext highlighter-rouge">psql</code> when there are <a href="https://wiki.postgresql.org/wiki/PostgreSQL_Clients">so many other clients</a> available? 🤔</p>
<p>My short answer is that, sooner or later <sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>, you can end up on the wrong side of an <code class="language-plaintext highlighter-rouge">ssh</code> session, and <code class="language-plaintext highlighter-rouge">psql</code> is going to be all you’ve got.</p>
<p>It also doesn’t hurt to learn a few tricks (or to bookmark this page).</p>
<p>This isn’t a general <code class="language-plaintext highlighter-rouge">psql</code> tutorial: you can easily <a href="https://www.google.com/search?q=psql+tutorial">find one elsewhere</a>.</p>
<h2 id="why-i-use-psql">Why I use psql</h2>
<p>I use <code class="language-plaintext highlighter-rouge">psql</code> because:</p>
<ol>
<li>I don’t have to install anything else, which aligns with my minimalistic sensibilities</li>
<li>I combine it with <a href="https://github.com/jpalardy/vim-slime">vim-slime</a>, which seems to give me the best of all worlds</li>
</ol>
<h2 id="better-defaults">Better defaults</h2>
<p>Locally, I keep this config in my <code class="language-plaintext highlighter-rouge">$HOME/.psqlrc</code></p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">\pset pager off
\pset null NULL
\x auto</code></pre></figure>
<p>When I’m on a remote machine, nothing prevents me from copy-pasting these 3 lines directly in <code class="language-plaintext highlighter-rouge">psql</code>.
What does it change?</p>
<p><code class="language-plaintext highlighter-rouge">\pset pager off</code></p>
<p>If a query scrolls off the page – which doesn’t take a lot – you end up in
the pager with the <code class="language-plaintext highlighter-rouge">:</code> prompt. This allows you to scroll up/down … but you
need to press <code class="language-plaintext highlighter-rouge">q</code> to return to the normal prompt. I <em>hate</em> this default
behavior … please let it scroll; and <code class="language-plaintext highlighter-rouge">pager off</code> does exactly that.</p>
<p><code class="language-plaintext highlighter-rouge">\pset null NULL</code></p>
<p>By default, <code class="language-plaintext highlighter-rouge">null</code> values look blank. With it, <code class="language-plaintext highlighter-rouge">null</code> will be shown as <code class="language-plaintext highlighter-rouge">NULL</code>. If you need something more obvious, you can use a different string:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">(null)</code></li>
<li><code class="language-plaintext highlighter-rouge"><null></code></li>
<li>or <a href="https://proinsias.github.io/til/PSQL-Pset-a-better-null-display-character/">this</a> 😄</li>
</ul>
<p><code class="language-plaintext highlighter-rouge">\x auto</code></p>
<p>If you <code class="language-plaintext highlighter-rouge">SELECT</code> and the result is too wide, <code class="language-plaintext highlighter-rouge">psql</code> will wrap the text. That usually ends up looking pretty bad.</p>
<p>So, most people end up using <code class="language-plaintext highlighter-rouge">\x on</code>, which shows up vertically:</p>
<p><a href="/assets/psql-tips-tricks/x-on.png"><img src="/assets/psql-tips-tricks/x-on.png" alt="going vertical with x on" /></a></p>
<p>But with <code class="language-plaintext highlighter-rouge">\x on</code>, even a <code class="language-plaintext highlighter-rouge">SELECT</code> that <em>could</em> fit will end up vertical (and harder to read) … with <code class="language-plaintext highlighter-rouge">\x auto</code>, you will get
horizontal <em>UNLESS</em> it’s too wide.</p>
<p><a href="/assets/psql-tips-tricks/x-auto.png"><img src="/assets/psql-tips-tricks/x-auto.png" alt="back to horizontal with x auto" /></a></p>
<h2 id="variables">Variables</h2>
<p>Variables in <code class="language-plaintext highlighter-rouge">psql</code> can be as useful as variables anywhere else you use them. For example:</p>
<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="err">\</span><span class="k">set</span> <span class="n">pages</span> <span class="mi">100</span>
<span class="c1">-- use it:</span>
<span class="k">SELECT</span> <span class="n">ASIN</span><span class="p">,</span> <span class="k">LEFT</span> <span class="p">(</span><span class="n">title</span><span class="p">,</span> <span class="mi">60</span><span class="p">)</span>
<span class="k">FROM</span> <span class="n">books</span>
<span class="k">WHERE</span> <span class="n">pages</span> <span class="o"><</span> <span class="p">:</span><span class="n">pages</span>
<span class="k">ORDER</span> <span class="k">BY</span> <span class="n">inserted_at</span> <span class="k">DESC</span>
<span class="k">LIMIT</span> <span class="mi">5</span><span class="p">;</span></code></pre></figure>
<p><a href="/assets/psql-tips-tricks/select-with-variable.png"><img src="/assets/psql-tips-tricks/select-with-variable.png" alt="SELECT with a variable" /></a></p>
<p>The main advantages of using variables in <code class="language-plaintext highlighter-rouge">psql</code>:</p>
<ul>
<li>write your query once</li>
<li>no painful text editing of the SQL<br />
(use the up arrow to rerun the same query)</li>
<li>only change the variable value</li>
</ul>
<p>One <em>gotcha</em> for text variables: if your value needs to be “quoted”, there’s a special syntax for that: <code class="language-plaintext highlighter-rouge">:'varname'</code></p>
<p><a href="/assets/psql-tips-tricks/quoted-var.png"><img src="/assets/psql-tips-tricks/quoted-var.png" alt="you need to quote text variables" /></a></p>
<h2 id="temporary-tables">Temporary Tables</h2>
<p>If variables are useful, temporary tables are an extension of that. You can capture multiple values in a
table and use them with the <code class="language-plaintext highlighter-rouge">IN</code> operator:</p>
<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="c1">-- capture multiple values</span>
<span class="k">DROP</span> <span class="k">TABLE</span> <span class="n">IF</span> <span class="k">EXISTS</span> <span class="n">results</span><span class="p">;</span>
<span class="k">CREATE</span> <span class="k">TEMPORARY</span> <span class="k">TABLE</span> <span class="n">results</span> <span class="k">AS</span> <span class="p">(</span>
<span class="k">SELECT</span> <span class="n">ASIN</span> <span class="k">FROM</span> <span class="n">books</span> <span class="k">WHERE</span> <span class="n">pages</span> <span class="o"><</span> <span class="mi">75</span>
<span class="p">);</span>
<span class="c1">-- then, later...</span>
<span class="k">SELECT</span> <span class="n">title</span>
<span class="k">FROM</span> <span class="n">books</span>
<span class="k">WHERE</span> <span class="n">ASIN</span> <span class="k">in</span> <span class="p">(</span><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">results</span><span class="p">)</span>
<span class="k">LIMIT</span> <span class="mi">10</span><span class="p">;</span></code></pre></figure>
<p><a href="/assets/psql-tips-tricks/temp-table.png"><img src="/assets/psql-tips-tricks/temp-table.png" alt="SELECT using a temporary table" /></a></p>
<p>I didn’t have many uses for <code class="language-plaintext highlighter-rouge">TEMPORARY TABLE</code> before I thought of this. But what’s the impact of creating these temporary tables?
And when would they disappear?</p>
<blockquote>
<p>A temporary table, as its name implied, is a short-lived table that exists
for the duration of a database session. PostgreSQL automatically drops the
temporary tables at the end of a session or a transaction.</p>
</blockquote>
<p>(<a href="https://www.postgresqltutorial.com/postgresql-tutorial/postgresql-temporary-table/">reference</a>)</p>
<p>So, your temporary tables will garbage collect themselves when you quit <code class="language-plaintext highlighter-rouge">psql</code>.
They also live in a separate schema, so they won’t be visible to other users.</p>
<hr />
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>Especially when things won’t be going well… <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
Data Analysis: Strange Loop 2023 Videos2023-10-18T17:44:00+00:00https://blog.jpalardy.com/posts/data-analysis-strange-loop-2023-videos<h2 id="what-is-this">What is this?</h2>
<p>A few days ago, I saw the <a href="https://www.youtube.com/@StrangeLoopConf/videos">videos for the 2023 Strange Loop conference</a> starting
to land in my YouTube feed.</p>
<p>This started out as a relatively simple question:</p>
<blockquote>
<p>which videos should I watch?</p>
</blockquote>
<p>It wasn’t my first time looking at a list of videos on YouTube and wondering how to find “the best ones”.</p>
<p>The rest of this post is <em>the journey</em>; trying to find an answer. I wrote this for a few reasons:</p>
<ul>
<li>to show how I did it – which <em>might</em> contain interesting techniques (<a href="https://www.urbandictionary.com/define.php?term=ymmv">ymmv</a>)</li>
<li>to show how messy this is – I think it’s normal and it’s useful to show</li>
<li>to invite feedback – do you have a better way to do this?</li>
</ul>
<h2 id="problem-statement">Problem Statement</h2>
<blockquote>
<p>which videos are worth watching?</p>
</blockquote>
<p>Of course, this is highly subjective. In my case, I’ll break it down as:</p>
<ul>
<li>which videos are other people excited about?</li>
<li>it can be felt indirectly from the buzz on twitter, hacker news, etc…</li>
<li>but it seems like number of <code class="language-plaintext highlighter-rouge">views</code> is a reasonably good proxy</li>
</ul>
<p><a href="/assets/sl-2023-analysis/sample-views.png"><img src="/assets/sl-2023-analysis/sample-views.png" alt="example of videos and views from youtube" /></a></p>
<p>(I’m not trying to pick on anyone, it’s just an example)</p>
<ul>
<li>the <code class="language-plaintext highlighter-rouge">views</code> metric follows a <a href="https://en.wikipedia.org/wiki/Power_law">power law</a> distribution, like many popular things</li>
<li>this is slightly complicated by when a video was published (<code class="language-plaintext highlighter-rouge">### days ago</code>)</li>
</ul>
<p>At this point, the plan usually looks like:</p>
<ul>
<li>let’s get some data</li>
<li>let’s throw it on a graph</li>
</ul>
<h2 id="getting-the-data">Getting The Data</h2>
<p>I tried not to overthink this; I decided to scrape YouTube straight from Chrome’s <code class="language-plaintext highlighter-rouge">Developer Tools</code>.</p>
<p><a href="/assets/sl-2023-analysis/developer-tools.png"><img src="/assets/sl-2023-analysis/developer-tools.png" alt="using Chrome Developer Tools on that youtube page" /></a></p>
<p>thoughts:</p>
<ul>
<li>I was surprised to find <em>everything I want</em> in the <code class="language-plaintext highlighter-rouge">aria-label</code></li>
<li>including a more precise number for <code class="language-plaintext highlighter-rouge">views</code> (2,880), instead of its abbreviated version (2.8k)</li>
<li>this is a nice surprise, because the metadata on the <a href="https://www.youtube.com/playlist?list=PLcGKfGEEONaBNsY_bOj8IhbCPvj6OAdWo">playlist</a> page isn’t as rich <code class="language-plaintext highlighter-rouge">¯\_(ツ)_/¯</code></li>
</ul>
<p>I used this snippet in the developer console:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">copy</span><span class="p">(</span>
<span class="p">[...</span><span class="nb">document</span><span class="p">.</span><span class="nx">querySelectorAll</span><span class="p">(</span><span class="dl">"</span><span class="s2">yt-formatted-string#video-title</span><span class="dl">"</span><span class="p">)].</span>
<span class="nx">map</span><span class="p">(</span><span class="nx">el</span> <span class="o">=></span> <span class="nx">el</span><span class="p">.</span><span class="nx">ariaLabel</span><span class="p">).</span>
<span class="nx">filter</span><span class="p">(</span><span class="nx">text</span> <span class="o">=></span> <span class="nx">text</span><span class="p">).</span>
<span class="nx">join</span><span class="p">(</span><span class="dl">"</span><span class="se">\n</span><span class="dl">"</span><span class="p">)</span>
<span class="p">)</span>
</code></pre></div></div>
<p>breakdown:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">querySelectorAll</code> to grab relevant DOM nodes</li>
<li><code class="language-plaintext highlighter-rouge">[...###]</code> to convert to a plain array</li>
<li>grab only the <code class="language-plaintext highlighter-rouge">aria-label</code></li>
<li>filter out <code class="language-plaintext highlighter-rouge">null</code> values – a better selector might fix this</li>
<li>convert to one big newline-separated string</li>
<li>use <code class="language-plaintext highlighter-rouge">copy</code> to send to the clipboard</li>
</ul>
<p><strong>Caveat</strong>: the page is lazy-loading, scroll enough to capture all the 2023 videos</p>
<h2 id="creating-a-project">Creating a project</h2>
<p><code class="language-plaintext highlighter-rouge">project</code> is a big word. But when I manipulate data, and it involves multiple steps, I usually
create a directory to hold my files. Here’s what I did:</p>
<ul>
<li>I created a directory with a name that contains a date: <code class="language-plaintext highlighter-rouge">2023-10-14-strange-loop-2023-data-analysis</code></li>
<li>I created a <code class="language-plaintext highlighter-rouge">README.md</code> and dumped my notes in there, best effort 😬</li>
<li>I added a <code class="language-plaintext highlighter-rouge">Makefile</code> to document <em>the logic</em>
<ul>
<li>how I fetch the raw data<br />
(although not in this instance, since I copy-pasted from Chrome)</li>
<li>how I transform the data</li>
<li>how to generate graphs / reports</li>
</ul>
</li>
<li>I copied a <a href="https://github.com/jpalardy/templates/tree/main/rmd">reference Rmd file</a></li>
<li>I created a <code class="language-plaintext highlighter-rouge">scripts/</code> subdirectory to hold helper scripts</li>
</ul>
<p>The point is to do an amount of bureaucracy proportional to the task at hand.</p>
<p><a href="/assets/sl-2023-analysis/project-dir.png"><img src="/assets/sl-2023-analysis/project-dir.png" alt="the project directory's content" /></a></p>
<h2 id="massaging-the-data">Massaging the data</h2>
<p>I pasted the data to a file:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>pbpaste <span class="o">></span> data.raw
</code></pre></div></div>
<p>Then, I opened the file in <code class="language-plaintext highlighter-rouge">vim</code> and cleaned up the entries:</p>
<p><a href="/assets/sl-2023-analysis/raw-entries.png"><img src="/assets/sl-2023-analysis/raw-entries.png" alt="raw data in vim" /></a></p>
<p>thoughts:</p>
<ul>
<li>the correct answer is 45; that’s how many files are in the playlist (2023-10-14)</li>
<li>obvious discontinuity: <code class="language-plaintext highlighter-rouge">weeks ago</code> vs <code class="language-plaintext highlighter-rouge">months ago</code></li>
<li>I usually automate this with a script
<ul>
<li>to document and reproduce later</li>
<li>but I was eyeballing the data and this wasn’t brain surgery</li>
</ul>
</li>
<li>I confirmed first and last entries, and deleted everything below line 45</li>
</ul>
<p>Looking at the data more closely:</p>
<p><a href="/assets/sl-2023-analysis/raw-entries-zoomed.png"><img src="/assets/sl-2023-analysis/raw-entries-zoomed.png" alt="zoomed in raw data in vim" /></a></p>
<p>I came up with this <code class="language-plaintext highlighter-rouge">awk</code> script:</p>
<div class="language-awk highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">match</span><span class="p">(</span><span class="nv">$0</span><span class="p">,</span> <span class="sr">/</span><span class="se">(</span><span class="sr">.*</span><span class="se">)</span><span class="sr"> </span><span class="se">(</span><span class="sr">.*</span><span class="se">)</span><span class="sr"> views </span><span class="se">(</span><span class="sr">.*</span><span class="se">)</span><span class="sr"> </span><span class="se">(</span><span class="sr">.*</span><span class="se">)</span><span class="sr"> ago/</span><span class="p">,</span> <span class="nx">arr</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">title</span> <span class="o">=</span> <span class="nx">arr</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="nx">views</span> <span class="o">=</span> <span class="nx">arr</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span>
<span class="nx">count</span> <span class="o">=</span> <span class="nx">arr</span><span class="p">[</span><span class="mi">3</span><span class="p">]</span>
<span class="nx">unit</span> <span class="o">=</span> <span class="nx">arr</span><span class="p">[</span><span class="mi">4</span><span class="p">]</span>
<span class="nb">sub</span><span class="p">(</span><span class="s2">","</span><span class="p">,</span> <span class="s2">""</span><span class="p">,</span> <span class="nx">views</span><span class="p">)</span>
<span class="nx">multiplier</span> <span class="o">=</span> <span class="mi">1</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">unit</span> <span class="o">==</span> <span class="s2">"week"</span> <span class="o">||</span> <span class="nx">unit</span> <span class="o">==</span> <span class="s2">"weeks"</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">multiplier</span> <span class="o">=</span> <span class="mi">7</span>
<span class="p">}</span>
<span class="nx">days</span> <span class="o">=</span> <span class="nx">count</span> <span class="o">*</span> <span class="nx">multiplier</span>
<span class="k">print</span> <span class="nx">views</span> <span class="s2">"\t"</span> <span class="nx">days</span> <span class="s2">"\t"</span> <span class="nx">title</span>
<span class="p">}</span>
</code></pre></div></div>
<p>breakdown:</p>
<ul>
<li>capture groups for relevant parts of each line
<ol>
<li>title of the video</li>
<li>number of views (with comma)</li>
<li>number of days/weeks/months</li>
<li>time unit (days/weeks/months)</li>
</ol>
</li>
<li>remove the comma from <code class="language-plaintext highlighter-rouge">number of views</code></li>
<li>multiplier: a <code class="language-plaintext highlighter-rouge">week</code> means 7 days
<ul>
<li>assumption: default is <code class="language-plaintext highlighter-rouge">days</code></li>
<li>assumption: no <code class="language-plaintext highlighter-rouge">months</code> in current subset of data</li>
</ul>
</li>
<li>outputting massaged data in tab-separated format (<code class="language-plaintext highlighter-rouge">.tsv</code>)
<ul>
<li>similar to <code class="language-plaintext highlighter-rouge">.csv</code></li>
<li>(almost) no need to worry about special characters</li>
<li>trivial to generate</li>
<li>easy to work with (Excel, R …)</li>
</ul>
</li>
</ul>
<p>Again, I only did what I needed for <em>TODAY</em>. This was a conscious decision.</p>
<p><strong>Caveats</strong></p>
<ul>
<li>it’s hard to guess how data that you don’t control is going to change</li>
<li>you might want to list your assumptions</li>
<li>and iterate…</li>
</ul>
<p>Here’s what the <code class="language-plaintext highlighter-rouge">.tsv</code> looked like:</p>
<p><a href="/assets/sl-2023-analysis/data-tsv.png"><img src="/assets/sl-2023-analysis/data-tsv.png" alt="sample tab-separated data" /></a></p>
<h2 id="exploring-the-data">Exploring the data</h2>
<p>Personally, I use <a href="https://www.r-project.org/">R</a>. Feel free to use something else. Pick a tool and learn it well.</p>
<p>I loaded the data into R:</p>
<div class="language-r highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">library</span><span class="p">(</span><span class="n">tidyverse</span><span class="p">)</span><span class="w">
</span><span class="n">d</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">read_tsv</span><span class="p">(</span><span class="s2">"data.tsv"</span><span class="p">,</span><span class="w"> </span><span class="n">col_names</span><span class="o">=</span><span class="nf">c</span><span class="p">(</span><span class="s2">"views"</span><span class="p">,</span><span class="w"> </span><span class="s2">"days"</span><span class="p">,</span><span class="w"> </span><span class="s2">"title"</span><span class="p">),</span><span class="w">
</span><span class="n">col_types</span><span class="o">=</span><span class="n">cols</span><span class="p">(</span><span class="w">
</span><span class="s2">"views"</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">col_double</span><span class="p">(),</span><span class="w">
</span><span class="s2">"days"</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">col_double</span><span class="p">(),</span><span class="w">
</span><span class="s2">"title"</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">col_character</span><span class="p">()</span><span class="w">
</span><span class="p">))</span><span class="w"> </span><span class="o">|></span><span class="w">
</span><span class="n">mutate</span><span class="p">(</span><span class="n">mean_daily_views</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">views</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="n">days</span><span class="p">)</span><span class="w">
</span></code></pre></div></div>
<p>breakdown:</p>
<ul>
<li>load the data</li>
<li>name the columns</li>
<li>cast the data to a type (not strictly necessary, but a good practice)</li>
<li>add a new column as <code class="language-plaintext highlighter-rouge">views</code> divided per <code class="language-plaintext highlighter-rouge">days</code> (the “average”)</li>
</ul>
<p>Let’s plot <code class="language-plaintext highlighter-rouge">views</code> against <code class="language-plaintext highlighter-rouge">days</code></p>
<div class="language-r highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">ggplot</span><span class="p">(</span><span class="n">d</span><span class="p">,</span><span class="w"> </span><span class="n">aes</span><span class="p">(</span><span class="n">days</span><span class="p">,</span><span class="w"> </span><span class="n">views</span><span class="p">))</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">geom_point</span><span class="p">(</span><span class="n">alpha</span><span class="o">=</span><span class="m">0.3</span><span class="p">,</span><span class="w"> </span><span class="n">size</span><span class="o">=</span><span class="m">2.5</span><span class="p">,</span><span class="w"> </span><span class="n">stroke</span><span class="o">=</span><span class="m">0</span><span class="p">,</span><span class="w"> </span><span class="n">color</span><span class="o">=</span><span class="s2">"red"</span><span class="p">)</span><span class="w">
</span></code></pre></div></div>
<p><a href="/assets/sl-2023-analysis/views-vs-days.png"><img src="/assets/sl-2023-analysis/views-vs-days.png" alt="sample tab-separated data" /></a></p>
<p>thoughts:</p>
<ul>
<li>more <code class="language-plaintext highlighter-rouge">views</code> is better</li>
<li>but a video published longer has had more chance to gather <code class="language-plaintext highlighter-rouge">views</code>
<ul>
<li>e.g. 5000 views in one day is more impressive than 5000 views in 20 days</li>
</ul>
</li>
</ul>
<p>Here’s the (power law) distribution for <code class="language-plaintext highlighter-rouge">views</code>:</p>
<div class="language-r highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">ggplot</span><span class="p">(</span><span class="n">d</span><span class="p">,</span><span class="w"> </span><span class="n">aes</span><span class="p">(</span><span class="n">reorder</span><span class="p">(</span><span class="n">str_trunc</span><span class="p">(</span><span class="n">title</span><span class="p">,</span><span class="w"> </span><span class="m">40</span><span class="p">),</span><span class="w"> </span><span class="n">views</span><span class="p">),</span><span class="w"> </span><span class="n">views</span><span class="p">))</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">geom_col</span><span class="p">(</span><span class="n">alpha</span><span class="o">=</span><span class="m">0.5</span><span class="p">,</span><span class="w"> </span><span class="n">fill</span><span class="o">=</span><span class="s2">"red"</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">xlab</span><span class="p">(</span><span class="s2">"title"</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">coord_flip</span><span class="p">()</span><span class="w">
</span></code></pre></div></div>
<p><a href="/assets/sl-2023-analysis/views-power-law.png"><img src="/assets/sl-2023-analysis/views-power-law.png" alt="sample tab-separated data" /></a></p>
<p>breakdown:</p>
<ul>
<li>a few very popular videos</li>
<li>a sharp “elbow” around 4-5th entry</li>
<li>a long tail of other videos</li>
</ul>
<p>What about <code class="language-plaintext highlighter-rouge">daily views</code> against <code class="language-plaintext highlighter-rouge">views</code>?</p>
<div class="language-r highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">ggplot</span><span class="p">(</span><span class="n">d</span><span class="p">,</span><span class="w"> </span><span class="n">aes</span><span class="p">(</span><span class="n">views</span><span class="p">,</span><span class="w"> </span><span class="n">mean_daily_views</span><span class="p">))</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">geom_point</span><span class="p">(</span><span class="n">alpha</span><span class="o">=</span><span class="m">0.3</span><span class="p">,</span><span class="w"> </span><span class="n">size</span><span class="o">=</span><span class="m">2.5</span><span class="p">,</span><span class="w"> </span><span class="n">stroke</span><span class="o">=</span><span class="m">0</span><span class="p">,</span><span class="w"> </span><span class="n">color</span><span class="o">=</span><span class="s2">"red"</span><span class="p">)</span><span class="w">
</span></code></pre></div></div>
<p><a href="/assets/sl-2023-analysis/daily-views-vs-views.png"><img src="/assets/sl-2023-analysis/daily-views-vs-views.png" alt="sample tab-separated data" /></a></p>
<p>breakdown:</p>
<ul>
<li>more <code class="language-plaintext highlighter-rouge">views</code> is better</li>
<li>more <code class="language-plaintext highlighter-rouge">views/days</code> is better</li>
<li>distance from the origin (in either direction) is a sign of popularity</li>
</ul>
<p>Finally, the videos, by top <code class="language-plaintext highlighter-rouge">views</code> and top <code class="language-plaintext highlighter-rouge">daily views</code>:</p>
<p><a href="/assets/sl-2023-analysis/tops.png"><img src="/assets/sl-2023-analysis/tops.png" alt="sample tab-separated data" /></a></p>
<h2 id="discussion">Discussion</h2>
<p>In the end, I didn’t find any deep insights in this data:</p>
<ul>
<li>few dimensions (<code class="language-plaintext highlighter-rouge">views</code> and <code class="language-plaintext highlighter-rouge">days</code>)</li>
<li>unfortunately, no per-day breakdowns …<br />
e.g. “this video had this many views on that day”</li>
<li>leading to averaging, which <a href="https://blog.jpalardy.com/posts/reject-summary-statistics/">I have feelings about</a></li>
</ul>
<p>Maybe the power law distribution leads to <em>obvious</em> conclusions: watch what everybody else watched?</p>
<p>Here’s another view of the same data:</p>
<div class="language-r highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">d</span><span class="w"> </span><span class="o">|></span><span class="w">
</span><span class="n">mutate</span><span class="p">(</span><span class="n">popular</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">mean_daily_views</span><span class="w"> </span><span class="o">>=</span><span class="w"> </span><span class="m">500</span><span class="p">)</span><span class="w"> </span><span class="o">|></span><span class="w">
</span><span class="n">ggplot</span><span class="p">(</span><span class="n">aes</span><span class="p">(</span><span class="n">days</span><span class="p">,</span><span class="w"> </span><span class="n">views</span><span class="p">))</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">geom_line</span><span class="p">(</span><span class="n">aes</span><span class="p">(</span><span class="n">x</span><span class="o">=</span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="n">y</span><span class="o">=</span><span class="n">y</span><span class="p">),</span><span class="w"> </span><span class="n">alpha</span><span class="o">=</span><span class="m">0.1</span><span class="p">,</span><span class="w"> </span><span class="n">size</span><span class="o">=</span><span class="m">3</span><span class="p">,</span><span class="w"> </span><span class="n">color</span><span class="o">=</span><span class="s2">"red"</span><span class="p">,</span><span class="w"> </span><span class="n">data</span><span class="o">=</span><span class="n">guide.data</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">geom_point</span><span class="p">(</span><span class="n">aes</span><span class="p">(</span><span class="n">color</span><span class="o">=</span><span class="n">popular</span><span class="p">),</span><span class="w"> </span><span class="n">alpha</span><span class="o">=</span><span class="m">0.3</span><span class="p">,</span><span class="w"> </span><span class="n">size</span><span class="o">=</span><span class="m">2.5</span><span class="p">,</span><span class="w"> </span><span class="n">stroke</span><span class="o">=</span><span class="m">0</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">scale_y_log10</span><span class="p">()</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">scale_x_log10</span><span class="p">()</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="kc">NULL</span><span class="w">
</span></code></pre></div></div>
<p><a href="/assets/sl-2023-analysis/popular.png"><img src="/assets/sl-2023-analysis/popular.png" alt="log-log view of same data" /></a></p>
<p>breakdown:</p>
<ul>
<li>same <code class="language-plaintext highlighter-rouge">views</code> over <code class="language-plaintext highlighter-rouge">days</code></li>
<li>careful: using log scales ⚠️
<ul>
<li>log scales allow different magnitudes to be compared; to fit a smaller area</li>
<li>they say all the benefits of log scales are cancelled out by having to explain log scales…</li>
</ul>
</li>
<li>pink line is <code class="language-plaintext highlighter-rouge">500 views per day</code></li>
<li>above the line is “popular” (teal)</li>
</ul>
entr: The Standalone File Watcher2023-08-02T00:00:00+00:00https://blog.jpalardy.com/posts/entr-the-standalone-file-watcher<h2 id="what-is-entr">What is entr?</h2>
<p><a href="http://eradman.com/entrproject/">entr</a> is <em>just</em> a file watcher. The <a href="https://github.com/eradman/entr">README</a> says:</p>
<blockquote>
<p>Run arbitrary commands when files change</p>
</blockquote>
<p>It’s available for most package managers: <code class="language-plaintext highlighter-rouge">brew install entr</code></p>
<h2 id="a-simple-example">A simple example</h2>
<p>List the files you’re interested in, one per line:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">ls</span> <span class="k">*</span>.txt
a.txt
b.txt
c.txt</code></pre></figure>
<p>Now feed them to <code class="language-plaintext highlighter-rouge">entr</code> with the command you want to run:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">ls</span> <span class="k">*</span>.txt | entr make</code></pre></figure>
<h2 id="why-entr">Why entr?</h2>
<p>A lot of projects and languages have their own “watcher” or “–watch” flag.</p>
<p>But I enjoy tools that work with everything that I want to do.</p>
<p>Even if you feel that <a href="https://webpack.js.org/">webpack</a> (for example) is fulfilling all your needs, would you use it
to watch a few <code class="language-plaintext highlighter-rouge">.txt</code> files and trigger some custom job?</p>
<p>I don’t want every tool to roll out their own “watcher”. That’s not the <a href="https://en.wikipedia.org/wiki/Unix_philosophy">Unix philosophy</a>.</p>
<h2 id="detour-how-to-list-files-one-per-line">Detour: how to list files, one per line</h2>
<p>Usually, you would feed the output of <code class="language-plaintext highlighter-rouge">ls</code> or <code class="language-plaintext highlighter-rouge">find</code> to <code class="language-plaintext highlighter-rouge">entr</code>.</p>
<p>There are occasions where that list will be <a href="https://eradman.com/entrproject/limits.html">too large</a>. Or, you might want a custom
list of files. How do you print arguments one per line?</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">printf</span> <span class="s2">"%s</span><span class="se">\n</span><span class="s2">"</span> a.txt b.txt c.txt
a.txt
b.txt
c.txt</code></pre></figure>
<p>(see: <a href="https://unix.stackexchange.com/questions/65803/why-is-printf-better-than-echo">Why is printf better than echo?</a>)</p>
<p>Other options I’ve used:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># output of "grep"</span>
<span class="o">></span> rg <span class="nt">-l</span> query
<span class="c"># current in-play git files</span>
<span class="o">></span> git status <span class="nt">--short</span> | <span class="nb">sed</span> <span class="s1">'s/^...//'</span>
<span class="c"># or some other git-generated list</span>
<span class="c"># specific source file and its test file</span>
<span class="o">></span> <span class="nb">printf</span> <span class="s2">"%s</span><span class="se">\n</span><span class="s2">"</span> lib/whatever.ex <span class="nb">test</span>/whatever.exs</code></pre></figure>
<h2 id="noteworthy-tips">Noteworthy tips</h2>
<p>The source of truth is the <a href="http://eradman.com/entrproject/entr.1.html">man page</a>.
Here are a few tips:</p>
<ul>
<li>the <code class="language-plaintext highlighter-rouge">-c</code> flag clears the screen every time entr triggers</li>
<li>the <code class="language-plaintext highlighter-rouge">-r</code> flag restarts the command on every trigger</li>
<li>the <code class="language-plaintext highlighter-rouge">-s</code> flag allows for more complete shell commands (with pipe…)</li>
<li>hitting ‘space’ will manually force-trigger entr</li>
<li><code class="language-plaintext highlighter-rouge">/_</code> stands for the first file to trigger an event</li>
</ul>
<p>I also found that combining <code class="language-plaintext highlighter-rouge">entr</code> with <a href="/posts/ding/">ding</a> allows me to run “headless”</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">printf</span> <span class="s2">"%s</span><span class="se">\n</span><span class="s2">"</span> lib/whatever.ex <span class="nb">test</span>/whatever.exs | entr ding mix <span class="nb">test</span></code></pre></figure>
<p>I can hear when tests pass/fail based on the exit code. 😎</p>
Reduce Anti-Patterns2023-07-12T01:37:00+00:00https://blog.jpalardy.com/posts/reduce-anti-patterns<p>I mentioned in <a href="/posts/reduce-is-not-the-answer/">Reduce is Not the Answer</a> how:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">reduce</code> is fairly low-level</li>
<li><code class="language-plaintext highlighter-rouge">reduce</code> doesn’t capture “intent”</li>
</ul>
<p>Very loosely, I believe that “loop constructs” fall somewhere on this abstraction scale:</p>
<p><a href="/assets/reduce-anti-patterns/loop-abstraction-scale.png"><img src="/assets/reduce-anti-patterns/loop-abstraction-scale.png" alt="loop abstraction scale" /></a></p>
<p>Exact placement is subjective; this is a starting point. As for <code class="language-plaintext highlighter-rouge">??</code> and beyond,
it might correspond to well-named functions with custom looping logic. For example, <code class="language-plaintext highlighter-rouge">mean</code> and <code class="language-plaintext highlighter-rouge">stdev</code> would
fit the bill. However, these function quickly become less generic.</p>
<p>If I missed something to the right, let’s talk 😄</p>
<p>With this post, I want to sketch typical <code class="language-plaintext highlighter-rouge">reduce</code> patterns and identify their higher-level alternatives.</p>
<h2 id="disclaimers">Disclaimers</h2>
<ul>
<li>Elixir is used; that should not stop anyone from understanding what’s going on</li>
<li>I’m aiming for <em>useful</em>, not <em>“revelation”</em></li>
<li>I put anchors ( 🔗 ) next to the section headers, copy-paste them as needed</li>
<li>your code <em>might</em> be an exception…</li>
</ul>
<p>I initially added a comment to almost all sections:</p>
<blockquote>
<p>This is usually combined with other logic, to “save the extra loop”.</p>
</blockquote>
<p>Yes, that’s usually how it goes.</p>
<h2 id="enummap-">Enum.map <a href="#enummap-">🔗</a></h2>
<p>Usual shape:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="no">Enum</span><span class="o">.</span><span class="n">reduce</span><span class="p">(</span><span class="n">collection</span><span class="p">,</span> <span class="p">[],</span> <span class="k">fn</span> <span class="n">item</span><span class="p">,</span> <span class="n">acc</span> <span class="o">-></span>
<span class="p">[</span><span class="n">item</span><span class="o">.</span><span class="n">value</span> <span class="o">|</span> <span class="n">acc</span><span class="p">]</span>
<span class="k">end</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">reverse</span><span class="p">()</span></code></pre></figure>
<p>Giveaways:</p>
<ul>
<li>initial empty list</li>
<li>item mapping or transformation</li>
<li>prepend-reverse, or append</li>
</ul>
<p>Alternative:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="no">Enum</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="n">collection</span><span class="p">,</span> <span class="n">transformFun</span><span class="p">)</span></code></pre></figure>
<p>Comments:</p>
<p>Not often <em>directly</em> found in the wild.</p>
<h2 id="enumfilter--enumreject-">Enum.filter / Enum.reject <a href="#enumfilter--enumreject-">🔗</a></h2>
<p>Usual shape:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="no">Enum</span><span class="o">.</span><span class="n">reduce</span><span class="p">(</span><span class="n">collection</span><span class="p">,</span> <span class="p">[],</span> <span class="k">fn</span> <span class="n">item</span><span class="p">,</span> <span class="n">acc</span> <span class="o">-></span>
<span class="k">if</span> <span class="n">condition</span> <span class="k">do</span>
<span class="p">[</span><span class="n">item</span> <span class="o">|</span> <span class="n">acc</span><span class="p">]</span>
<span class="k">else</span>
<span class="n">acc</span>
<span class="k">end</span>
<span class="k">end</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">reverse</span><span class="p">()</span></code></pre></figure>
<p>Giveaways:</p>
<ul>
<li>initial empty list</li>
<li>some conditional</li>
<li>prepend-reverse, or append</li>
</ul>
<p>Alternative:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="no">Enum</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">collection</span><span class="p">,</span> <span class="n">predicateFun</span><span class="p">)</span>
<span class="c1"># or, depending on the exact logic</span>
<span class="no">Enum</span><span class="o">.</span><span class="n">reject</span><span class="p">(</span><span class="n">collection</span><span class="p">,</span> <span class="n">predicateFun</span><span class="p">)</span></code></pre></figure>
<p>Comments:</p>
<p><em>Probably</em> combined with a <code class="language-plaintext highlighter-rouge">map</code>.</p>
<h2 id="enumfind-">Enum.find <a href="#enumfind-">🔗</a></h2>
<p>Usual shape:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="no">Enum</span><span class="o">.</span><span class="n">reduce_while</span><span class="p">(</span><span class="n">collection</span><span class="p">,</span> <span class="no">nil</span><span class="p">,</span> <span class="k">fn</span> <span class="n">item</span><span class="p">,</span> <span class="n">acc</span> <span class="o">-></span>
<span class="k">if</span> <span class="n">condition</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:halt</span><span class="p">,</span> <span class="n">item</span><span class="p">}</span>
<span class="k">else</span>
<span class="p">{</span><span class="ss">:cont</span><span class="p">,</span> <span class="n">acc</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">end</span><span class="p">)</span></code></pre></figure>
<p>Giveaways:</p>
<ul>
<li>initial <code class="language-plaintext highlighter-rouge">nil</code> or sentinel value</li>
<li>conditional; rest of loop doesn’t matter…</li>
</ul>
<p>Alternative:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="no">Enum</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="n">collection</span><span class="p">,</span> <span class="n">transformFun</span><span class="p">)</span></code></pre></figure>
<p>Comments:</p>
<p>It might be driven by an edge case or custom logic.</p>
<h2 id="enumall--enumany-">Enum.all? / Enum.any? <a href="#enumall--enumany-">🔗</a></h2>
<p>Usual shape:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="no">Enum</span><span class="o">.</span><span class="n">reduce_while</span><span class="p">(</span><span class="n">collection</span><span class="p">,</span> <span class="no">true</span><span class="p">,</span> <span class="k">fn</span> <span class="n">item</span><span class="p">,</span> <span class="n">acc</span> <span class="o">-></span>
<span class="k">if</span> <span class="n">condition</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:cont</span><span class="p">,</span> <span class="n">acc</span><span class="p">}</span>
<span class="k">else</span>
<span class="p">{</span><span class="ss">:halt</span><span class="p">,</span> <span class="no">false</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">end</span><span class="p">)</span></code></pre></figure>
<p>Giveaways:</p>
<ul>
<li>initial flag</li>
<li>conditional toggles a flag; rest of loop doesn’t matter…</li>
</ul>
<p>Alternative:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="no">Enum</span><span class="o">.</span><span class="n">all?</span><span class="p">(</span><span class="n">collection</span><span class="p">,</span> <span class="n">predicateFun</span><span class="p">)</span>
<span class="c1"># or, depending on the exact logic</span>
<span class="no">Enum</span><span class="o">.</span><span class="n">any?</span><span class="p">(</span><span class="n">collection</span><span class="p">,</span> <span class="n">predicateFun</span><span class="p">)</span></code></pre></figure>
<p>Comments:</p>
<p>In some languages (e.g. JavaScript), this was be a later addition. It
also goes under various names (<code class="language-plaintext highlighter-rouge">some</code>, <code class="language-plaintext highlighter-rouge">every</code>…) and it might not be easy to
understand what it does and how/when to use it.</p>
<h2 id="enumcount-">Enum.count <a href="#enumcount-">🔗</a></h2>
<p>Usual shape:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="no">Enum</span><span class="o">.</span><span class="n">reduce</span><span class="p">(</span><span class="n">collection</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="k">fn</span> <span class="n">item</span><span class="p">,</span> <span class="n">acc</span> <span class="o">-></span>
<span class="k">if</span> <span class="n">condition</span> <span class="k">do</span>
<span class="n">acc</span> <span class="o">+</span> <span class="mi">1</span>
<span class="k">else</span>
<span class="n">acc</span>
<span class="k">end</span>
<span class="k">end</span><span class="p">)</span></code></pre></figure>
<p>Giveaways:</p>
<ul>
<li>initial zero</li>
<li>conditional math</li>
</ul>
<p>Alternative:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="no">Enum</span><span class="o">.</span><span class="n">count</span><span class="p">(</span><span class="n">collection</span><span class="p">,</span> <span class="n">predicateFun</span><span class="p">)</span></code></pre></figure>
<p>Comments:</p>
<p>This isn’t always available; in that case, I would go for:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="no">Enum</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">collection</span><span class="p">,</span> <span class="n">predicateFun</span><span class="p">)</span> <span class="o">|></span> <span class="n">length</span><span class="p">()</span></code></pre></figure>
<h2 id="enumsum-">Enum.sum <a href="#enumsum-">🔗</a></h2>
<p>Usual shape:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="no">Enum</span><span class="o">.</span><span class="n">reduce</span><span class="p">(</span><span class="n">collection</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="k">fn</span> <span class="n">item</span><span class="p">,</span> <span class="n">acc</span> <span class="o">-></span>
<span class="k">if</span> <span class="n">condition</span> <span class="k">do</span>
<span class="n">acc</span> <span class="o">+</span> <span class="n">item</span><span class="o">.</span><span class="n">value</span>
<span class="k">else</span>
<span class="n">acc</span>
<span class="k">end</span>
<span class="k">end</span><span class="p">)</span></code></pre></figure>
<p>Giveaways:</p>
<ul>
<li>initial zero</li>
<li>conditional math?</li>
<li>value extraction/generation</li>
</ul>
<p>Alternative:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="no">Enum</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">collection</span><span class="p">,</span> <span class="n">predicateFun</span><span class="p">)</span> <span class="c1"># if needed</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="n">transformFun</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">sum</span><span class="p">()</span></code></pre></figure>
<h2 id="enumgroup_by--enumfrequencies--enumsplit_with-">Enum.group_by / Enum.frequencies / Enum.split_with <a href="#enumgroup_by--enumfrequencies--enumsplit_with-">🔗</a></h2>
<p>Usual shape:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="no">Enum</span><span class="o">.</span><span class="n">reduce</span><span class="p">(</span><span class="n">collection</span><span class="p">,</span> <span class="p">%{},</span> <span class="k">fn</span> <span class="n">item</span><span class="p">,</span> <span class="n">acc</span> <span class="o">-></span>
<span class="n">key</span> <span class="o">=</span> <span class="k">fn</span><span class="o">.</span><span class="p">(</span><span class="n">item</span><span class="p">)</span>
<span class="k">if</span> <span class="no">Map</span><span class="o">.</span><span class="n">has_key?</span><span class="p">(</span><span class="n">acc</span><span class="p">,</span> <span class="n">key</span><span class="p">)</span> <span class="k">do</span>
<span class="no">Map</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">acc</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="p">[</span><span class="n">item</span> <span class="o">|</span> <span class="no">Map</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">acc</span><span class="p">,</span> <span class="n">key</span><span class="p">)])</span>
<span class="k">else</span>
<span class="no">Map</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">acc</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="p">[</span><span class="n">item</span><span class="p">])</span>
<span class="k">end</span>
<span class="k">end</span><span class="p">)</span></code></pre></figure>
<p>Giveaways:</p>
<ul>
<li>initial empty map</li>
<li>conditional update of map</li>
<li>management of “not yet in map” edge case</li>
</ul>
<p>Alternative:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="no">Enum</span><span class="o">.</span><span class="n">group_by</span><span class="p">(</span><span class="n">collection</span><span class="p">,</span> <span class="n">keyFun</span><span class="p">)</span>
<span class="c1"># or, depending on the exact logic</span>
<span class="no">Enum</span><span class="o">.</span><span class="n">frequencies</span><span class="p">(</span><span class="n">collection</span><span class="p">)</span>
<span class="c1"># or, depending on the exact logic</span>
<span class="no">Enum</span><span class="o">.</span><span class="n">split_with</span><span class="p">(</span><span class="n">collection</span><span class="p">,</span> <span class="n">predicateFun</span><span class="p">)</span></code></pre></figure>
<p>Comments:</p>
<p>This is all the same logic: divide things into groups.</p>
<ul>
<li>for <code class="language-plaintext highlighter-rouge">frequencies</code>, you only care about the size of the groups</li>
<li>for <code class="language-plaintext highlighter-rouge">split_with</code> (often named <code class="language-plaintext highlighter-rouge">partition</code>), there are only 2 groups</li>
</ul>
<h2 id="enummin--enummax-">Enum.min / Enum.max <a href="#enummin--enummax-">🔗</a></h2>
<p>Usual shape:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="no">Enum</span><span class="o">.</span><span class="n">reduce</span><span class="p">(</span><span class="n">collection</span><span class="p">,</span> <span class="k">fn</span> <span class="n">item</span><span class="p">,</span> <span class="n">acc</span> <span class="o">-></span>
<span class="k">if</span> <span class="n">item</span> <span class="o"><</span> <span class="n">acc</span> <span class="k">do</span>
<span class="n">item</span>
<span class="k">else</span>
<span class="n">acc</span>
<span class="k">end</span>
<span class="k">end</span><span class="p">)</span></code></pre></figure>
<p>Giveaways:</p>
<ul>
<li>initial sentinel value?</li>
<li>conditional update</li>
</ul>
<p>Alternative:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="no">Enum</span><span class="o">.</span><span class="n">min</span><span class="p">(</span><span class="n">collection</span><span class="p">)</span>
<span class="c1"># or, depending on the exact logic</span>
<span class="no">Enum</span><span class="o">.</span><span class="n">max</span><span class="p">(</span><span class="n">collection</span><span class="p">)</span></code></pre></figure>
<h2 id="discussion">Discussion</h2>
<p>Eventually, more languages and compilers will be able to grab a combination of
map-filter-etc and compile it down to one loop.</p>
<p>In the meantime, I’m afraid we will have to live with “optimized” one-loop <code class="language-plaintext highlighter-rouge">reduce</code>.</p>
<p>I’m always happy to discuss specific benchmarks regarding why your <code class="language-plaintext highlighter-rouge">reduce</code> can’t
be a combination of <code class="language-plaintext highlighter-rouge">map</code>, <code class="language-plaintext highlighter-rouge">filter</code> and other <code class="language-plaintext highlighter-rouge">Enum</code> functions.</p>
Reduce is Not the Answer2023-05-28T19:27:00+00:00https://blog.jpalardy.com/posts/reduce-is-not-the-answer<h2 id="tldr">TLDR</h2>
<p>The <code class="language-plaintext highlighter-rouge">reduce</code> function:</p>
<ul>
<li>can do <em>anything</em>, and often does…</li>
<li>fails to communicate intent compared to its alternatives</li>
</ul>
<p><code class="language-plaintext highlighter-rouge">reduce</code> is the for-loop of functional programming…</p>
<h2 id="what-is-reduce">What is reduce?</h2>
<p>The <a href="https://en.wikipedia.org/wiki/Fold_(higher-order_function)">reduce</a>
function takes a collection, iterates over it, and generates a single value.</p>
<p>Depending on the language, it might also be called: <strong>fold</strong>, <strong>accumulate</strong>, <strong>aggregate</strong>, <strong>compress</strong>, or <strong>inject</strong>.</p>
<figure class="highlight"><pre><code class="language-js" data-lang="js"><span class="c1">// a quick JavaScript example</span>
<span class="o">></span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">].</span><span class="nx">reduce</span><span class="p">((</span><span class="nx">acc</span><span class="p">,</span> <span class="nx">x</span><span class="p">)</span> <span class="o">=></span> <span class="nx">acc</span> <span class="o">+</span> <span class="nx">x</span><span class="p">);</span>
<span class="mi">15</span></code></pre></figure>
<p>If you’re not already familiar with <code class="language-plaintext highlighter-rouge">reduce</code>, let me <a href="https://www.google.com/search?q=reduce+function+explained">punt to Google</a>.</p>
<h2 id="whats-the-problem">What’s the problem?</h2>
<p><code class="language-plaintext highlighter-rouge">reduce</code> is just a for-loop, with some “accumulator”. When the accumulator is a list/map, things can get complicated pretty fast.</p>
<p>Here’s a for-loop:</p>
<figure class="highlight"><pre><code class="language-js" data-lang="js"><span class="kd">function</span> <span class="nx">doSomething</span><span class="p">(</span><span class="nx">arr</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">let</span> <span class="nx">acc</span> <span class="o">=</span> <span class="p">[];</span>
<span class="k">for</span> <span class="p">(</span><span class="kd">let</span> <span class="nx">value</span> <span class="k">of</span> <span class="nx">arr</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// complicated code</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nx">acc</span><span class="p">;</span>
<span class="p">}</span></code></pre></figure>
<p>And here’s the same code as a <code class="language-plaintext highlighter-rouge">reduce</code>:</p>
<figure class="highlight"><pre><code class="language-js" data-lang="js"><span class="kd">function</span> <span class="nx">doSomething</span><span class="p">(</span><span class="nx">arr</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">arr</span><span class="p">.</span><span class="nx">reduce</span><span class="p">((</span><span class="nx">acc</span><span class="p">,</span> <span class="nx">value</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="c1">// complicated code</span>
<span class="p">},</span> <span class="p">[]);</span>
<span class="p">}</span></code></pre></figure>
<p>What <em>COULD</em> this code do? 🤔</p>
<p>Anything, really.</p>
<p>It could be a <code class="language-plaintext highlighter-rouge">map</code> or a <code class="language-plaintext highlighter-rouge">filter</code>, or some terrible combination of both or neither.</p>
<h2 id="my-point-reduce-is-too-low-level">My Point: reduce is too low-level</h2>
<p>A for-loop can do anything and <code class="language-plaintext highlighter-rouge">reduce</code> has the same lack of clarity and intent… because it is a
<a href="https://hexdocs.pm/elixir/Enum.html#reduce/3-reduce-as-a-building-block">building block</a>:</p>
<blockquote>
<p>Reduce (sometimes called fold) is a basic building block in functional
programming. Almost all of the functions in the Enum module can be
implemented on top of reduce. <sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup></p>
</blockquote>
<p>But a pile of building materials is not a house.</p>
<p>A <code class="language-plaintext highlighter-rouge">reduce</code>, given an anonymous function, is similar to tricky, inlined, and uncommented code.
It’s a missed opportunity to extract that logic to a function and give it a name.</p>
<p>But first…</p>
<h2 id="alternatives">Alternatives</h2>
<p>That <code class="language-plaintext highlighter-rouge">reduce</code> is <em>probably</em> not a special case, and it’s <em>probably</em> one of these:</p>
<ul>
<li><a href="https://hexdocs.pm/elixir/Enum.html#map/2">map</a></li>
<li><a href="https://hexdocs.pm/elixir/Enum.html#filter/2">filter</a></li>
<li><a href="https://hexdocs.pm/elixir/Enum.html#any?/2">any?</a>, <a href="https://hexdocs.pm/elixir/Enum.html#all?/2">all?</a></li>
<li><a href="https://hexdocs.pm/elixir/Enum.html#count/2">count?</a></li>
<li><a href="https://hexdocs.pm/elixir/Enum.html#min/3">min</a>, <a href="https://hexdocs.pm/elixir/Enum.html#max/3">max</a></li>
<li><a href="https://hexdocs.pm/elixir/Enum.html#sum/1">sum</a>, <a href="https://hexdocs.pm/elixir/Enum.html#product/1">product</a></li>
<li><a href="https://hexdocs.pm/elixir/Enum.html#group_by/3">group_by</a>, <a href="https://hexdocs.pm/elixir/Enum.html#frequencies/1">frequencies</a></li>
</ul>
<p>If you use a functional language <sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">2</a></sup>, refer to its excellent “enumerable” module:</p>
<ul>
<li><a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array">JavaScript</a></li>
<li><a href="https://ruby-doc.org/core/Enumerable.html">Ruby</a></li>
<li><a href="https://hexdocs.pm/elixir/Enum.html">Elixir</a></li>
<li><a href="https://package.elm-lang.org/packages/elm/core/latest/List">Elm</a></li>
</ul>
<h2 id="whats-next">What’s Next?</h2>
<p>In my next post <sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote" rel="footnote">3</a></sup>, I’m going to cover <code class="language-plaintext highlighter-rouge">reduce</code> anti-patterns.</p>
<hr />
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>This is a typical homework assignment. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>If missing, email me your own language’s link <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>Coming soon… <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
How to Export Excel Formulas to CSV2023-04-12T00:00:00+00:00https://blog.jpalardy.com/posts/how-to-export-excel-formulas-to-csv<p>Spreadsheets aren’t my favorite things, but I’ve spent a decent amount of time
using them. A few weeks ago, I stumbled on a blind spot … is there a way to
<em>EXPORT</em> formulas?</p>
<h2 id="looking-at-formulas">Looking At Formulas</h2>
<p>You might already be familiar with this trick: <code class="language-plaintext highlighter-rouge">ctrl</code> <code class="language-plaintext highlighter-rouge">backtick</code> <sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup> will toggle cells
between showing their values and their formulas:</p>
<p><a href="/assets/exporting-formulas/toggling.png"><img src="/assets/exporting-formulas/toggling.png" alt="toggling formulas back and forth" /></a></p>
<p><em>⚠️ This works in Google Spreadsheet as well ⚠️</em></p>
<p>This can really help making sense of things – especially, if you’re trying to eyeball a problem.</p>
<p>What’s the alternative?<br />
Clicking on each cell and looking, <em>carefully</em>, at the formula bar?! 🤔</p>
<h2 id="exporting-formulas">Exporting Formulas?</h2>
<p>If you’re <em>REALLY</em> trying to audit a spreadsheet and need to resort to
scripting, or other external tools – it would be really useful if you could
export not the cell values, but their formulas instead.</p>
<p>Here’s the trick:</p>
<ul>
<li>toggle to view the formulas (as described above)</li>
<li><em>NOW</em>, export to CSV</li>
</ul>
<p>It was so easy … I didn’t think about it. 😄</p>
<p>I found this gem on <a href="https://superuser.com/questions/466419/how-to-save-xls-x-as-csv-file-so-that-formulas-show-as-text">superuser.com</a>.</p>
<hr />
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p><a href="https://en.wikipedia.org/wiki/Backtick">backtick</a>? Probably to left of the <code class="language-plaintext highlighter-rouge">1</code> key on your keyboard, above the <code class="language-plaintext highlighter-rouge">tab</code> key <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
INNER JOIN Files on the Command-Line2023-03-08T00:00:00+00:00https://blog.jpalardy.com/posts/inner-join-files-on-the-command-line<p>A lot of people don’t know that <code class="language-plaintext highlighter-rouge">join</code> exists, or what it does 🤔</p>
<p>I mean the <code class="language-plaintext highlighter-rouge">join</code> command that’s already on your computer and ready to go:</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text"># BSD (macOS)
> join
usage: join [-a fileno | -v fileno ] [-e string] [-1 field] [-2 field]
[-o list] [-t char] file1 file2
# or
# GNU (linux)
> join
join: missing operand
Try 'join --help' for more information.</code></pre></figure>
<h2 id="what-is-join">What is join?</h2>
<p>Believe it or not, <code class="language-plaintext highlighter-rouge">join</code> is part of
<a href="https://www.gnu.org/software/coreutils/manual/html_node/index.html">coreutils</a>,
which – as its name implies – is pretty central to UNIX-like systems.</p>
<p>It sits there, with its better known siblings <code class="language-plaintext highlighter-rouge">cut</code> and <code class="language-plaintext highlighter-rouge">paste</code></p>
<p><a href="/assets/inner-join-cli/sample-cores.png"><img src="/assets/inner-join-cli/sample-cores.png" alt="together: cut, paste and join" /></a></p>
<p>What does <code class="language-plaintext highlighter-rouge">join</code> do?</p>
<p><code class="language-plaintext highlighter-rouge">join</code> does to files what SQL <code class="language-plaintext highlighter-rouge">INNER JOIN</code> does to tables.</p>
<h2 id="join-examples">join Examples</h2>
<p>Imagine two files:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">cat </span>english.txt
1 one
2 two
3 three
4 four
<span class="o">></span> <span class="nb">cat </span>spanish.txt
1 uno
2 dos
4 cuatro</code></pre></figure>
<p>By default, the first column (numbers here, but it doesn’t have to be) is the join key:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">join </span>english.txt spanish.txt
1 one uno
2 two dos
4 four cuatro</code></pre></figure>
<p>Notice that <code class="language-plaintext highlighter-rouge">3</code> is missing from <code class="language-plaintext highlighter-rouge">spanish.txt</code>. It works the way <code class="language-plaintext highlighter-rouge">INNER JOIN</code> works.</p>
<p>Same, flipping the files: (<code class="language-plaintext highlighter-rouge">3</code> still missing)</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">join </span>spanish.txt english.txt
1 uno one
2 dos two
4 cuatro four</code></pre></figure>
<p>Comments:</p>
<ul>
<li>files must be sorted ~ that’s important!</li>
<li>fields are separated by blanks – but field separator can be specified with <code class="language-plaintext highlighter-rouge">-t</code></li>
<li>there are other options, check the <a href="https://linux.die.net/man/1/join">man page</a> as needed</li>
</ul>
<p>You can even coerce <code class="language-plaintext highlighter-rouge">join</code> into doing:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">LEFT JOIN</code></li>
<li><code class="language-plaintext highlighter-rouge">RIGHT JOIN</code></li>
<li><code class="language-plaintext highlighter-rouge">UNION</code></li>
<li><code class="language-plaintext highlighter-rouge">DIFFERENCE</code></li>
</ul>
<p>check the <a href="https://www.gnu.org/software/coreutils/manual/html_node/Paired-and-unpaired-lines.html">online documentation</a> for examples.</p>
<h2 id="is-join-useful">Is join Useful?</h2>
<p>I recently had to join 2 large <a href="https://en.wikipedia.org/wiki/Tab-separated_values">.tsv</a> files. My first instinct was to reach for
<a href="https://blog.jpalardy.com/posts/why-learn-awk/">AWK</a> … but I remembered <code class="language-plaintext highlighter-rouge">join</code> and it did <strong>exactly</strong> what I needed.</p>
<p>Knowing that <code class="language-plaintext highlighter-rouge">join</code> exists is 80% of its value 😄</p>
Livebook: Getting Started2023-02-13T00:00:00+00:00https://blog.jpalardy.com/posts/livebook-getting-started<h2 id="what-is-livebook">What is Livebook?</h2>
<p>The easiest way to explain <a href="https://livebook.dev/">Livebook</a> is to say it’s <a href="https://jupyter.org/">Jupyter notebook</a> for <a href="https://elixir-lang.org/">Elixir</a>.</p>
<p>Another way? What if Word and Excel had a baby?</p>
<ul>
<li>combine text</li>
<li>and formula (or code)</li>
<li>see the results</li>
<li>in a cohesive document</li>
</ul>
<p><a href="/assets/livebook-getting-started/small-livebook-annotated.png"><img src="/assets/livebook-getting-started/small-livebook-annotated.png" alt="screenshot of a livebook" /></a></p>
<h2 id="why-livebook">Why Livebook?</h2>
<p>As an Elixir developer, I didn’t really understand where Livebook was supposed to fit. After all, I already knew how to start a new project, or launch iex (Elixir’s REPL).</p>
<p>What I found:</p>
<ul>
<li>it’s so easy to get started – more below – for small to medium experiments<br />
(as far as <a href="https://adventofcode.com/">The Advent of Code</a>, if you wanted – <a href="https://www.youtube.com/playlist?list=PLNP8vc86_-SOV1ZEvX_q9BLYWL586zWnF">watch José Valim attempt 2021’s</a>)</li>
<li>it’s more approachable for your colleagues – <em>especially</em> the less technical ones<br />
(comparable to Excel, if they feel comfortable with that)</li>
</ul>
<p>Maybe it fits a middle ground between throw-away script and new GitHub project 🤔</p>
<p>Last, but not least, with syntax coloring, code completion, and code formatting, it’s a surprisingly good coding environment:</p>
<p><a href="/assets/livebook-getting-started/features.png"><img src="/assets/livebook-getting-started/features.png" alt="features mentioned above, visually" /></a></p>
<p>Let’s get you started.</p>
<h2 id="how-to-install">How to Install?</h2>
<ul>
<li>google: <a href="https://www.google.com/search?q=livebook">livebook</a></li>
<li>click on the first page</li>
<li>click “Install Livebook”</li>
<li>download the app</li>
<li>install like any other app</li>
</ul>
<p><a href="/assets/livebook-getting-started/install-livebook-summary.png"><img src="/assets/livebook-getting-started/install-livebook-summary.png" alt="steps above, visually" /></a></p>
<p>What did you just install?</p>
<ul>
<li>a very recent version of Erlang and Elixir</li>
<li>Livebook, the “app”</li>
</ul>
<p>Livebook runs in your browser – but the supporting code is what you installed.</p>
<p>There are <a href="https://github.com/livebook-dev/livebook#installation">other ways to install</a>, but I wouldn’t recommend them unless you have very specific needs.</p>
<h2 id="how-not-to-run-livebook">How NOT to Run Livebook…</h2>
<p>Of course, you <em>COULD</em> just double-click the Livebook icon. Personally, I found that confusing (especially when I was getting started)</p>
<p><a href="/assets/livebook-getting-started/default-start.png"><img src="/assets/livebook-getting-started/default-start.png" alt="what Livebook looks like, when you first open it" /></a></p>
<p>What is all this? How do I start to code?<br />
(let’s come back to this page <a href="#wheres-the-documentation">later</a>)</p>
<h2 id="how-to-run-livebook-what-i-recommend-instead">How to Run Livebook: What I Recommend Instead</h2>
<p>If you have some Livebook files already:</p>
<ul>
<li>open that folder</li>
<li>double-click a Livebook file</li>
</ul>
<p><a href="/assets/livebook-getting-started/folder-with-livebooks.png"><img src="/assets/livebook-getting-started/folder-with-livebooks.png" alt="a folder with some Livebook files" /></a></p>
<p>If you don’t have Livebook files ready to go: know that an empty file is a completely valid Livebook file! 😄</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">touch </span>first.livebook
<span class="o">></span> open first.livebook <span class="c"># open, on macos, is like double-clicking...</span></code></pre></figure>
<p><a href="/assets/livebook-getting-started/empty-notebook.png"><img src="/assets/livebook-getting-started/empty-notebook.png" alt="what an empty Livebook looks like" /></a></p>
<p>Starting from a file has a few advantages:</p>
<ul>
<li>less magic: you know where you files are</li>
<li>Livebook starts in the right directory – no need to navigate your file system from the browser…</li>
<li>your file will start to autosave as you modify it</li>
</ul>
<h2 id="what-now">What Now?</h2>
<p>To get an idea, you can watch the “Announcing Livebook” video</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/RKvqc-UEe34" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen=""></iframe>
<p><br />
Another perspective is the one I linked above, solving Advent of Code with Elixir:</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/mDxJjqx5-ns" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen=""></iframe>
<h2 id="wheres-the-documentation">Where’s the Documentation?</h2>
<p>When I started out, I was really confused: where’s the documentation?!</p>
<p>The <a href="https://livebook.dev/">Livebook website</a> feels like a sales pitch … no documentation? 🤔</p>
<p>Fortunately (or unfortunately), the Livebook documentation is found … inside Livebook. (and if it’s somewhere else, I haven’t found it)</p>
<p>Let’s loop back to what you see if you just open Livebook (without a file).</p>
<p><a href="/assets/livebook-getting-started/default-start.png"><img src="/assets/livebook-getting-started/default-start.png" alt="what Livebook looks like, when you first open it" /></a></p>
<p>If you click “Learn” on the left, or “LEARN – see all >” in the middle, you will end up on this page:</p>
<p><a href="/assets/livebook-getting-started/learn.png"><img src="/assets/livebook-getting-started/learn.png" alt="the documentation page is found" /></a></p>
<p>If you click on one of the cards, it will open a Livebook to explain some aspect of Livebook (insert <a href="https://www.imdb.com/title/tt1375666/">Inception</a> reference)</p>
<p>I strongly recommend the one called “Welcome to Livebook” (click the blue button, seen in the screenshot above). That Livebook does a good job getting you
to 80-90% of what you need to know.</p>
<h2 id="conclusions">Conclusions</h2>
<p>Using Livebook is a real pleasure, and I regret waiting so long to install and play with it.</p>
<p>Even using it a little bit allows you to really “get it”.</p>
<p>Whether this is your way to get started with Elixir, or you’re a veteran, Livebook has something to offer 🔥</p>
<p>(if not, it’s also easy to uninstall 😜)</p>
The Best Books I Read in 20222022-12-25T00:00:00+00:00https://blog.jpalardy.com/posts/best-books-i-read-2022<p>I have read many books in 2022; let’s forget most and talk about the good ones.</p>
<p>There is no particular order, but I broke down my recommendations by
categories: <a href="#technical">technical</a>, <a href="#non-fiction">non-fiction</a> and
<a href="#fiction">fiction</a>.</p>
<p>This is a yearly tradition! You can read my book reviews from previous years:<br />
<a href="https://blog.jpalardy.com/posts/best-books-i-read-2015/">2015</a> <a href="https://blog.jpalardy.com/posts/best-books-i-read-2016/">2016</a> <a href="https://blog.jpalardy.com/posts/best-books-i-read-2017/">2017</a> <a href="https://blog.jpalardy.com/posts/best-books-i-read-2018/">2018</a> <a href="https://blog.jpalardy.com/posts/best-books-i-read-2019/">2019</a> <a href="https://blog.jpalardy.com/posts/best-books-i-read-2020/">2020</a> and <a href="https://blog.jpalardy.com/posts/best-books-i-read-2021/">2021</a>.</p>
<hr />
<h2 id="technical">Technical</h2>
<h3 id="domain-modeling-made-functional">Domain Modeling Made Functional</h3>
<p><a href="https://www.amazon.com/dp/1680502549/"><img class="book-cover" src="/assets/best-books-2022/1680502549.jpg" alt="Domain Modeling Made Functional" /></a></p>
<p>If you wanted to learn how to do object-oriented design, there are many books
out there (<em>though none that I can recommend; that’s a different topic</em>).
However, when it comes to functional programmning, where do you even start?</p>
<p>This book exceeded my expectations:</p>
<ul>
<li>it’s easy to read</li>
<li>it’s practical</li>
<li>it covers functional architecture</li>
<li>it dives deep on types</li>
</ul>
<p>I had already heard about the book before. Then I stumbled on this fantastic video from the author:</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/Up7LcbGZFuo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<p><br />
I love this slide:</p>
<p><a href="/assets/best-books-2022/things-wrong.png"><img src="/assets/best-books-2022/things-wrong.png" alt="picture from video: how many things are wrong with this design" /></a></p>
<p>(spoiler: so many things…)</p>
<p>I’ve heard some people complain about F#, but I didn’t think it was a big problem.</p>
<p>I only wish Scott Wlaschin would write more books, so that I could read those too…</p>
<ul>
<li>book: <a href="https://www.amazon.com/dp/1680502549/">amazon</a></li>
<li>book: <a href="https://pragprog.com/titles/swdddf/domain-modeling-made-functional/">pragprog</a></li>
<li>author: <a href="https://fsharpforfunandprofit.com/books/">website</a></li>
</ul>
<!-- ------------------------------------------------- -->
<h3 id="property-based-testing-with-proper-erlang-and-elixir">Property-Based Testing with PropEr, Erlang, and Elixir</h3>
<p><a href="https://www.amazon.com/dp/1680506218/"><img class="book-cover" src="/assets/best-books-2022/1680506218.jpg" alt="Property-Based Testing with PropEr, Erlang, and Elixir" /></a></p>
<p>The premise of property-based testing is intriguing: writing <em>less</em> tests but getting <em>better</em> coverage.</p>
<p>It forces you to think about your functions; what are the properties of a sorting function?</p>
<ul>
<li>obviously: output is sorted (need another function to check <code class="language-plaintext highlighter-rouge">is_sorted</code>?)</li>
<li>output list same length as input list</li>
<li>output contains <em>only</em> elements from the input</li>
<li>duplicates sorting is <a href="https://en.wikipedia.org/wiki/Category:Stable_sorts">stable</a>? (depending on your requirements)</li>
</ul>
<p>Currently, it’s the only book that covers the topic; your alternatives might be blog
posts and documentation pages. And though this book covers Erlang/Elixir, its
lessons are broadly applicable to other languages and frameworks.</p>
<p>At a high level, it generates random inputs – albeit meaningful ones, this
isn’t <a href="https://en.wikipedia.org/wiki/Fuzzing">fuzzing</a> – and explores the
problem space automatically. Meanwhile, example-based testing often depends on
hardcoded values, which makes me wonder “does this value mean something?”. And
if it doesn’t, it might be better to generate one (e.g. <a href="https://fakerjs.dev/">faker</a>).</p>
<p>Property-based testing is perfect for functional code, but it’s useful for
stateful properties too (it’s covered later in the book). Imagine generating
random “scenarios” from list of actions your users might perform. Check out
David Nolen’s <a href="https://youtu.be/qDGTxyIrKJY?t=1914">The Power of Toys</a> to get a
better idea.</p>
<p>In the end, it’s not right for every project and you need to consider your
team/audience. But even when I don’t use property-based testing, it’s changed
the way I test: I write more parametrized tests and try to randomize all the
things that aren’t supposed to matter.</p>
<ul>
<li>book: <a href="https://www.amazon.com/dp/1680506218/">amazon</a></li>
<li>book: <a href="https://pragprog.com/titles/fhproper/property-based-testing-with-proper-erlang-and-elixir/">pragprog</a></li>
<li>author: <a href="https://ferd.ca/">website</a></li>
</ul>
<hr />
<h2 id="non-fiction">Non-Fiction</h2>
<h3 id="ducks">Ducks</h3>
<p><a href="https://www.amazon.com/dp/1770462899/"><img class="book-cover" src="/assets/best-books-2022/1770462899.jpg" alt="Ducks" /></a></p>
<p>I didn’t know much about this book before I started reading it. I already knew
the author from her previous work, and the early recommendations closed
the deal.</p>
<p>This is a <em>weird</em> book to describe: it’s a graphic novel, it’s autobiographical
and has a flavor of <a href="https://en.wikipedia.org/wiki/Slice_of_life">slice of life</a>.
It is as the alternate title says: “Two Years in the Oil Sands”.</p>
<p>It is at times funny … but it’s <em>mostly</em> sad. It touches a variety of
societal problems in a deeply personal way. It is a kind of documentary,
without an <em>obvious</em> agenda. It is more <strong>demanding</strong>: it shows you and asks you to
reach your own conclusions.</p>
<p>It takes you on a journey. At some point, towards the end, I felt <em>different</em>.</p>
<p>Thank you for this book, Kate.</p>
<p>(<em>Disclaimer: without going into details, bad things happen in Alberta.</em>)</p>
<ul>
<li>book: <a href="https://www.amazon.com/dp/1770462899/">amazon</a></li>
<li>book: <a href="https://en.wikipedia.org/wiki/Ducks:_Two_Years_in_the_Oil_Sands">wikipedia</a></li>
<li>book: <a href="https://drawnandquarterly.com/books/ducks/">website</a></li>
<li>author: <a href="https://en.wikipedia.org/wiki/Kate_Beaton">wikipedia</a></li>
</ul>
<!-- ------------------------------------------------- -->
<h3 id="soft-city">Soft City</h3>
<p><a href="https://www.amazon.com/dp/1642830186/"><img class="book-cover" src="/assets/best-books-2022/1642830186.jpg" alt="Soft City" /></a></p>
<p>I found this book on <a href="https://blog.jonnew.com/posts/books-i-read-2020">Jonathan New’s blog</a>.
It sounded like something I would enjoy reading. (Thanks Jon!)</p>
<p><strong>Soft City</strong> is filled with beautiful images; it’s pornography for people who
love urbanism. Everyday, for the month I spent reading it, it
was something that I looked forward to.</p>
<p>Depending on where you live, what is described in this book is normal … or pure fantasy. It covers a
lot of the same ground that <a href="https://www.youtube.com/@NotJustBikes">Not Just Bikes</a> does.</p>
<p>A “soft” city is:</p>
<ul>
<li>human scale</li>
<li>comfortable</li>
<li>walkable</li>
<li>multifunctional</li>
</ul>
<p>The opposite is probably … a suburb: a place where the next thing you want to
do starts with a car ride. If you’ve been to Europe or Japan, you already know
that there are other ways to live.</p>
<ul>
<li>book: <a href="https://www.amazon.com/dp/1642830186/">amazon</a></li>
<li>book: <a href="https://islandpress.org/books/soft-city">website</a></li>
<li>author: <a href="https://gehlpeople.com/people/david-sim/">website</a></li>
</ul>
<!-- ------------------------------------------------- -->
<h3 id="the-effective-manager">The Effective Manager</h3>
<p><a href="https://www.amazon.com/dp/1119244609/"><img class="book-cover" src="/assets/best-books-2022/1119244609.jpg" alt="The Effective Manager" /></a></p>
<p>No, I am not a manager.</p>
<p>But I stumbled on a <a href="https://news.ycombinator.com/item?id=32790064">hacker news thread</a> about “Which books do
you consider real gems in your field of work/study?” The reviews were glowing.</p>
<blockquote>
<p>All the things that no one says or tells about management and communication.</p>
</blockquote>
<p>Right in the introduction, I was struck by:</p>
<blockquote>
<p>Hundreds, if not thousands, of managers describe their “training” this way: I
got promoted, and they didn’t tell me anything about what I was supposed to
do or how I was supposed to do it.</p>
</blockquote>
<p>I know many people who went through this exact experience; with various results…</p>
<p>The book asks “What is the definition of a good manager?” And tells you all the
wrong answers first 😄 (no, I didn’t guess correctly either…)</p>
<p>You’ll be guided through the 2 <em>real</em> responsibilities of managers. Then you’ll
be explained the 4 critical behaviors and, specifically, <em>HOW</em> to apply them to
your team.</p>
<p>There’s more good news:</p>
<ul>
<li>it’s a thin book, around 200 pages</li>
<li>it’s “front loaded”, the book starts with the most important content</li>
<li>it’s no-nonsense; straight talk for people who want it</li>
</ul>
<p>Even if you’re not a manager, it gives you better soft skills and better tools
to cooperate with your teammates.</p>
<ul>
<li>book: <a href="https://www.amazon.com/dp/1119244609/">amazon</a></li>
<li>book: <a href="https://www.wiley.com/en-us/The+Effective+Manager-p-9781119244608">website</a></li>
<li>author: <a href="https://www.manager-tools.com/">website</a></li>
</ul>
<!-- ------------------------------------------------- -->
<h3 id="the-end-of-the-world-is-just-the-beginning">The End of the World Is Just the Beginning</h3>
<p><a href="https://www.amazon.com/dp/006323047X/"><img class="book-cover" src="/assets/best-books-2022/006323047X.jpg" alt="The End of the World Is Just the Beginning" /></a></p>
<p>(Thanks to Grof for this recommendation!)</p>
<p>In many ways, this book <em>defined</em> my 2022. I devoured it; then I read Zeihan’s 3
other books. I subscribed to his YouTube channel. I couldn’t get enough.</p>
<p>This book so changed how I think about the world that I can barely discuss the news
without bringing up Zeihan or his books.</p>
<p>As a child of an unprecedented era of prosperity (and relative peace), I never
<em>really</em> thought that not all countries have everything they need to succeed
(or even survive).</p>
<p>This book gave me enough context to understand geopolitics:</p>
<ul>
<li>where countries are (<em>it matters!</em>)</li>
<li>what good and bad things they have going for them</li>
<li>how these things affect their destiny, their options, and their relationship with other countries</li>
</ul>
<p>So many things suddenly made sense!</p>
<p>The writing is fresh and funny. I think his last book is the best of the bunch.
If you’re not convinced, try his YouTube channel first.</p>
<ul>
<li>book: <a href="https://www.amazon.com/dp/006323047X/">amazon</a></li>
<li>book: <a href="https://zeihan.com/end-of-the-world/">website</a></li>
<li>author: <a href="https://en.wikipedia.org/wiki/Peter_Zeihan">wikipedia</a></li>
<li>author: <a href="https://www.youtube.com/c/ZeihanonGeopolitics">youtube</a></li>
</ul>
<hr />
<h2 id="fiction">Fiction</h2>
<h3 id="project-hail-mary">Project Hail Mary</h3>
<p><a href="https://www.amazon.com/dp/B08GB58KD5/"><img class="book-cover" src="/assets/best-books-2022/B08GB58KD5.jpg" alt="Project Hail Mary" /></a></p>
<p>This is … a flawed book. 😬</p>
<p>I found the characters <em>simple</em>. I found the dialogues unrealistic. I found all
kind of “side trips” that an editor might have wanted to cut.</p>
<p>Also, in many ways, it’s a repeat of <a href="https://www.amazon.com/dp/0553418025/">The Martian</a>:</p>
<ul>
<li>single guy stranded in space</li>
<li>going to <a href="https://www.google.com/search?q=science+the+shit+out+of+this&tbm=isch">science the shit out of</a> every problems</li>
</ul>
<p>And I have read <a href="https://www.amazon.com/dp/B0721NKNHR/">Artemis</a> and thought it was nothing special.</p>
<p>But …….. <strong>despite</strong> all this, it passes the most important test: I wanted to
keep reading and find out what was going to happen next.</p>
<p>Now that I’ve adjusted your expectations, I think you’ll be able to enjoy this book too.</p>
<ul>
<li>book: <a href="https://www.amazon.com/dp/B08GB58KD5/">amazon</a></li>
<li>book: <a href="https://en.wikipedia.org/wiki/Project_Hail_Mary">wikipedia</a></li>
<li>author: <a href="https://en.wikipedia.org/wiki/Andy_Weir">wikipedia</a></li>
<li>author: <a href="https://andyweirauthor.com/">website</a></li>
</ul>
<!-- ------------------------------------------------- -->
<h3 id="a-memory-called-empire">A Memory Called Empire</h3>
<p><a href="https://www.amazon.com/dp/B07PWHQW1F/"><img class="book-cover" src="/assets/best-books-2022/B07PWHQW1F.jpg" alt="A Memory Called Empire" /></a></p>
<p>This book kept popping up on Audible and I became curious. But some of the
criticism was <em>harsh</em> and it’s been a challenge to decide which reviews to take
seriously and which to ignore.</p>
<p>In the end, it didn’t hurt that it won the 2020 <a href="https://en.wikipedia.org/wiki/Hugo_Award_for_Best_Novel">Hugo award</a> for best novel.</p>
<p>Who knew that I would find a political thriller <em>delightful</em>? I loved listening to
intelligent conversations: everything that was said, and everything that wasn’t…</p>
<p>It starts as a murder mystery (kind of) but goes in different directions.
You’re the outsider and there’s a lot to learn: the city, the empire, the
players and the stakes. Things happen fast and there’s always something else coming.</p>
<p>I found the ending bittersweet. And I found the sequel, <a href="https://www.amazon.com/dp/B082WK914S/">A Desolation Called Peace</a>, good
but … smaller too. It’s hard to explain. (maybe it closed too many story lines? 🤔)</p>
<p>This might not be for everyone, but it worked for me.</p>
<ul>
<li>book: <a href="https://www.amazon.com/dp/B07PWHQW1F/">amazon</a></li>
<li>book: <a href="https://en.wikipedia.org/wiki/A_Memory_Called_Empire">wikipedia</a></li>
<li>author: <a href="https://en.wikipedia.org/wiki/Arkady_Martine">wikipedia</a></li>
<li>author: <a href="https://www.arkadymartine.net/">website</a></li>
</ul>
<hr />
<h2 id="closing-thoughts">Closing Thoughts</h2>
<p>If you have liked this blog post, please write your own. I would love to read your book
reviews and recommendations.</p>
ding: Audio Feedback for Exit Codes2022-10-28T00:00:00+00:00https://blog.jpalardy.com/posts/ding<p>Do you babysit long-running applications?</p>
<p>You <em>could</em> display a desktop notification, but it’s even nicer (and mouseless) to know how things went without looking.</p>
<h2 id="my-use-case">My use case</h2>
<p>I run <code class="language-plaintext highlighter-rouge">mix test</code> in another tmux window; it’s “offscreen” for all intent and purposes.</p>
<p>I use <a href="https://github.com/eradman/entr">entr</a> to trigger <code class="language-plaintext highlighter-rouge">mix test</code>: my tests will run when files change.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> git ls-files | entr mix <span class="nb">test</span>
<span class="c"># if `git ls-files` is too large, I use a filter</span>
<span class="o">></span> git ls-files | rg some-filter | entr mix <span class="nb">test</span></code></pre></figure>
<p>I want to know: did the tests pass?</p>
<h2 id="ding">Ding</h2>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> ding successful-command <span class="c"># e.g. ding echo hello</span>
<span class="c"># plays good sound</span>
<span class="o">></span> ding UNsuccessful-command <span class="c"># e.g. ding rm doesnt-exist.txt</span>
<span class="c"># plays sad sound</span>
<span class="o">></span> ding
<span class="c"># plays good sound</span></code></pre></figure>
<p>That last one is more useful than it looks: if you <em>already</em> started something
and realize it’s going to take a while; type <code class="language-plaintext highlighter-rouge">ding</code> and it will execute when
the first command finishes.</p>
<p>All together now:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> git ls-files | entr ding mix <span class="nb">test</span></code></pre></figure>
<h2 id="simple-enough">Simple Enough</h2>
<p><a href="https://github.com/jpalardy/dotfiles/blob/main/bin/ding">ding</a> is a simple script:</p>
<ul>
<li>it runs your command</li>
<li>grabs the exit code</li>
<li>plays a sound for success (exit code == 0)</li>
<li>plays a different sound for failure (exit code != 0)</li>
</ul>
<p>Here’s the script as of the writing of this post:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c">#!/bin/bash</span>
play_sound<span class="o">()</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">!</span> <span class="nb">command</span> <span class="nt">-v</span> afplay &>/dev/null<span class="p">;</span> <span class="k">then
</span><span class="nb">echo</span> <span class="o">></span>&2 <span class="s2">"ding: cannot play sound!"</span>
<span class="k">return
fi
if</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span> <span class="nt">-eq</span> 0 <span class="o">]</span><span class="p">;</span> <span class="k">then
</span>afplay <span class="s2">"/System/Library/Sounds/Purr.aiff"</span>
<span class="k">return
fi
</span>afplay <span class="s2">"/System/Library/Sounds/Basso.aiff"</span>
<span class="o">}</span>
<span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
<span class="nv">lastexit</span><span class="o">=</span><span class="nv">$?</span>
play_sound <span class="s2">"</span><span class="nv">$lastexit</span><span class="s2">"</span> &
<span class="nb">exit</span> <span class="s2">"</span><span class="nv">$lastexit</span><span class="s2">"</span></code></pre></figure>
<p>Both <code class="language-plaintext highlighter-rouge">afplay</code> and the <code class="language-plaintext highlighter-rouge">.aiff</code> files assume a macOS environment.</p>
<p>But there’s so little logic here, it’s easy to adapt to your own needs.</p>
Understanding Overlap: in 1 or 2 dimensions2022-10-09T00:00:00+00:00https://blog.jpalardy.com/posts/understanding-overlap<h2 id="tldr">tl;dr</h2>
<p>The answer is on <a href="https://stackoverflow.com/questions/306316/determine-if-two-rectangles-overlap-each-other">stackoverflow</a>:</p>
<div class="language-cpp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="p">(</span><span class="n">RectA</span><span class="p">.</span><span class="n">Left</span> <span class="o"><</span> <span class="n">RectB</span><span class="p">.</span><span class="n">Right</span> <span class="o">&&</span> <span class="n">RectA</span><span class="p">.</span><span class="n">Right</span> <span class="o">></span> <span class="n">RectB</span><span class="p">.</span><span class="n">Left</span> <span class="o">&&</span>
<span class="n">RectA</span><span class="p">.</span><span class="n">Top</span> <span class="o">></span> <span class="n">RectB</span><span class="p">.</span><span class="n">Bottom</span> <span class="o">&&</span> <span class="n">RectA</span><span class="p">.</span><span class="n">Bottom</span> <span class="o"><</span> <span class="n">RectB</span><span class="p">.</span><span class="n">Top</span><span class="p">)</span>
</code></pre></div></div>
<p>which is correct but not especially obvious.</p>
<p>A <em>LOT</em> of people think the stackoverflow answer is wrong and try to “fix” it 🤣</p>
<h2 id="step-by-step-in-one-dimension">Step by step: in one dimension</h2>
<p>Let’s simplify the problem:</p>
<ul>
<li>let’s focus only on one dimension – the x axis</li>
<li>let’s use rectangles – just ignore their heights<br />
<em>(it’s easier to visualize than line segments)</em></li>
</ul>
<p>How can 2 rectangles overlap in one dimension?</p>
<p><a href="/assets/overlap/overlap-examples.png"><img src="/assets/overlap/overlap-examples.png" alt="examples of overlap with rectangles" /></a></p>
<p>There seems to be a lot of cases 🤔</p>
<ul>
<li>left of B could be inside A</li>
<li>(or) right of B could be inside A</li>
</ul>
<p><a href="/assets/overlap/simple-cases.png"><img src="/assets/overlap/simple-cases.png" alt="show partial overlap" /></a></p>
<p>But what about this?</p>
<p><a href="/assets/overlap/inside-case.png"><img src="/assets/overlap/inside-case.png" alt="show complete overlap" /></a></p>
<p>Both sides of B are inside A. <strong>But neither of A’s sides are inside B.</strong> We need symmetry…</p>
<p>What are we left with?</p>
<ul>
<li>left of B could be inside A</li>
<li>(or) right of B could be inside A</li>
<li>(or) left of A could be inside B</li>
<li>(or) right of A could be inside B</li>
</ul>
<p>Simple? Did we cover all the cases? (no! read until the end)</p>
<p>Personally, that’s when I turned to stackoverflow.</p>
<h2 id="a-different-perspective">A different perspective</h2>
<p>If overlap is hard to pin down, let’s try to define the opposite: what does it look like when rectangles do NOT overlap?</p>
<p><a href="/assets/overlap/no-overlap.png"><img src="/assets/overlap/no-overlap.png" alt="revisit two rectangles that do not overlap" /></a></p>
<p>in English:</p>
<ul>
<li>A is to the left of B</li>
<li>(or) A is to the right of B</li>
</ul>
<p>let’s turn this to pseudo-code:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">no_overlap</span> <span class="o">=</span> <span class="nx">a</span><span class="p">.</span><span class="nx">right</span> <span class="o"><</span> <span class="nx">b</span><span class="p">.</span><span class="nx">left</span> <span class="o">||</span> <span class="nx">a</span><span class="p">.</span><span class="nx">left</span> <span class="o">></span> <span class="nx">b</span><span class="p">.</span><span class="nx">right</span>
<span class="c1">// in other words, overlap is the inverse</span>
<span class="nx">overlap</span> <span class="o">=</span> <span class="o">!</span><span class="p">(</span><span class="nx">a</span><span class="p">.</span><span class="nx">right</span> <span class="o"><</span> <span class="nx">b</span><span class="p">.</span><span class="nx">left</span> <span class="o">||</span> <span class="nx">a</span><span class="p">.</span><span class="nx">left</span> <span class="o">></span> <span class="nx">b</span><span class="p">.</span><span class="nx">right</span><span class="p">)</span>
</code></pre></div></div>
<p><a href="https://en.wikipedia.org/wiki/De_Morgan%27s_laws">De Morgan’s Laws</a> say that:</p>
<ul>
<li>not (A or B) = (not A) and (not B)</li>
<li>not (A and B) = (not A) or (not B)</li>
</ul>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// unchanged, from above</span>
<span class="nx">overlap</span> <span class="o">=</span> <span class="o">!</span><span class="p">(</span><span class="nx">a</span><span class="p">.</span><span class="nx">right</span> <span class="o"><</span> <span class="nx">b</span><span class="p">.</span><span class="nx">left</span> <span class="o">||</span> <span class="nx">a</span><span class="p">.</span><span class="nx">left</span> <span class="o">></span> <span class="nx">b</span><span class="p">.</span><span class="nx">right</span><span class="p">)</span>
<span class="c1">// applying De Morgan: conditions inverted, || becomes &&</span>
<span class="nx">overlap</span> <span class="o">=</span> <span class="o">!</span><span class="p">(</span><span class="nx">a</span><span class="p">.</span><span class="nx">right</span> <span class="o"><</span> <span class="nx">b</span><span class="p">.</span><span class="nx">left</span><span class="p">)</span> <span class="o">&&</span> <span class="o">!</span><span class="p">(</span><span class="nx">a</span><span class="p">.</span><span class="nx">left</span> <span class="o">></span> <span class="nx">b</span><span class="p">.</span><span class="nx">right</span><span class="p">)</span>
<span class="c1">// flipping > and < comparisons</span>
<span class="nx">overlap</span> <span class="o">=</span> <span class="nx">a</span><span class="p">.</span><span class="nx">right</span> <span class="o">></span> <span class="nx">b</span><span class="p">.</span><span class="nx">left</span> <span class="o">&&</span> <span class="nx">a</span><span class="p">.</span><span class="nx">left</span> <span class="o"><</span> <span class="nx">b</span><span class="p">.</span><span class="nx">right</span>
<span class="c1">// reorganizing order of conditions (to match stackoverflow...)</span>
<span class="nx">overlap</span> <span class="o">=</span> <span class="nx">a</span><span class="p">.</span><span class="nx">left</span> <span class="o"><</span> <span class="nx">b</span><span class="p">.</span><span class="nx">right</span> <span class="o">&&</span> <span class="nx">a</span><span class="p">.</span><span class="nx">right</span> <span class="o">></span> <span class="nx">b</span><span class="p">.</span><span class="nx">left</span>
</code></pre></div></div>
<p>Looping back to the beginning, check the first line of answer:</p>
<div class="language-cpp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="p">(</span><span class="n">RectA</span><span class="p">.</span><span class="n">Left</span> <span class="o"><</span> <span class="n">RectB</span><span class="p">.</span><span class="n">Right</span> <span class="o">&&</span> <span class="n">RectA</span><span class="p">.</span><span class="n">Right</span> <span class="o">></span> <span class="n">RectB</span><span class="p">.</span><span class="n">Left</span> <span class="o">&&</span>
<span class="n">RectA</span><span class="p">.</span><span class="n">Top</span> <span class="o">></span> <span class="n">RectB</span><span class="p">.</span><span class="n">Bottom</span> <span class="o">&&</span> <span class="n">RectA</span><span class="p">.</span><span class="n">Bottom</span> <span class="o"><</span> <span class="n">RectB</span><span class="p">.</span><span class="n">Top</span><span class="p">)</span>
</code></pre></div></div>
<h2 id="what-about-2d">What about 2d?</h2>
<p>The same “to-the-left-of” or “to-the-right-of” cases apply. You also need to consider “above” and “below” cases.</p>
<p><em>(top and bottom are relative to … the direction of your y-axis)</em> 😬</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// unchanged, from above</span>
<span class="nx">overlap</span> <span class="o">=</span> <span class="o">!</span><span class="p">(</span><span class="nx">a</span><span class="p">.</span><span class="nx">right</span> <span class="o"><</span> <span class="nx">b</span><span class="p">.</span><span class="nx">left</span> <span class="o">||</span> <span class="nx">a</span><span class="p">.</span><span class="nx">left</span> <span class="o">></span> <span class="nx">b</span><span class="p">.</span><span class="nx">right</span> <span class="o">||</span>
<span class="nx">a</span><span class="p">.</span><span class="nx">bottom</span> <span class="o">></span> <span class="nx">b</span><span class="p">.</span><span class="nx">top</span> <span class="o">||</span> <span class="nx">a</span><span class="p">.</span><span class="nx">top</span> <span class="o"><</span> <span class="nx">b</span><span class="p">.</span><span class="nx">bottom</span><span class="p">)</span>
<span class="c1">// applying De Morgan: conditions inverted, || becomes &&</span>
<span class="nx">overlap</span> <span class="o">=</span> <span class="o">!</span><span class="p">(</span><span class="nx">a</span><span class="p">.</span><span class="nx">right</span> <span class="o"><</span> <span class="nx">b</span><span class="p">.</span><span class="nx">left</span><span class="p">)</span> <span class="o">&&</span> <span class="o">!</span><span class="p">(</span><span class="nx">a</span><span class="p">.</span><span class="nx">left</span> <span class="o">></span> <span class="nx">b</span><span class="p">.</span><span class="nx">right</span><span class="p">)</span> <span class="o">&&</span>
<span class="o">!</span><span class="p">(</span><span class="nx">a</span><span class="p">.</span><span class="nx">bottom</span> <span class="o">></span> <span class="nx">b</span><span class="p">.</span><span class="nx">top</span><span class="p">)</span> <span class="o">&&</span> <span class="o">!</span><span class="p">(</span><span class="nx">a</span><span class="p">.</span><span class="nx">top</span> <span class="o"><</span> <span class="nx">b</span><span class="p">.</span><span class="nx">bottom</span><span class="p">)</span>
<span class="c1">// flipping > and < comparison</span>
<span class="nx">overlap</span> <span class="o">=</span> <span class="nx">a</span><span class="p">.</span><span class="nx">right</span> <span class="o">></span> <span class="nx">b</span><span class="p">.</span><span class="nx">left</span> <span class="o">&&</span> <span class="nx">a</span><span class="p">.</span><span class="nx">left</span> <span class="o"><</span> <span class="nx">b</span><span class="p">.</span><span class="nx">right</span> <span class="o">&&</span>
<span class="nx">a</span><span class="p">.</span><span class="nx">bottom</span> <span class="o"><</span> <span class="nx">b</span><span class="p">.</span><span class="nx">top</span> <span class="o">&&</span> <span class="nx">a</span><span class="p">.</span><span class="nx">top</span> <span class="o">></span> <span class="nx">b</span><span class="p">.</span><span class="nx">bottom</span>
<span class="c1">// reorganizing order of conditions (to match stackoverflow...)</span>
<span class="nx">overlap</span> <span class="o">=</span> <span class="nx">a</span><span class="p">.</span><span class="nx">left</span> <span class="o"><</span> <span class="nx">b</span><span class="p">.</span><span class="nx">right</span> <span class="o">&&</span> <span class="nx">a</span><span class="p">.</span><span class="nx">right</span> <span class="o">></span> <span class="nx">b</span><span class="p">.</span><span class="nx">left</span> <span class="o">&&</span>
<span class="nx">a</span><span class="p">.</span><span class="nx">top</span> <span class="o">></span> <span class="nx">b</span><span class="p">.</span><span class="nx">bottom</span> <span class="o">&&</span> <span class="nx">a</span><span class="p">.</span><span class="nx">bottom</span> <span class="o"><</span> <span class="nx">b</span><span class="p">.</span><span class="nx">top</span>
</code></pre></div></div>
<p>Back to this:</p>
<div class="language-cpp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="p">(</span><span class="n">RectA</span><span class="p">.</span><span class="n">Left</span> <span class="o"><</span> <span class="n">RectB</span><span class="p">.</span><span class="n">Right</span> <span class="o">&&</span> <span class="n">RectA</span><span class="p">.</span><span class="n">Right</span> <span class="o">></span> <span class="n">RectB</span><span class="p">.</span><span class="n">Left</span> <span class="o">&&</span>
<span class="n">RectA</span><span class="p">.</span><span class="n">Top</span> <span class="o">></span> <span class="n">RectB</span><span class="p">.</span><span class="n">Bottom</span> <span class="o">&&</span> <span class="n">RectA</span><span class="p">.</span><span class="n">Bottom</span> <span class="o"><</span> <span class="n">RectB</span><span class="p">.</span><span class="n">Top</span><span class="p">)</span>
</code></pre></div></div>
<h2 id="edge-case">Edge Case</h2>
<p>What case did we miss earlier?</p>
<p><a href="/assets/overlap/complete-overlap.png"><img src="/assets/overlap/complete-overlap.png" alt="identical rectangle in perfect overlap" /></a></p>
<ul>
<li>2 rectangles</li>
<li>same dimensions</li>
<li>exactly on top of each other</li>
</ul>
<p>In that case, neither rectangles is <em>inside</em> the other… but the stackoverflow solution works for all cases 😄</p>
<h2 id="discussion">Discussion</h2>
<p>Depending on your situation, the width of your rectangle boundary (1 pixel, or whatever units)
might make a difference: consider changing <code class="language-plaintext highlighter-rouge"><</code>/<code class="language-plaintext highlighter-rouge">></code> to <code class="language-plaintext highlighter-rouge"><=</code>/<code class="language-plaintext highlighter-rouge">>=</code> if needed.</p>
<p>If you’re still not convinced, try this <a href="https://silentmatt.com/rectangle-intersection/">interactive tool</a>.</p>
Serious Talk: Environment Variables2022-08-09T00:00:00+00:00https://blog.jpalardy.com/posts/serious-talk-environment-variables<h2 id="not-just-javascript">Not just JavaScript…</h2>
<p>While this post uses <code class="language-plaintext highlighter-rouge">process.env</code> – JavaScript, TypeScript, and node – to make a point, none of the problems
listed are JavaScript-specific.</p>
<p>I’ve seen this done in almost every languages.</p>
<h2 id="the-setup">The Setup</h2>
<p>Using <code class="language-plaintext highlighter-rouge">process.env</code> is <em>sooooooo</em> easy … too easy:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="kd">const</span> <span class="nx">databaseName</span> <span class="o">=</span> <span class="nx">process</span><span class="p">.</span><span class="nx">env</span><span class="p">.</span><span class="nx">DATABASE_NAME</span><span class="p">;</span>
<span class="c1">// use databaseName to connect to a database?</span></code></pre></figure>
<p>What is wrong with this code?</p>
<h2 id="problem-1-not-always-a-string">Problem 1: not <em>always</em> a string</h2>
<p>There’s a definite “happy path” attitude to reaching into <code class="language-plaintext highlighter-rouge">process.env</code> and expecting a string. If you use TypeScript, the type hint
says:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>const databaseName: string | undefined
</code></pre></div></div>
<p>Correct: reaching for a non-existing environment variable gives you <code class="language-plaintext highlighter-rouge">undefined</code>.</p>
<p>What happens if <code class="language-plaintext highlighter-rouge">databaseName</code> is <code class="language-plaintext highlighter-rouge">undefined</code>? 🤔 That depends what you do with it. If your next line of code looks like:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="kd">const</span> <span class="nx">connection</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">openConnection</span><span class="p">(</span><span class="nx">databaseName</span><span class="p">);</span> <span class="c1">// and password, and...</span></code></pre></figure>
<p>then you’re in trouble…</p>
<blockquote>
<p>just let it crash!</p>
</blockquote>
<p>I don’t disagree! If you always expect to run your application with all
(required) environment variables defined … and missing variables will make it
crash… that’s not bad. But propagating that <code class="language-plaintext highlighter-rouge">undefined</code> throughout your
application will probably not give you a great error message.</p>
<p>In TypeScript, you get a definite smell when you see:</p>
<figure class="highlight"><pre><code class="language-typescript" data-lang="typescript"><span class="kd">const</span> <span class="nx">databaseName</span> <span class="o">=</span> <span class="nx">process</span><span class="p">.</span><span class="nx">env</span><span class="p">.</span><span class="nx">DATABASE_NAME</span> <span class="k">as</span> <span class="kr">string</span><span class="p">;</span>
<span class="c1">// you "fixed" it, but I think TypeScript was trying to tell you something...</span></code></pre></figure>
<p>That <code class="language-plaintext highlighter-rouge">as string</code> is you saying “this will NEVER be undefined!”</p>
<h2 id="problem-2-maybe-not-a-string-either">Problem 2: maybe not a string either</h2>
<p>Wait, what? What about this?</p>
<figure class="highlight"><pre><code class="language-typescript" data-lang="typescript"><span class="kd">const</span> <span class="nx">replicaCount</span> <span class="o">=</span> <span class="nx">process</span><span class="p">.</span><span class="nx">env</span><span class="p">.</span><span class="nx">REPLICA_COUNT</span><span class="p">;</span></code></pre></figure>
<p>Oh no…</p>
<blockquote>
<p>use parseInt!</p>
</blockquote>
<p>Indeed. But my greater point is that you are now “parsing” (it’s in the name) a string. There are many ways that can go wrong:</p>
<ul>
<li>not a number! e.g. <code class="language-plaintext highlighter-rouge">"MUFFIN", "eleven", "🐴", "INFINITY!"</code> …</li>
<li>not an integer! e.g. <code class="language-plaintext highlighter-rouge">3.3, 0.125, ½</code> …</li>
<li>not a positive integer! e.g. <code class="language-plaintext highlighter-rouge">-42, -1, -99999</code></li>
<li>or zero? <code class="language-plaintext highlighter-rouge">0</code></li>
<li>too big? <code class="language-plaintext highlighter-rouge">9999999999999</code></li>
</ul>
<p>At this point, you need to parse and you probably need to validate. That sounds like more code than a one-liner.</p>
<h2 id="problem-3-its-easy-to-sprinkle-around-your-project">Problem 3: it’s easy to sprinkle around your project</h2>
<p>In the end, maybe what you have is closer to:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="kd">const</span> <span class="nx">databaseName</span> <span class="o">=</span> <span class="nx">process</span><span class="p">.</span><span class="nx">env</span><span class="p">.</span><span class="nx">DATABASE_NAME</span> <span class="o">||</span> <span class="dl">"</span><span class="s2">aquarius</span><span class="dl">"</span><span class="p">;</span></code></pre></figure>
<ul>
<li>always a string</li>
<li>a reasonable default</li>
<li>no obvious validations: you will take <code class="language-plaintext highlighter-rouge">DATABASE_NAME</code> as given<br />
(even if an empty/blank string?)</li>
</ul>
<p>The problem is where this code lives… would you expect someone to <code class="language-plaintext highlighter-rouge">grep</code> your
project to find all environment variables? Finding a <code class="language-plaintext highlighter-rouge">process.env</code> in the
middle on some module makes me wonder what else is scattered all over the code.</p>
<p>Counterpoint: what if you tried to reach get <code class="language-plaintext highlighter-rouge">databaseName</code> from <code class="language-plaintext highlighter-rouge">process.argv</code>?</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="c1">// don't do this...</span>
<span class="kd">const</span> <span class="nx">databaseNameIndex</span> <span class="o">=</span> <span class="nx">process</span><span class="p">.</span><span class="nx">argv</span><span class="p">.</span><span class="nx">indexOf</span><span class="p">(</span><span class="dl">"</span><span class="s2">--database-name</span><span class="dl">"</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">databaseName</span> <span class="o">=</span> <span class="nx">process</span><span class="p">.</span><span class="nx">argv</span><span class="p">[</span><span class="nx">databaseNameIndex</span> <span class="o">+</span> <span class="mi">1</span><span class="p">];</span></code></pre></figure>
<p>Would that deserve a comment in the code review?! In fact, it’s hard to account for everything that’s wrong with this approach… 😬 (make your own list!)</p>
<p>Why does <code class="language-plaintext highlighter-rouge">process.env</code> feel different than <code class="language-plaintext highlighter-rouge">process.argv</code>?</p>
<p>I think programmers understand that <code class="language-plaintext highlighter-rouge">process.argv</code> is a mess, in the best of
times. You cannot take anything for granted and handling it (correctly) is
enough work to justify bringing it a library.</p>
<p>But <code class="language-plaintext highlighter-rouge">process.env</code> <em>feels</em> like a Dictionary of string to string. Maybe we
can skip all that gross work, cross our fingers, and, worst-case, let it crash?</p>
<p>I think that attitude accounts for the popularity of passing config through environment variables. (I know there are other reasons)</p>
<h2 id="what-to-do-instead">What to do instead?</h2>
<p>By now, I hope to have convinced you that config is complicated enough to handle separately and <em>seriously</em>.</p>
<p>My recommendations are:</p>
<ul>
<li>centralize ALL config in one place</li>
<li>combine different sources – config files, environment variables, command-line arguments, overriding in that order</li>
<li>parse, validate, and sanity check early</li>
<li>generate meaningful error messages</li>
<li>let it crash, if appropriate</li>
</ul>
<p>From the code that starts your app, you are aiming for:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="kd">const</span> <span class="nx">config</span> <span class="o">=</span> <span class="nx">getConfig</span><span class="p">();</span>
<span class="c1">// let getConfig crash or throw if something is wrong</span></code></pre></figure>
<p>and you can hide all that ugly and unexciting code in there.</p>
<p>If you get a <code class="language-plaintext highlighter-rouge">config</code> object, you can be sure that it contains everything you
need, that all its values have been validated (no nulls!). You can inject
<code class="language-plaintext highlighter-rouge">config</code> everywhere it’s needed … and remove your guard clauses and
special cases.</p>
<p>It also makes mocking easier, no need to write to or modify <code class="language-plaintext highlighter-rouge">process.env</code> 😬</p>
Trying vim-slime in Docker2022-07-08T00:00:00+00:00https://blog.jpalardy.com/posts/trying-vim-slime-in-docker<p>(Disclaimer: I wrote <a href="https://github.com/jpalardy/vim-slime">vim-slime</a>)</p>
<h2 id="installing-a-vim-plugin-probably-not-fun">Installing a vim plugin: probably not fun…</h2>
<p>Maybe you’ve heard about <a href="https://github.com/jpalardy/vim-slime">vim-slime</a>, thought it was cool … but you weren’t sure what you were getting yourself into 🤔</p>
<p>That’s understandable … I feel the same about every vim plugin:</p>
<ul>
<li>is it going to be simple to install?</li>
<li>will it interfere with other plugins?</li>
<li>if something breaks, will it be easy to fix?</li>
<li>how much config will be necessary? (for optimum experience)</li>
<li>is it going to be simple to uninstall?</li>
</ul>
<p>In this case, I propose a “clean slate” experience. Since you probably don’t have a new computer with a fresh OS install lying around, let’s talk about Docker.</p>
<h2 id="why-docker">Why Docker?</h2>
<p>Is Docker perfect? <em>No</em>. (especially on macOS)</p>
<p>What I appreciate about Docker is: the mess stays inside a container. You’re
always a few commands (<code class="language-plaintext highlighter-rouge">docker rm</code>, <code class="language-plaintext highlighter-rouge">docker rmi</code>, <code class="language-plaintext highlighter-rouge">docker system prune</code>) away from returning to a known/clean state.</p>
<p>I often have to reproduce
<a href="https://github.com/jpalardy/vim-slime/issues">problems</a> with combinations of
config and tools completely different from how I usually work. I would:</p>
<ul>
<li>change my config…</li>
<li>install new tools…</li>
<li>hope I could reproduce the problem 🤞</li>
<li><code class="language-plaintext highlighter-rouge">git checkout .</code> my dotfiles back to sanity</li>
<li>remember to uninstall things…</li>
</ul>
<p>Nothing about this process would “feel good”. Now I try to contain the blast radius inside Docker.</p>
<h2 id="instructions-vim-slime-in-docker">Instructions: vim-slime in Docker</h2>
<p>Set up the foundations:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">docker run <span class="nt">-ti</span> <span class="nt">--rm</span> alpine
apk update
apk add vim tmux git</code></pre></figure>
<p>(if, at this point, you’re thinking “why alpine?” … adjust these instructions to your own preferences)</p>
<p>Install vim-slime:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">mkdir</span> <span class="nt">-p</span> ~/.vim/pack/plugins/start
<span class="nb">cd</span> ~/.vim/pack/plugins/start
git clone https://github.com/jpalardy/vim-slime.git
<span class="nb">cd</span> <span class="c"># optional, but gets you back $HOME</span></code></pre></figure>
<p>Configure vim-slime:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">echo</span> <span class="s1">'let g:slime_target = "tmux"'</span> <span class="o">></span> ~/.vim/vimrc</code></pre></figure>
<p>Try vim-slime:</p>
<figure class="highlight"><pre><code class="language-txt" data-lang="txt">1. start tmux: tmux
2. split your window: ctrl-b " <-- ctrl-b, double-quote
3. start vim: vim
4: in vim, type: ls -l / | head -n 10
5: in vim, type: ctrl-c, ctrl-c <-- from normal mode
at this point, vim-slime will prompt for your tmux config:
1. "tmux socket name or absolute path: default" -- accept
2. "tmux target pane: :" -- replace ":" with "{last}"
"ls -l / | head -n 10" runs in the other pane (where bash was running)</code></pre></figure>
<p><img src="/assets/trying-vim-slime/minimal-vim-slime.png" alt="showing the results of the instructions above" /></p>
<h2 id="whats-next">What’s next?</h2>
<p>Everything above was “the minimum”; the least number of steps to try vim-slime.<br />
No magic ❌🪄</p>
<p>Some improvements:</p>
<ul>
<li>default tmux is pretty bare, dump your config to <code class="language-plaintext highlighter-rouge">~/.tmux.conf</code></li>
<li>default vim is pretty bare, dump your config to <code class="language-plaintext highlighter-rouge">~/.vim/vimrc</code><br />
(remember to add some vim-slime config…)</li>
</ul>
<p>A little bit more vim-slime config for your vimrc:</p>
<figure class="highlight"><pre><code class="language-vim" data-lang="vim"><span class="k">let</span> <span class="nv">g:slime_target</span> <span class="p">=</span> <span class="s2">"tmux"</span>
<span class="k">let</span> <span class="nv">g:slime_default_config</span> <span class="p">=</span> <span class="p">{</span><span class="s2">"socket_name"</span><span class="p">:</span> <span class="s2">"default"</span><span class="p">,</span> <span class="s2">"target_pane"</span><span class="p">:</span> <span class="s2">"{last}"</span><span class="p">}</span>
<span class="k">let</span> <span class="nv">g:slime_dont_ask_default</span> <span class="p">=</span> <span class="m">1</span></code></pre></figure>
<p>This will use <code class="language-plaintext highlighter-rouge">{last}</code> and won’t even ask for you to confirm the config!<br />
(this is how I work)</p>
<p>If you don’t want to use tmux, there are many other options:</p>
<ul>
<li>(GNU) screen</li>
<li>dtach</li>
<li>kitty</li>
<li>x11</li>
<li>whimrepl</li>
<li>conemu</li>
<li>vimterminal</li>
<li>neovim terminal</li>
</ul>
<p>See the <a href="https://github.com/jpalardy/vim-slime">README</a> for details on how to configure things.</p>
<p>Finally, go step by step – it’s easier to debug that way. And if you need help, <a href="https://github.com/jpalardy/vim-slime/issues">open an issue</a> and we’ll figure it out.</p>
The World Changes Around Your Code2022-06-13T00:00:00+00:00https://blog.jpalardy.com/posts/the-world-changes-around-your-code<p>It started with a conversation with my parents. We were talking about my work, legacy software and maintenance:</p>
<blockquote>
<p>But if that software is working … why does it need maintenance?</p>
</blockquote>
<p>At the time, I didn’t have a ready-made answer.</p>
<ul>
<li>old software is crap?</li>
<li>we do things differently now?</li>
<li>things need to be updated?</li>
</ul>
<h2 id="why-does-software-rot">Why does software rot?</h2>
<p>It is sometimes called <a href="https://en.wikipedia.org/wiki/Software_rot">software rot</a> (or bit rot):</p>
<blockquote>
<p>[…] <strong>a slow deterioration of software quality over time</strong> or its diminishing
responsiveness that will eventually lead to software becoming faulty,
unusable, or in need of upgrade.</p>
</blockquote>
<p>Further:</p>
<blockquote>
<p>This is not a physical phenomenon: <strong>the software does not actually decay</strong>, but
rather suffers from a lack of being responsive and updated with respect to
<strong>the changing environment</strong> in which it resides.</p>
</blockquote>
<p>(emphasis mine)</p>
<p>In fact, you could argue that software rot is the opposite of regular rot: the
software is unchanged, but the <strong>world itself</strong> started to rot…</p>
<h2 id="no-software-is-an-island">No Software Is An Island…</h2>
<p>Modern software rests on tremendous piles of dependencies:</p>
<ul>
<li>specific hardware</li>
<li>operating systems</li>
<li>libraries</li>
<li>protocols</li>
<li>conventions</li>
</ul>
<p>Worse, your control over these things is limited (at best).</p>
<p>When — not if — these things change, you have 2 choices:</p>
<ul>
<li>keeping up — paying it down; incurring the cost</li>
<li>punting — falling behind, revisiting this problem later… maybe?</li>
</ul>
<h2 id="technical-debt">Technical Debt</h2>
<p>I find it hard to talk about this problem without resorting to the language of economics.</p>
<p>People understand <a href="https://en.wikipedia.org/wiki/Technical_debt">technical debt</a>
in the context of their own software: cutting corners <em>today</em> for some benefit
— usually meeting a deadline. Some extra costs, or “interests” in this analogy,
are assumed under rework and future slowdowns.</p>
<p><em>(in the context of this blog post, whether or not cutting corners will make things faster, or
ever be paid back is besides the point)</em></p>
<p>In that sense, your dependencies are a type of technical debt — it’s software
you didn’t want to write and test. The hope is that the <a href="https://en.wikipedia.org/wiki/Total_cost_of_ownership">total cost</a>
of your dependencies is lower than software you could write yourself.</p>
<p>In some cases, this is true! But it’s certainly not true in all cases and
wisdom resides in knowing the difference (or hedging your bets).</p>
Parsing and Validating Dates in Awk2022-04-13T00:00:00+00:00https://blog.jpalardy.com/posts/parsing-and-validate-dates-in-awk<p>I recently stumbled on something that I <em>thought</em> would be easy: parsing and validating dates in Awk.</p>
<p>Some guidelines:</p>
<ul>
<li><a href="https://en.wikipedia.org/wiki/ISO_8601">ISO 8601</a> format, e.g. <code class="language-plaintext highlighter-rouge">2022-04-12</code></li>
<li>months between 01 and 12</li>
<li>days between 01 and 28-31 (depending on the month, leap year)</li>
</ul>
<p>A quick lookup for GNU Awk’s <a href="https://www.gnu.org/software/gawk/manual/html_node/Time-Functions.html">time functions</a> points to <code class="language-plaintext highlighter-rouge">mktime("YYYY MM DD HH MM SS")</code></p>
<p>Let’s try it:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">echo</span> <span class="s2">"2022 04 12"</span> | <span class="nb">awk</span> <span class="s1">'{ print mktime($0 " 0 0 0") }'</span>
1649746800
<span class="c"># removed hyphens for now, will fix in solution below</span>
<span class="c"># padded HH MM SS with 0 0 0, to keep mktime happy</span></code></pre></figure>
<p>Looking good! How about something wrong?</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">echo</span> <span class="s2">"not a date"</span> | <span class="nb">awk</span> <span class="s1">'{ print mktime($0 " 0 0 0") }'</span>
<span class="nt">-1</span></code></pre></figure>
<p>Oh yeah! What about a “bad” date?</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">echo</span> <span class="s2">"2022 44 78"</span> | <span class="nb">awk</span> <span class="s1">'{ print mktime($0 " 0 0 0") }'</span>
1760684400</code></pre></figure>
<p>Wait, what?! Oh no……..</p>
<h2 id="going-full-circle">Going Full Circle</h2>
<p>If invalid dates returned -1, we would be done by now.</p>
<p>1760684400 is <code class="language-plaintext highlighter-rouge">2025-10-17</code> … <code class="language-plaintext highlighter-rouge">mktime</code> takes 44 and 78 and (probably) multiplies those by seconds-per-month, and
seconds-per-day.</p>
<p>When I looked at the other <a href="https://www.gnu.org/software/gawk/manual/html_node/Time-Functions.html">time functions</a>,
there didn’t seem to be anything that helped either.</p>
<p>The <em>eureka!</em> was to think about using the invalid date to format a date back to ISO
8601 format. If the input and output dates are different, the date is wrong!</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># good example</span>
<span class="o">></span> <span class="nb">echo</span> <span class="s2">"2022 04 12"</span> | <span class="nb">awk</span> <span class="s1">'{ d = mktime($0 " 0 0 0"); print strftime("%F", d) }'</span>
2022-04-12</code></pre></figure>
<p><img src="/assets/awk-dates/date-roundtrip.png" alt="parsing date and formatting the same date" /></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># bad example</span>
<span class="o">></span> <span class="nb">echo</span> <span class="s2">"2022 44 78"</span> | <span class="nb">awk</span> <span class="s1">'{ d = mktime($0 " 0 0 0"); print strftime("%F", d) }'</span>
2025-10-17</code></pre></figure>
<p>Sidenote: I’m using <code class="language-plaintext highlighter-rouge">%F</code> to format dates. <a href="https://man7.org/linux/man-pages/man3/strftime.3.html">man 3 strftime</a> says:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>%F Equivalent to %Y-%m-%d (the ISO 8601 date format)
</code></pre></div></div>
<h2 id="test-script">Test Script</h2>
<p>Here’s my test cases:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">cat </span>test.txt
2022-04-12 <span class="nt">--</span> regular day
bad_date <span class="nt">--</span> not even a <span class="nb">date
</span>1981-11-20 <span class="nt">--</span> 1980s
2022-44-78 <span class="nt">--</span> nonsense month/day
2022-09-30 <span class="nt">--</span> september has 30 days
2022-09-31 <span class="nt">--</span> but not 31 ...
2016-02-28 <span class="nt">--</span> leap year: february has 28 days
2016-02-29 <span class="nt">--</span> leap year: even 29 days
2016-02-30 <span class="nt">--</span> leap year: but not 30 days
2000-02-28 <span class="nt">--</span> special leap year: february has 28 days
2000-02-29 <span class="nt">--</span> special leap year: even 29 days
2000-02-30 <span class="nt">--</span> special leap year: but not 30 days
2001-02-28 <span class="nt">--</span> regular year: february has 28 days
2001-02-29 <span class="nt">--</span> regular year: but not 29 days
2001-02-30 <span class="nt">--</span> regular year: but not 30 days
1965-04-12 <span class="nt">--</span> past, before 1970
1935-04-12 <span class="nt">--</span> past, before 1970</code></pre></figure>
<p>The Awk script:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># hyphens now removed</span>
<span class="o">></span> <span class="nb">cat </span>test.awk
<span class="o">{</span>
<span class="nb">date</span> <span class="o">=</span> mktime<span class="o">(</span>gensub<span class="o">(</span><span class="s2">"-"</span>, <span class="s2">" "</span>, <span class="s2">"g"</span>, <span class="nv">$1</span><span class="o">)</span> <span class="s2">" 0 0 0"</span><span class="o">)</span>
<span class="k">if</span> <span class="o">(</span>strftime<span class="o">(</span><span class="s2">"%F"</span>, <span class="nb">date</span><span class="o">)</span> <span class="o">!=</span> <span class="nv">$1</span><span class="o">)</span> <span class="o">{</span>
print <span class="s2">"bad: "</span>, <span class="nv">$0</span>
next
<span class="o">}</span>
print <span class="s2">"good:"</span>, <span class="nv">$0</span>
<span class="o">}</span></code></pre></figure>
<p>Results:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">awk</span> <span class="nt">-f</span> test.awk test.txt ~/Documents/blog <span class="o">(</span>main<span class="o">)</span>
good: 2022-04-12 <span class="nt">--</span> regular day
bad: bad_date <span class="nt">--</span> not even a <span class="nb">date
</span>good: 1981-11-20 <span class="nt">--</span> 1980s
bad: 2022-44-78 <span class="nt">--</span> nonsense month/day
good: 2022-09-30 <span class="nt">--</span> september has 30 days
bad: 2022-09-31 <span class="nt">--</span> but not 31 ...
good: 2016-02-28 <span class="nt">--</span> leap year: february has 28 days
good: 2016-02-29 <span class="nt">--</span> leap year: even 29 days
bad: 2016-02-30 <span class="nt">--</span> leap year: but not 30 days
good: 2000-02-28 <span class="nt">--</span> special leap year: february has 28 days
good: 2000-02-29 <span class="nt">--</span> special leap year: even 29 days
bad: 2000-02-30 <span class="nt">--</span> special leap year: but not 30 days
good: 2001-02-28 <span class="nt">--</span> regular year: february has 28 days
bad: 2001-02-29 <span class="nt">--</span> regular year: but not 29 days
bad: 2001-02-30 <span class="nt">--</span> regular year: but not 30 days
good: 1965-04-12 <span class="nt">--</span> past, before 1970
good: 1935-04-12 <span class="nt">--</span> past, before 1970</code></pre></figure>
<p>I was surprised that pre-1970 (<a href="https://en.wikipedia.org/wiki/Unix_time">epoch</a>) also worked! Their <code class="language-plaintext highlighter-rouge">mktime</code> values are negative:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">echo</span> <span class="s2">"1935 04 12"</span> | <span class="nb">awk</span> <span class="s1">'{ print $0, "=>", mktime($0 " 0 0 0")}'</span>
1935 04 12 <span class="o">=></span> <span class="nt">-1095782400</span>
<span class="o">></span> <span class="nb">echo</span> <span class="s2">"1969 12 31"</span> | <span class="nb">awk</span> <span class="s1">'{ print $0, "=>", mktime($0 " 0 0 0")}'</span>
1969 12 31 <span class="o">=></span> <span class="nt">-57600</span>
<span class="o">></span> <span class="nb">echo</span> <span class="s2">"1970 01 01"</span> | <span class="nb">awk</span> <span class="s1">'{ print $0, "=>", mktime($0 " 0 0 0")}'</span>
1970 01 01 <span class="o">=></span> 28800</code></pre></figure>
Stop Typing Into REPLs2022-03-16T00:00:00+00:00https://blog.jpalardy.com/posts/stop-typing-into-repls<h2 id="help-me-help-you">Help me help you</h2>
<p>When I’m at someone’s desks, in person or virtually, it drives me crazy to see
the <em>heroics</em> of typing everything in a <a href="https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop">REPL</a>…</p>
<p>Let alone trying to manually fix broken code on in a REPL…</p>
<p>Or losing their work when exiting the REPL, and trying to grab fragments from the scrollback 😿</p>
<p>It doesn’t have to be like this…</p>
<h2 id="repls-arent-perfect">REPLs aren’t perfect…</h2>
<p>I ❤️ REPLs and they have great features:</p>
<ul>
<li>interactivity (including control of evaluation and tab-completion)</li>
<li>feedback speed</li>
</ul>
<p>but they also have drawbacks, <a href="https://dictionary.cambridge.org/dictionary/english/ymmv">YMMV</a>:</p>
<ul>
<li>no syntax highlighting</li>
<li>poor keyboard editing – probably Emacs-style shortcuts</li>
<li>poor history – since session began; up/down arrow</li>
</ul>
<p>Since text editors are becoming more interactive (i.e. like IDEs), and
computers are <em>generally</em> becoming faster, the advantages of using a REPL have been
decreasing over time. It seems common to settle for “whatever, I’ll just run
the code…”.</p>
<p>But what if you could have both?</p>
<h2 id="mixed-mode-environments">Mixed mode environments</h2>
<p>The most obvious example might be a <a href="https://jupyter.org/">Jupyter notebook</a></p>
<p><img src="/assets/stop-typing-repls/jupyterlab.png" alt="screenshot of jupyter notebook" /></p>
<p>You mix code cells and their results (whether text, number, graph) and weave,
step-by-step, a report/presentation. If you’ve ever used a Jupyter
notebook, or seen someone use one, you can see how powerful and liberating it
can be.</p>
<p>But, if you’re willing to accept it, Microsoft Excel is probably the embodiment
of an interactive programming environment:</p>
<p><img src="/assets/stop-typing-repls/excel.png" alt="screenshot of Excel" /></p>
<blockquote>
<p>Excel formulas are the world’s most widely used programming language.</p>
</blockquote>
<p>– <a href="https://www.theregister.com/2020/12/04/microsoft_excel_lambda/">Microsoft, via The Register</a></p>
<p>And while these environments might be blissful for specific tasks, they certainly are not general purpose…</p>
<h2 id="turning-every-repl-into-a-jupiter-notebook">Turning EVERY REPL into a “Jupiter notebook”</h2>
<p>We already have everything we need:</p>
<ul>
<li>a REPL – any REPL will do</li>
<li>a text editor</li>
</ul>
<p>the REPL does what it’s good for (interactivity, speed, …) and the text editor can focus on its strengths:</p>
<ul>
<li>great editing</li>
<li>syntax highlighting</li>
<li>organising, formatting, saving text</li>
</ul>
<p>What’s missing? You need a way to “send text from the text editor to the REPL”.</p>
<p>Good news if you’re using vim, that’s already available: <a href="https://github.com/jpalardy/vim-slime">vim-slime</a></p>
<p><img src="/assets/stop-typing-repls/vim-slime.gif" alt="screenshot of vim-slime" /></p>
<p>(disclosure: I am the author of vim-slime)</p>
<p>You end up with sessions that look like this:</p>
<p><img src="/assets/stop-typing-repls/16-9.png" alt="sample R session" /></p>
<ul>
<li>vim on one side</li>
<li>R, or another REPL, on the other side</li>
<li>graphs can pop in another window (this one opened by R)</li>
</ul>
<p>I usually keep my eyes in vim, unless some error message shows up on the right.</p>
<h2 id="what-do-i-use-vim-slime-for">What do I use vim-slime for?</h2>
<p>From memory, I’ve used vim-slime with:</p>
<ul>
<li>node (javascript)</li>
<li>elm</li>
<li>irb (ruby)</li>
<li>python</li>
<li>iex (elixir)</li>
<li>R, R markdown</li>
<li>octave</li>
<li>mysql</li>
<li>psql</li>
<li>sqlite</li>
<li>even shells! (fish, bash, zsh)</li>
</ul>
<p>I haven’t found a REPL where it wasn’t advantageous to “wrap” in vim-slime.</p>
<h2 id="it-doesnt-have-to-be-vim-slime">It doesn’t have to be vim-slime…</h2>
<p>I understand that vim isn’t for everybody. The other good news is that vim-slime
isn’t complicated … most of the work happens when “shelling out”. I don’t see
why your favorite text editor couldn’t support this type of workflow.</p>
<p>If you find compelling non-vim alternatives, email me and I’ll include them on
this page for others to find.</p>
Fish Shell is Awesome2022-02-10T00:00:00+00:00https://blog.jpalardy.com/posts/fish-shell-is-awesome<p>(when I <a href="https://blog.jpalardy.com/posts/first-look-at-zsh/">switched my shell to zsh</a>, I
mentioned that I would try <a href="https://fishshell.com/">fish</a> and report back: this is it.)</p>
<p>TLDR:</p>
<ul>
<li>fish shell is awesome!</li>
<li>including features I didn’t know I wanted</li>
<li>but, it’s not common – enjoy it on your own devices, but don’t expect it to be the default shell when you ssh somewhere</li>
</ul>
<h2 id="truth-in-advertising">Truth in Advertising</h2>
<p>The front page of the <a href="https://fishshell.com/">fish shell</a> has a “sales pitch”</p>
<p><a href="https://fishshell.com/"><img src="/assets/fish-shell/sales-pitch.png" alt="the fish shell advantages, as shown on the home page" /></a></p>
<p>I rediscovered it while coming up with my own list of what makes fish great. But the pitch
already has most of my points – fish shell had tried to tell me all along, I just hadn’t believed it!</p>
<h2 id="try-before-you-buy">Try Before You Buy</h2>
<p>I think the biggest fears when switching shells come down to:</p>
<ul>
<li>how to try it without messing up your current setup?</li>
<li>how much configuration/googling will you need before being productive?</li>
</ul>
<p>For the first point, once installed (<code class="language-plaintext highlighter-rouge">brew install fish</code>, or equivalent), just
type <code class="language-plaintext highlighter-rouge">fish</code>. You will be running fish without fiddling with your terminal or
default shell. If you don’t like it, type <code class="language-plaintext highlighter-rouge">exit</code> to return to your “real”
shell. This is low commitment!</p>
<p>For the second point, I have some good news:</p>
<ul>
<li>the fish config is sane by default!<br />
(a refreshing development over bash/zsh)</li>
<li>there are surprisingly little knobs and settings to play with: trust the defaults!</li>
</ul>
<p>If you want to customize the colors or the prompt, type <code class="language-plaintext highlighter-rouge">fish_config</code> and enjoy the
browser-based config and preview.</p>
<p><img src="/assets/fish-shell/web_config.png" alt="preview of the fish_config web based user interface" /></p>
<h2 id="all-about-completion">All About Completion</h2>
<p>The feature that makes or breaks a shell is what happens when you press <code class="language-plaintext highlighter-rouge">TAB</code>.
I would <em>NOT</em> be talking about fish if its completion wasn’t excellent.</p>
<p>Let me try to summarize with a few examples:</p>
<p><img src="/assets/fish-shell/command-not-found.png" alt="fish knows in advance when commands won't work" /></p>
<ul>
<li>the red highlight had already told me that the command wouldn’t work…</li>
<li>fish checks while you type</li>
</ul>
<p><img src="/assets/fish-shell/history-hints.png" alt="fish remembers what you typed and hints from history" /></p>
<ul>
<li>valid commands are cyan (customizable)</li>
<li>completed words are underlined (I pressed <code class="language-plaintext highlighter-rouge">TAB</code> to complete the filename)</li>
<li>the grey text comes from the command history</li>
<li>right-arrow will complete the grey text (not <code class="language-plaintext highlighter-rouge">TAB</code>)</li>
<li><code class="language-plaintext highlighter-rouge">ctrl-f</code> would grab the first word from the grey text (if there were multiple words)</li>
</ul>
<p>Yes, other shells have history completion – the difference is that it shows up
automatically. With bash/zsh, you have to remember that a command is in
your history to want to grab it with <code class="language-plaintext highlighter-rouge">ctrl-r</code>.</p>
<p>More about command history:</p>
<ul>
<li>fish history in <em>almost</em> infinite (it stores <a href="https://github.com/fish-shell/fish-shell/issues/2674">256k entries, in a LRU</a>)</li>
<li>history hints are directory-aware … offering you commands you would only type in certain directories!</li>
<li>multi-shell histories are merged on shell exit: no clobbers, no worries</li>
</ul>
<p>Other features worth mentioning:</p>
<ul>
<li>out-of-the-box installation comes with completions for most commands</li>
<li>man page can be parsed to generate completions (as claimed in sales pitch)</li>
<li>rsync/scp commands complete paths on the other side of an ssh connection (!)</li>
<li>flag completion, with helpful text (below)</li>
</ul>
<p><img src="/assets/fish-shell/flag-completion.png" alt="press TAB to show command-line flag completion and help" /></p>
<p>A few other things:</p>
<ul>
<li>case-insensitive (case-flexible?) completion – type it lowercase and <code class="language-plaintext highlighter-rouge">TAB</code> fix it!</li>
<li>middle-of-text completion – you don’t have to type the beginning of a filename/directory, any unambiguous part will do</li>
</ul>
<h2 id="whats-the-catch">What’s the Catch?</h2>
<p>The biggest catch is that it’s not default. You have to install it
(probably) and enable it. Default choices are powerful, and that’s why bash/zsh
still rule.</p>
<p>You can reduce the pain by “cheating”. Add <code class="language-plaintext highlighter-rouge">fish</code> to the bottom of your <code class="language-plaintext highlighter-rouge">.bashrc</code>/<code class="language-plaintext highlighter-rouge">.zshrc</code></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># pretty easy, no?</span>
fish</code></pre></figure>
<p>Comment it out if you change your mind.</p>
<p>A few other things to know:</p>
<ul>
<li>fish syntax, if you are scripting, isn’t the same bash/zsh syntax – you might say it’s a good thing. I read through the <a href="https://fishshell.com/docs/current/language.html">language reference</a> and found it reasonable. (better than sh’s <code class="language-plaintext highlighter-rouge">CASE</code>-<code class="language-plaintext highlighter-rouge">ESAC</code>, <code class="language-plaintext highlighter-rouge">IF</code>-<code class="language-plaintext highlighter-rouge">FI</code>, <code class="language-plaintext highlighter-rouge">DO</code>-<code class="language-plaintext highlighter-rouge">DONE</code> craziness…)</li>
<li>it’s worth reading through <a href="https://fishshell.com/docs/current/fish_for_bash_users.html">Fish for bash users</a>; it’s not too long</li>
</ul>
<p>So, yes, fish will spoil you. And if you have to use other shells, you will miss its features.</p>
<p>Is that such a bad thing?</p>
<h2 id="final-worlds">Final Worlds</h2>
<blockquote>
<p>Technology becomes truly useful when it becomes invisible.</p>
</blockquote>
<p>I spent years trying to beat bash (and zsh) into shape. Many of my blog posts
were about removing friction from the command-line experience. There is <em>some</em>
joy in mastering complicated tools…</p>
<p>But fish allowed me to do everything I wanted, and more, without feeling like a struggle. (or worse: spending days/weeks/months getting my configuration just right)</p>
<p>Give fish a try and let me know what’s missing.</p>
The Best Books I Read in 20212021-12-17T00:00:00+00:00https://blog.jpalardy.com/posts/best-books-i-read-2021<p>I have read many books in 2021; let’s forget most and talk about the good ones.</p>
<p>There is no particular order, but I broke down my recommendations by
categories: <a href="#technical">technical</a>, <a href="#non-fiction">non-fiction</a> and
<a href="#fiction">fiction</a>.</p>
<p>This is a yearly tradition! You can read my book reviews from previous years:<br />
<a href="https://blog.jpalardy.com/posts/best-books-i-read-2015/">2015</a> <a href="https://blog.jpalardy.com/posts/best-books-i-read-2016/">2016</a> <a href="https://blog.jpalardy.com/posts/best-books-i-read-2017/">2017</a> <a href="https://blog.jpalardy.com/posts/best-books-i-read-2018/">2018</a> <a href="https://blog.jpalardy.com/posts/best-books-i-read-2019/">2019</a> and <a href="https://blog.jpalardy.com/posts/best-books-i-read-2020/">2020</a>.</p>
<hr />
<h2 id="technical">Technical</h2>
<h3 id="crafting-interpreters">Crafting Interpreters</h3>
<p><a href="https://www.amazon.com/dp/0990582930/"><img class="book-cover" src="/assets/best-books-2021/0990582930.jpg" alt="Crafting Interpreters" /></a></p>
<p>I knew I would buy and read this book since the first time I heard about it.</p>
<p>I loved <a href="https://blog.jpalardy.com/posts/best-books-i-read-2020/#game-programming-patterns">Nystrom’s previous book</a> and this is more of the good stuff.
It’s <em>deep</em> but strangely light-hearted.</p>
<p>If you follow along, you’ll end up writing not one but two (!)
interpreters: one in Java and another more optimized and lower-level
in clean/modern C.</p>
<p>This wasn’t my first look at interpreters (see <a href="https://www.nand2tetris.org/book">From Nand to Tetris</a>), but it felt better organized.
There were no leaps of faith; it was a logical progression with reasonable compromises (discussed in the text).</p>
<p>Some caveats:</p>
<ul>
<li>you’ll need an interest in the details of interpreters; which might not be immediately applicable on a day-to-day basis</li>
<li>
<p>it’s really two books in one (it’s the size of a phone book!) – I would have been perfectly happy to stop after the first (Java) interpreter. The second half of the book was interesting, but I was getting a little tired by then… 😅</p>
</li>
<li><a href="https://www.amazon.com/dp/0990582930/">amazon</a></li>
<li><a href="https://craftinginterpreters.com/">website</a></li>
</ul>
<hr />
<h2 id="non-fiction">Non-Fiction</h2>
<h3 id="your-money-or-your-life">Your Money or Your Life</h3>
<p><a href="https://www.amazon.com/dp/0143115766/"><img class="book-cover" src="/assets/best-books-2021/0143115766.jpg" alt="Your Money or Your Life" /></a></p>
<p>When I read an article about a man who retired in his early 40s and he mentioned
that <strong>Your Money or Your Life</strong> had changed his life … I was intrigued. What am I missing? Why am I not retired?! 🤔</p>
<p>To be honest, I spent most of the book rolling my eyes 🙄. This felt so obvious… I’ve read other books and my finances are in order; I’m not starting from zero!</p>
<p>But … the major insight, from my perspective, was about cashflow: if you have more passive revenue than expenses; you can retire. The very short version:</p>
<ul>
<li>calculate all your expenses, every month</li>
<li>cut everything superfluous; the more you spend, the more you need to finance your lifestyle – that’s what the book title refers to: you exchange your life/time against money to pay for things</li>
<li>retire when <code class="language-plaintext highlighter-rouge">$monthly_expenses = $total_retirement_money * 5% / 12</code></li>
</ul>
<p>Obviously, the devil is in the details. And you might not agree with the premise or the conclusions. In my case, it gave me a target, and a framework.</p>
<ul>
<li><a href="https://www.amazon.com/dp/0143115766/">amazon</a></li>
<li><a href="https://en.wikipedia.org/wiki/Vicki_Robin#Your_Money_or_Your_Life">wikipedia</a></li>
</ul>
<h3 id="how-asia-works">How Asia Works</h3>
<p><a href="https://www.amazon.com/dp/0802121322/"><img class="book-cover" src="/assets/best-books-2021/0802121322.jpg" alt="How Asia Works" /></a></p>
<p>Following a Twitter thread, I ended up on <a href="https://noahpinion.substack.com/p/against-hickelism">this post</a>. This quote caught my eye:</p>
<blockquote>
<p>My friend once joked that I was a neural net trained on the book How Asia Works, since I recommend it so often.</p>
</blockquote>
<p>For another (shorter) take, Bill Gates also <a href="https://www.gatesnotes.com/Books/How-Asia-Works">reviewed the book</a>.</p>
<blockquote>
<p>How did countries like Japan, Taiwan, South Korea, and China achieve
sustained, high growth and turn into development success stories? And why
have so few other countries managed to do so?</p>
<p>[…]</p>
<p>He offers a simple, three-part formula:</p>
<ol>
<li>Create conditions for small farmers to thrive.</li>
<li>Use the proceeds from agricultural surpluses to build a manufacturing base that is tooled from the start to produce exports.</li>
<li>Nurture both these sectors (small farming and export-oriented manufacturing) with financial institutions closely controlled by the government.</li>
</ol>
</blockquote>
<p>It seems that countries that “made it” followed a variation of this script –
in contrast to unproven neoliberal alternatives. Faced with western economists, the best idea
might be to agree … and completely ignore their ideas.</p>
<ul>
<li><a href="https://www.amazon.com/dp/0802121322/">amazon</a></li>
<li><a href="https://howasiaworks.wordpress.com/">website</a></li>
</ul>
<h3 id="how-your-house-works">How Your House Works</h3>
<p><a href="https://www.amazon.com/dp/1119467616/"><img class="book-cover" src="/assets/best-books-2021/1119467616.jpg" alt="How Your House Works" /></a></p>
<p>How would you (really) build a house?</p>
<p>I found this book while trying to answer this question beyond hand-waving
explanations and generalities.</p>
<p>But <strong>How Your House Works</strong> is a slightly different book. I had
planned to read other <a href="https://www.amazon.com/s?i=stripbooks&rh=p_27%3ACharlie+Wing&s=relevancerank&text=Charlie+Wing&ref=dp_byline_sr_book_1">books by the same author</a>, but this one certainly looked
simple, well-illustrated, and had excellent reviews.</p>
<p>From the introduction:</p>
<blockquote>
<p>After observing neighbors, friends, and family through decades of home
ownership, I’m convinced most live in a perpetual state of anxiety. The log
cabin with a privy, a fireplace, and a bucket for hauling water has been
replaced by homes with sophisticated wiring, plumbing, and appliances. What
happens if something goes wrong?</p>
</blockquote>
<p>Beyond the child-like glee how looking inside things and figuring out how
things work … I learned <strong>many things</strong> I didn’t know. If that sounds like your
kind of thing, you’ll be delighted too.</p>
<ul>
<li><a href="https://www.amazon.com/dp/1119467616/">amazon</a></li>
<li><a href="https://www.rsmeans.com/products/books/reference-books/how-your-house-works-3rd-edition">website</a></li>
</ul>
<hr />
<h2 id="fiction">Fiction</h2>
<h3 id="the-ministry-for-the-future">The Ministry for the Future</h3>
<p><a href="https://www.amazon.com/dp/B08K1YZBYN/"><img class="book-cover" src="/assets/best-books-2021/B08K1YZBYN.jpg" alt="The Ministry for the Future" /></a></p>
<p>I’ve read, and recommended, almost everything that <a href="https://en.wikipedia.org/wiki/Kim_Stanley_Robinson">Kim Stanley Robinson</a> wrote. He mentioned that it might be his last big novel… 😢</p>
<p>When <a href="https://en.wikipedia.org/wiki/The_Ministry_for_the_Future">Ministry for the
Future</a> came out, it
was hard to miss: there were a
<a href="https://jacobinmag.com/2020/10/kim-stanley-robinson-ministry-future-science-fiction">bunch</a>
<a href="https://ew.com/books/the-ministry-for-the-future-climate-change/">of</a>
<a href="https://www.thenation.com/article/culture/qa-kim-stanley-robinson/">articles</a>
about it. A quote struck me:</p>
<blockquote>
<p>It is easier to imagine the end of the world than to imagine the end of capitalism.</p>
</blockquote>
<p>If capitalism is a <a href="https://en.wikipedia.org/wiki/Late_capitalism">dead end</a>, because its externalities will eventually kill us all, what are the alternatives?
And if there are none, or we can’t imagine them, things are grim indeed.</p>
<p>The <a href="https://www.theguardian.com/commentisfree/2021/dec/09/climate-crisis-kim-stanley-robinson">book came up again</a> recently, to frame some of the recent climate disasters.</p>
<p>As a society, we are going to need to make some tough decisions. And
I believe things are going to be bad. This book can give you an idea of what’s to come.</p>
<ul>
<li><a href="https://www.amazon.com/dp/B08K1YZBYN/">amazon</a></li>
<li><a href="https://en.wikipedia.org/wiki/The_Ministry_for_the_Future">wikipedia</a></li>
</ul>
<h3 id="inhibitor-phase">Inhibitor Phase</h3>
<p><a href="https://www.amazon.com/dp/B09F92QRBP/"><img class="book-cover" src="/assets/best-books-2021/B09F92QRBP.jpg" alt="Inhibitor Phase" /></a></p>
<p><strong>Inhibitor Phase</strong> is book 7 of the <a href="https://en.wikipedia.org/wiki/Revelation_Space">Revelation Space</a> series.
I <a href="https://blog.jpalardy.com/posts/best-books-i-read-2015/#revelation-space">already gave the first books</a>
my seal of approval. It was good then and it’s still good now.</p>
<p>This year, I first read <a href="https://www.amazon.com/dp/B01282ENRS/">Diamond Dogs, Turquoise Days</a> (book 6). It contained two excellent short stories, on the sidelines of the main story. It threw me back into that universe.</p>
<p>The prologue mentions that <strong>Inhibitor Phase</strong> could be a self-standing book; not necessarily requiring having read the rest of the series. I’m not 100% sure that’s true 🤔</p>
<p>It can be intimidating to jump into such a long series; not knowing exactly what you are getting yourself into… I would still recommend giving it a try; starting from the beginning.</p>
<ul>
<li><a href="https://www.amazon.com/dp/B09F92QRBP/">amazon</a></li>
</ul>
<hr />
<h2 id="closing-thoughts">Closing Thoughts</h2>
<p>If you have liked this blog post, please write your own. I want to read your book
reviews and recommendations.</p>
Reading Raw Text into jq2021-11-03T00:00:00+00:00https://blog.jpalardy.com/posts/reading-raw-text-into-jq<p>How do you read non-JSON inputs with jq?</p>
<p>This isn’t obvious if you don’t <em>already</em> know how to do it…</p>
<p><a href="https://stedolan.github.io/jq/manual/">The manual</a> says:</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">--raw-input/-R:
Don't parse the input as JSON. Instead, each line of text is passed to the
filter as a string. If combined with --slurp, then the entire input is passed
to the filter as a single long string.</code></pre></figure>
<p>Clear yet?</p>
<h2 id="an-example">An Example</h2>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">cat </span>fruits.txt
apple
banana
cherry
kiwi
orange
peach
<span class="nv">$ </span><span class="nb">cat </span>fruits.txt | jq <span class="nt">--raw-input</span> <span class="nb">.</span>
<span class="s2">"apple"</span>
<span class="s2">"banana"</span>
<span class="s2">"cherry "</span>
<span class="s2">"kiwi"</span>
<span class="s2">"orange"</span>
<span class="s2">"peach"</span></code></pre></figure>
<p>The lines in <code class="language-plaintext highlighter-rouge">fruits.txt</code> are “naked” strings; not JSON strings (missing double
quotes). What <code class="language-plaintext highlighter-rouge">--raw-input</code> does is wrap your lines in double quotes: notice
the dangling spaces after <code class="language-plaintext highlighter-rouge">cherry</code>.</p>
<p>If you try to load <code class="language-plaintext highlighter-rouge">fruits.txt</code> without <code class="language-plaintext highlighter-rouge">--raw-input</code>, you get:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">cat </span>fruits.txt | jq <span class="nb">.</span>
parse error: Invalid numeric literal at line 2, column 0</code></pre></figure>
<p>Once “bootstrapped” into jq, you can treat it as any JSON input:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">cat </span>fruits.txt | jq <span class="nt">--raw-input</span> <span class="s1">'{(.): (. | length)}'</span> <span class="nt">-c</span>
<span class="o">{</span><span class="s2">"apple"</span>:5<span class="o">}</span>
<span class="o">{</span><span class="s2">"banana"</span>:6<span class="o">}</span>
<span class="o">{</span><span class="s2">"cherry "</span>:10<span class="o">}</span>
<span class="o">{</span><span class="s2">"kiwi"</span>:4<span class="o">}</span>
<span class="o">{</span><span class="s2">"orange"</span>:6<span class="o">}</span>
<span class="o">{</span><span class="s2">"peach"</span>:5<span class="o">}</span></code></pre></figure>
<h2 id="slurp">Slurp</h2>
<p>Technically, <code class="language-plaintext highlighter-rouge">fruits.txt</code> contains 6 lines: it becomes 6 JSON items, treated separately.</p>
<p>If you’ve done enough jq, your solution is to <code class="language-plaintext highlighter-rouge">--slurp</code> it, to get an array of strings:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">cat </span>fruits.txt | jq <span class="nt">--raw-input</span> <span class="nt">--slurp</span> <span class="nb">.</span>
<span class="s2">"apple</span><span class="se">\n</span><span class="s2">banana</span><span class="se">\n</span><span class="s2">cherry </span><span class="se">\n</span><span class="s2">kiwi</span><span class="se">\n</span><span class="s2">orange</span><span class="se">\n</span><span class="s2">peach</span><span class="se">\n</span><span class="s2">"</span></code></pre></figure>
<p>Oh no … 😬</p>
<p>As per the <code class="language-plaintext highlighter-rouge">--raw-input</code> documentation (helpfully at the top of this
post), <code class="language-plaintext highlighter-rouge">--slurp</code> works surprisingly for this case… The whole input, all the
lines, is considered as one giant string.</p>
<p>What’s the workaround? As far as I could tell, you are going to need two jq commands to make this work:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">cat </span>fruits.txt | jq <span class="nt">--raw-input</span> <span class="nb">.</span> | jq <span class="nt">--slurp</span> <span class="nb">.</span>
<span class="o">[</span>
<span class="s2">"apple"</span>,
<span class="s2">"banana"</span>,
<span class="s2">"cherry "</span>,
<span class="s2">"kiwi"</span>,
<span class="s2">"orange"</span>,
<span class="s2">"peach"</span>
<span class="o">]</span>
<span class="nv">$ </span><span class="nb">cat </span>fruits.txt | jq <span class="nt">--raw-input</span> <span class="nb">.</span> | jq <span class="nt">--slurp</span> <span class="nb">.</span> <span class="nt">-c</span>
<span class="o">[</span><span class="s2">"apple"</span>,<span class="s2">"banana"</span>,<span class="s2">"cherry "</span>,<span class="s2">"kiwi"</span>,<span class="s2">"orange"</span>,<span class="s2">"peach"</span><span class="o">]</span></code></pre></figure>
<h2 id="discussion">Discussion</h2>
<p>jq is wonderful once you have JSON … but a lot of command-line tools don’t produce JSON natively.</p>
<p>Using this technique, you can uplift raw text into JSON and modify it into the
shape you need. All the while, you are guaranteed to keep and generate valid
JSON outputs.</p>
<p>The alternatives of text manipulation with sed/awk/perl/etc are (at best!) too error-prone.</p>
Software I like: Chrome Coverage Tab2021-10-07T00:00:00+00:00https://blog.jpalardy.com/posts/software-i-like-chrome-coverage-tab<p><a href="https://prodocs24.com/articles/chrome-coverage-tab/">Azerbaijanian Translation</a>, thanks to Amir.<br />
<a href="http://expereb.com/software-que-me-gusta-chrome-coverage-tab/">Spanish Translation</a>, thanks to Laura.</p>
<p>While both Chrome and Firefox have great developer tools; I had <em>almost</em> forgotten
that Chrome has a <a href="https://developer.chrome.com/docs/devtools/coverage/">Coverage Tab</a> ~ I hadn’t needed it in a long time.</p>
<p><img src="/assets/like-coverage/coverage.webp" alt="coverage tab screenshot" /></p>
<p>It does both JavaScript and CSS; although I’ve only used the CSS part. For
JavaScript, it is often easier to instrument procedural code with
<code class="language-plaintext highlighter-rouge">console.log</code>.</p>
<h2 id="how-to-use-it">How to Use it?</h2>
<p>I think the instructions in the <a href="https://developer.chrome.com/docs/devtools/coverage/">documentation</a> (same link as above) are pretty good.</p>
<h2 id="my-experience">My Experience</h2>
<p>I recently inherited an application that someone else wrote. If CSS is hard to
understand, someone else’s CSS can be a real challenge. This tweet felt
especially relevant:</p>
<p><img src="/assets/like-coverage/css-on-fire.jpg" alt="css is hell" /></p>
<p>With the coverage tab, you get an idea of what’s used and what isn’t:</p>
<ul>
<li>everything that’s red is a candidate for deletion</li>
<li>everything that’s green is a candidate for replacement</li>
</ul>
<p>One thing to keep in mind: the coverage tab is <em>dynamic</em> … something highlighted in red might turn green
when you navigate to different part of the page – or when you activate it; either through hover, or some JavaScript action.</p>
<p>The coverage tab is more of a “guide” than a guarantee… Something green is definitely in use; while something red is not <em>currently</em> used.</p>
<h2 id="where-is-it">Where is it?</h2>
<p>It’s not easily <em>discoverable</em> – If you don’t know it’s there, it’s almost hopeless… (shame 😢)</p>
<p>Open <code class="language-plaintext highlighter-rouge">Developer Tools</code> and follow the clicks:</p>
<p><img src="/assets/like-coverage/coverage-where.png" alt="finding the coverage tab in chrome" /></p>
<p>(<em>sidenote: whoah… there’s a lot more tools here than I expected</em>)</p>
Elixir Notes: Module Names and File Names Mismatch2021-08-19T00:00:00+00:00https://blog.jpalardy.com/posts/elixir-notes-module-names-and-file-names-mismatch<p>Short version: module names and file names don’t have to match! There is a convention, but <code class="language-plaintext highlighter-rouge">mix</code> doesn’t enforce it.</p>
<h2 id="background">Background</h2>
<p>I became fully aware of this after I renamed a directory under <code class="language-plaintext highlighter-rouge">lib/</code>, but everything kept working…</p>
<p>This wasn’t what I expected. I ran my tests <em>expecting</em> to be told what I had broken.</p>
<p>At first, I thought it was a caching problem. I removed the
<code class="language-plaintext highlighter-rouge">_build</code> and <code class="language-plaintext highlighter-rouge">deps</code> directories and … everything still worked!</p>
<h2 id="why-believe-that-names-are-important">Why believe that names are important?</h2>
<p>It was partly my experience with other languages. Coming from Go (or Java), we expect moving files around to break the build.</p>
<p>But, mostly, it was <em>BECAUSE</em> most Elixir projects and examples are so
well-behaved! Inside a file named <code class="language-plaintext highlighter-rouge">lib/some/thing.ex</code>, we expect to find a module named <code class="language-plaintext highlighter-rouge">Some.Thing</code>.</p>
<p><img src="/assets/module-file-mismatch/elixir-books.png" alt="Elixir and Phoenix books" /></p>
<p>This naming is used consistently in all the Elixir and Phoenix books I read.</p>
<h2 id="its-a-convention">It’s a convention…</h2>
<p>If you’re feeling unsatisfied/surprised by this behavior, you’re not alone. I found <a href="https://elixirforum.com/t/module-naming-conventions/16636">a thread</a> by Joe Armstrong (no less!) discussing this. And here’s the answer, from José Valim himself:</p>
<p><img src="/assets/module-file-mismatch/jose-1.png" alt="Jose Valim's answer on ElixirForum" /></p>
<p>Here’s <a href="https://elixirforum.com/t/module-naming-conventions-and-paths/1836">another thread</a> on the topic, and José’s answer:</p>
<p><img src="/assets/module-file-mismatch/jose-2.png" alt="Jose Valim's answer on ElixirForum" /></p>
<p>Finally, here’s the text from the <a href="https://hexdocs.pm/elixir/1.12/naming-conventions.html">Naming Convention</a> reference:</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">Generally speaking, filenames follow the snake_case convention of the module
they define. For example, MyApp should be defined inside the my_app.ex file.
However, this is only a convention. At the end of the day, any filename can
be used as they do not affect the compiled code in any way.</code></pre></figure>
<h2 id="detour-a-naming-recipe">Detour: a naming recipe</h2>
<p>How to convert from a full path to a module name?</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>e.g. lib/text_client/prompter.ex -> TextClient.Prompter
</code></pre></div></div>
<ul>
<li>start with a full path (dir + file name)</li>
<li>remove the leading <code class="language-plaintext highlighter-rouge">lib/</code> or <code class="language-plaintext highlighter-rouge">test/</code> directory</li>
<li>remove the extension</li>
<li>capitalize each word</li>
<li>replace <code class="language-plaintext highlighter-rouge">/</code> with <code class="language-plaintext highlighter-rouge">.</code></li>
<li>remove all <code class="language-plaintext highlighter-rouge">_</code></li>
</ul>
<h2 id="implications">Implications</h2>
<p>I read <em>“it doesn’t matter”</em> many times, but never understanding <em>EXACTLY</em> what that meant in practice.</p>
<p>I ran a few experiments to convince myself of the behavior. Here’s my new mental model:</p>
<ul>
<li>mix reads all <code class="language-plaintext highlighter-rouge">.ex</code> files under <code class="language-plaintext highlighter-rouge">lib/</code>, anywhere</li>
<li>all these files <em>could be</em> one big concatenated file</li>
<li>all defined modules get compiled into their own <code class="language-plaintext highlighter-rouge">.beam</code> file</li>
</ul>
<p>What does that imply?</p>
<p><strong>The name of a module and the file name where it is defined don’t need to be related:</strong></p>
<ul>
<li><code class="language-plaintext highlighter-rouge">Dog</code> module defined in <code class="language-plaintext highlighter-rouge">cat.ex</code></li>
</ul>
<p><strong>The directory structure doesn’t matter:</strong></p>
<ul>
<li><code class="language-plaintext highlighter-rouge">Animal.Mammal.Dog</code> module defined in <code class="language-plaintext highlighter-rouge">dog.ex</code></li>
<li><code class="language-plaintext highlighter-rouge">Dog</code> module defined in <code class="language-plaintext highlighter-rouge">animal/mammal/dog.ex</code></li>
</ul>
<p><strong>Multiple modules can be defined in one file:</strong></p>
<ul>
<li><code class="language-plaintext highlighter-rouge">Dog</code> and <code class="language-plaintext highlighter-rouge">Cat</code> modules both defined in <code class="language-plaintext highlighter-rouge">animals.ex</code></li>
</ul>
<p><strong>Combinations of the above:</strong></p>
<ul>
<li><code class="language-plaintext highlighter-rouge">Animal.Mammal.Dog</code> and <code class="language-plaintext highlighter-rouge">Cat</code> modules both defined in <code class="language-plaintext highlighter-rouge">some/thing/unrelated.ex</code></li>
</ul>
<h2 id="is-this-a-good-thing-or-a-bad-thing">Is this a good thing or a bad thing?</h2>
<p>I was looking for understanding: what is allowed and not allowed. I had been working with Elixir for months
before <em>REALLY</em> wondering how this worked… I had been following the naming convention and everything was great.</p>
<p>But <a href="https://en.wikipedia.org/wiki/With_great_power_comes_great_responsibility">with great power comes great responsibility</a>:
the naming convention exists so that we don’t have to grep around all the time
– use the convention, please. The alternative is complete chaos…</p>
<p>(that being said, there are times when it’s useful to bend the rules)</p>
<p>Back to my original problem – it would have been useful to “detect” that I had moved some files to a different directory and that <em>some</em> renaming would be necessary to, once again, comply with the convention.</p>
<p>I wondered if there were tools to help catch this (and briefly considered writing my own). In the end, I found that <a href="https://github.com/mirego/credo_naming">credo_naming</a> will detect exactly these types of mismatches.</p>
Elixir Notes: Mix Tasks and @shortdoc2021-07-04T00:00:00+00:00https://blog.jpalardy.com/posts/elixir-notes-mix-tasks-and-shortdoc<p>I had not tried to write my own mix tasks until this week.
The <a href="https://hexdocs.pm/mix/Mix.Task.html">Mix.Task documentation</a> made it look ridiculously easy:</p>
<p><img src="/assets/mix-tasks-shortdoc/mix-task.png" alt="Mix.Task documentation screenshot" /></p>
<p>In summary:</p>
<ul>
<li>create a file under <code class="language-plaintext highlighter-rouge">lib/tasks</code></li>
<li><code class="language-plaintext highlighter-rouge">use Mix.Task</code></li>
<li>define a <code class="language-plaintext highlighter-rouge">run/1</code> function – receives the command-line arguments (as a list of strings)</li>
</ul>
<p>And… it worked. I felt <a href="https://headrush.typepad.com/creating_passionate_users/2005/05/users_dont_care.html">awesome</a>.</p>
<h2 id="a-small-problem">A Small Problem</h2>
<p>My new task didn’t show up with the others in <code class="language-plaintext highlighter-rouge">mix help</code>. A quick search took
me, <em>ironically</em>, 3-4 paragraphs below the above screenshot, on the same page 😅</p>
<blockquote>
<p>Define the @shortdoc attribute if you wish to make the task publicly visible
on mix help. Omit this attribute if you do not want your task to be listed
via mix help.</p>
</blockquote>
<p>Except … that didn’t work…</p>
<p>⚠️ note that <code class="language-plaintext highlighter-rouge">mix help</code> doesn’t recompile the code … try <code class="language-plaintext highlighter-rouge">mix compile</code> first, to ensure your changes show up.</p>
<h2 id="deep-dive">Deep Dive</h2>
<p>I couldn’t find a quick answer, so I cloned <a href="https://github.com/elixir-lang/elixir">the Elixir code</a>
and started to grep.<br />
(specifically: <code class="language-plaintext highlighter-rouge">rg '[^@]shortdoc'</code>)</p>
<p>I found this:</p>
<p><img src="/assets/mix-tasks-shortdoc/build-task-doc-list.png" alt="code that checks for @shortdoc" /></p>
<p>As per line 178 (@ elixir 1.12), <code class="language-plaintext highlighter-rouge">Mix.Task.shortdoc(module)</code> is used to filter qualifying tasks. What did <code class="language-plaintext highlighter-rouge">iex</code> have to say about that?</p>
<p><img src="/assets/mix-tasks-shortdoc/iex-check.png" alt="check in iex" /></p>
<p>Hmmm… it works?! 🤔</p>
<p>It took <em>a lot</em> more digging, but here’s the complete picture:</p>
<p><a href="/assets/mix-tasks-shortdoc/mystery-solved.png"><img src="/assets/mix-tasks-shortdoc/mystery-solved.png" alt="complete code picture" /></a>
(<em>click to enlarge</em>)</p>
<ul>
<li>yes, <code class="language-plaintext highlighter-rouge">@shortdoc</code> comes into play to filter the list, at #3</li>
<li>but, before that, the listed tasks are filtered by <code class="language-plaintext highlighter-rouge">@moduledoc</code> also…</li>
</ul>
<h2 id="lessons-learned">Lessons Learned?</h2>
<p>I found out about the behavior of <code class="language-plaintext highlighter-rouge">@moduledoc</code> before I found it in the code. That certainly <em>directed</em> my search. The reasons that I kept digging:</p>
<ul>
<li>as always, I found the Elixir code to be approachable 💖</li>
<li>I couldn’t find <em>WHY</em> the <code class="language-plaintext highlighter-rouge">@moduledoc</code> had to be present – which seemed like a learning opportunity</li>
<li>I couldn’t find documentation that explicitly explained this behavior – <code class="language-plaintext highlighter-rouge">@moduledoc</code> is said to provide the documentation printed by <code class="language-plaintext highlighter-rouge">mix help taskname</code></li>
</ul>
Life After KeyCode2021-05-21T00:00:00+00:00https://blog.jpalardy.com/posts/life-after-keycode<p>Over the years, I wrote a bunch of quick JavaScript apps with code similar to:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="nb">document</span><span class="p">.</span><span class="nx">addEventListener</span><span class="p">(</span><span class="dl">"</span><span class="s2">keydown</span><span class="dl">"</span><span class="p">,</span> <span class="nx">ev</span> <span class="o">=></span> <span class="p">{</span>
<span class="c1">// ESC</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">ev</span><span class="p">.</span><span class="nx">keyCode</span> <span class="o">===</span> <span class="mi">27</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// ...</span>
<span class="p">}</span>
<span class="c1">// Enter</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">ev</span><span class="p">.</span><span class="nx">keyCode</span> <span class="o">===</span> <span class="mi">13</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// ...</span>
<span class="p">}</span>
<span class="c1">// ...</span></code></pre></figure>
<p>but I found out recently that <a href="https://developer.mozilla.org/en-US/docs/Web/API/KeyboardEvent/keyCode">keyCode is deprecated</a>:</p>
<p><img src="/assets/keycode/keycode-deprecated.png" alt="deprecation warning from MDN" /></p>
<h2 id="what-now">What now?</h2>
<p>There are better options! Let’s have a quick look at available <a href="https://developer.mozilla.org/en-US/docs/Web/API/KeyboardEvent">KeyboardEvent</a> properties:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="nb">document</span><span class="p">.</span><span class="nx">addEventListener</span><span class="p">(</span><span class="dl">"</span><span class="s2">keydown</span><span class="dl">"</span><span class="p">,</span> <span class="p">(</span><span class="nx">ev</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">({</span>
<span class="na">code</span><span class="p">:</span> <span class="nx">ev</span><span class="p">.</span><span class="nx">code</span><span class="p">,</span>
<span class="na">key</span><span class="p">:</span> <span class="nx">ev</span><span class="p">.</span><span class="nx">key</span><span class="p">,</span>
<span class="na">keyCode</span><span class="p">:</span> <span class="nx">ev</span><span class="p">.</span><span class="nx">keyCode</span><span class="p">,</span>
<span class="na">which</span><span class="p">:</span> <span class="nx">ev</span><span class="p">.</span><span class="nx">which</span><span class="p">,</span>
<span class="p">});</span>
<span class="p">});</span></code></pre></figure>
<p>and try pressing:</p>
<ul>
<li>esc</li>
<li>right square bracket</li>
<li>space</li>
<li>“arrow right”</li>
<li>lowercase n</li>
<li>uppercase n</li>
</ul>
<p><img src="/assets/keycode/event-log.png" alt="screenshot of keydown properties" /></p>
<h2 id="observations-keycode-and-which">Observations: keyCode and which</h2>
<p>Both <a href="https://developer.mozilla.org/en-US/docs/Web/API/KeyboardEvent/keyCode">keyCode</a>
and <a href="https://developer.mozilla.org/en-US/docs/Web/API/KeyboardEvent/which">which</a>
are deprecated.</p>
<p>Also, I can’t see a difference in values (using Chrome 90), but it seems like it might
<a href="https://stackoverflow.com/questions/4471582/keycode-vs-which/4471635">differ</a>
<a href="https://thisthat.dev/key-vs-key-code-vs-which/">between</a>
<a href="https://stackoverflow.com/questions/19249351/what-is-the-difference-between-e-keycode-and-e-which">browsers</a>…
And having worked with them before, and their numeric outputs, I would say that’s good riddance.</p>
<h2 id="observations-code">Observations: code</h2>
<p>The other difference that stood out was the handling of <code class="language-plaintext highlighter-rouge">shift</code>. Both
upper/lower <code class="language-plaintext highlighter-rouge">n</code> registered as <code class="language-plaintext highlighter-rouge">KeyN</code> for the <a href="https://developer.mozilla.org/en-US/docs/Web/API/KeyboardEvent/code">code</a>
property. That would put the responsibility of checking the <a href="https://developer.mozilla.org/en-US/docs/Web/API/KeyboardEvent/shiftKey">shiftKey</a>
property ourselves, to differentiate the events. 😩</p>
<p>That’s a bit more surprising for keys we don’t typically associate as “being the same”:</p>
<p><img src="/assets/keycode/bracketRight.png" alt="bracket, but not curly bracket..." /></p>
<p>Are <code class="language-plaintext highlighter-rouge">]</code> and <code class="language-plaintext highlighter-rouge">}</code> both “BracketRight”, with only a difference in <code class="language-plaintext highlighter-rouge">shift</code>? 🤔<br />
That doesn’t sound right to me…</p>
<p>(is <code class="language-plaintext highlighter-rouge">#</code> a shifted <code class="language-plaintext highlighter-rouge">Digit3</code>? … on all keyboards, all the time?)</p>
<h2 id="observations-key">Observations: key</h2>
<p>Finally, the <a href="https://developer.mozilla.org/en-US/docs/Web/API/KeyboardEvent/key">key</a> property seems to be the way to go.</p>
<ul>
<li>it generates what I would expect (usually)</li>
<li>including handling <code class="language-plaintext highlighter-rouge">shift</code></li>
<li>isn’t numeric (which reeked of magic numbers)</li>
</ul>
<p>The downside would be differentiating a <code class="language-plaintext highlighter-rouge">left/right</code> with <code class="language-plaintext highlighter-rouge">shift/ctrl/alt/meta</code> – a case that I don’t usually handle. (though something to keep in mind)</p>
<p>If you’re wondering exactly what you’re going to get when pressing a key, check out the KeyboardEvent <a href="https://developer.mozilla.org/en-US/docs/Web/API/KeyboardEvent/key/Key_Values">Key Values</a> page.</p>
<h2 id="sidenote-shiftctrlaltmeta">Sidenote: shift/ctrl/alt/meta</h2>
<p><a href="https://developer.mozilla.org/en-US/docs/Web/API/KeyboardEvent">KeyboardEvent</a> has boolean properties for “control keys”:</p>
<ul>
<li><a href="https://developer.mozilla.org/en-US/docs/Web/API/KeyboardEvent/altKey">altKey</a></li>
<li><a href="https://developer.mozilla.org/en-US/docs/Web/API/KeyboardEvent/ctrlKey">ctrlKey</a></li>
<li><a href="https://developer.mozilla.org/en-US/docs/Web/API/KeyboardEvent/metaKey">metaKey</a> – aka: “windows”, “command” keys</li>
<li><a href="https://developer.mozilla.org/en-US/docs/Web/API/KeyboardEvent/shiftKey">shiftKey</a></li>
</ul>
<p>which I haven’t historically been great at keeping track of… It’s easy to trigger on the <code class="language-plaintext highlighter-rouge">a</code> key, but to forget to check the <code class="language-plaintext highlighter-rouge">metaKey</code> status and prevent <code class="language-plaintext highlighter-rouge">cmd-a</code> from triggering normally. (usually due to an eager <a href="https://developer.mozilla.org/en-US/docs/Web/API/Event/preventDefault">preventDefault</a>). Which leads me to…</p>
<h2 id="quick-tip-key-combinations">Quick tip: key-combinations</h2>
<p>Since I stumbled on the <code class="language-plaintext highlighter-rouge">keyCode</code> deprecation, I’ve been refactoring my apps. I’ve been happy with this snippet of code:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="nb">document</span><span class="p">.</span><span class="nx">addEventListener</span><span class="p">(</span><span class="dl">"</span><span class="s2">keydown</span><span class="dl">"</span><span class="p">,</span> <span class="nx">ev</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">keys</span> <span class="o">=</span> <span class="p">[</span>
<span class="nx">ev</span><span class="p">.</span><span class="nx">metaKey</span> <span class="o">&&</span> <span class="dl">"</span><span class="s2">Meta</span><span class="dl">"</span><span class="p">,</span>
<span class="nx">ev</span><span class="p">.</span><span class="nx">ctrlKey</span> <span class="o">&&</span> <span class="dl">"</span><span class="s2">Ctrl</span><span class="dl">"</span><span class="p">,</span>
<span class="nx">ev</span><span class="p">.</span><span class="nx">altKey</span> <span class="o">&&</span> <span class="dl">"</span><span class="s2">Alt</span><span class="dl">"</span><span class="p">,</span>
<span class="nx">ev</span><span class="p">.</span><span class="nx">key</span><span class="p">,</span>
<span class="p">].</span><span class="nx">filter</span><span class="p">((</span><span class="nx">v</span><span class="p">)</span> <span class="o">=></span> <span class="nx">v</span><span class="p">).</span><span class="nx">join</span><span class="p">(</span><span class="dl">"</span><span class="s2">-</span><span class="dl">"</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">keys</span> <span class="o">===</span> <span class="dl">"</span><span class="s2">a</span><span class="dl">"</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// ...</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">keys</span> <span class="o">===</span> <span class="dl">"</span><span class="s2">Meta-a</span><span class="dl">"</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// ...</span>
<span class="p">}</span>
<span class="p">});</span></code></pre></figure>
<p>which keeps these boolean cases more explicit and, thankfully, mutually exclusive.</p>
<p>Some things to consider:</p>
<ul>
<li>the code depends on the behavior of <code class="language-plaintext highlighter-rouge">shift</code> discussed above – so it’s already “handled”</li>
<li><code class="language-plaintext highlighter-rouge">altKey</code> is “problematic” on the Mac … it might unlock alternative symbols, for example <code class="language-plaintext highlighter-rouge">Alt-z</code> becomes <code class="language-plaintext highlighter-rouge">Alt-Ω</code> … a kind of “feature”.</li>
<li>there are cases where I would consider <code class="language-plaintext highlighter-rouge">Meta-a</code> and <code class="language-plaintext highlighter-rouge">Ctrl-a</code> to be the same; what a Windows user vs a Mac user would expect to type. It would be relatively easy to handle this.</li>
</ul>
Rebase a Tmux Session2021-04-08T00:00:00+00:00https://blog.jpalardy.com/posts/rebase-a-tmux-session<p><a href="http://expereb.com/reajustar-de-una-sesion-de-tmux/">Spanish Translation</a>, thanks to Laura.<br />
<a href="http://lpacode.com/rebase-a-tmux-session/">Georgian Translation</a>, thanks to Ana.<br />
<a href="https://guideslib.com/publications/rebase-a-tmux-session/">Bulgarian Translation</a>, thanks to Zlatan.<br />
<a href="https://www.ibidemgroup.com/edu/traduccion-frances-ingenieria-software/">French Translation</a>, thanks to Chema.</p>
<p>Whatever directory you start a new Tmux session in, that’s the directory that
will be used for each new window you create.</p>
<p>Tmux is the workhorse of my local development workflow. When I <code class="language-plaintext highlighter-rouge">cd</code>
into a directory and start working outside a Tmux session, it <em>feels</em> like
something is missing.</p>
<p>There’s always a need to get a new shell to check something or run a quick
test. I open a bunch of windows: some permanent (which I usually name), most disposable.</p>
<p><img src="/assets/rebase-tmux/session-with-named-windows.png" alt="example tmux session with named windows" /></p>
<h2 id="the-problem">The Problem</h2>
<p>Once the session is started, the “start directory” is set and fixed.</p>
<p>If, in the middle of a session, I needed to focus on a subdirectory, or move
elsewhere, I had several sub-optimal choices:</p>
<ul>
<li>explicitly <code class="language-plaintext highlighter-rouge">cd</code> to the new directory, after opening each new window</li>
<li>exit the session, and start a new one in the new directory</li>
<li>start a new session, in another terminal, in the new directory</li>
</ul>
<p>This had happened enough times to justify looking for a better way…</p>
<h2 id="the-solution">The Solution</h2>
<p>Add this to your <code class="language-plaintext highlighter-rouge">$HOME/.tmux.conf</code> (<a href="https://github.com/jpalardy/dotfiles/blob/54c2d416f2a291c6c7f932d9d930b560f819c828/tmux.conf#L22">here’s mine</a>):</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">bind _ attach-session -t . -c '#{pane_current_path}'</code></pre></figure>
<p>(you might need to restart Tmux, or reload its config)</p>
<p>My prefix is <code class="language-plaintext highlighter-rouge">ctrl-a</code>, and this binds a command to the underscore: <code class="language-plaintext highlighter-rouge">_</code>. The
<code class="language-plaintext highlighter-rouge">pane_current_path</code> is the <code class="language-plaintext highlighter-rouge">$PWD</code> of the window you are using when invoking the
binding. <a href="https://codelearn.me/2018/12/14/tmux-change-default-start-dir.html">You can do this from Tmux’s command mode too</a>.</p>
<p>The underscore made sense to me, but feel free to customize the binding.</p>
<h2 id="experience">Experience</h2>
<p>Being able to “re-use” my sessions has been very convenient. I don’t spend too
much time thinking about “the best” directory to start a session from; I can
always fix it later.</p>
cd.. Until .git Directory2021-03-21T00:00:00+00:00https://blog.jpalardy.com/posts/cd-until-git-directory<p>For years, I’ve been solving the wrong problem:</p>
<blockquote>
<p>how do I <code class="language-plaintext highlighter-rouge">cd..</code> efficiently?</p>
</blockquote>
<p>I realized that my <strong>real</strong> problem was:</p>
<blockquote>
<p>how do I <code class="language-plaintext highlighter-rouge">cd..</code> back to a project’s root directory?</p>
</blockquote>
<p>The insight, obvious in retrospect, is that most projects have a <code class="language-plaintext highlighter-rouge">.git</code> directory
at their base.</p>
<h2 id="the-manual-way">The Manual Way…</h2>
<p>I’ve played with multiple solutions over the years, the fully <a href="https://github.com/jpalardy/dotfiles/blob/03099574c68edfb56e21ff8fec4bf3da09339ac7/zsh/commands/cd.zsh#L16-L18">manual one</a>:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># almost this, see link above</span>
<span class="nb">alias </span><span class="nv">b</span><span class="o">=</span><span class="s2">"cd .."</span>
<span class="c"># it's easy to type :-)</span></code></pre></figure>
<p>I also tried a <a href="https://github.com/jpalardy/dotfiles/blob/03099574c68edfb56e21ff8fec4bf3da09339ac7/zsh/commands/cd.zsh#L21-29">pick from a list</a> one:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># pick parent dir with fzf</span>
bu<span class="o">()</span> <span class="o">{</span>
run-not-blank <span class="nb">cd</span> <span class="si">$(</span>
<span class="nb">local </span><span class="nv">p</span><span class="o">=</span><span class="nv">$PWD</span>
<span class="k">while</span> <span class="o">[</span> <span class="nv">$p</span> <span class="o">!=</span> <span class="s2">"/"</span> <span class="o">]</span><span class="p">;</span> <span class="k">do
</span><span class="nb">echo</span> <span class="nv">$p</span>
<span class="nv">p</span><span class="o">=</span><span class="k">${</span><span class="nv">p</span>:h<span class="k">}</span>
<span class="k">done</span> | fzf
<span class="si">)</span>
<span class="o">}</span></code></pre></figure>
<p>which looks like this (not bad):</p>
<p><img src="/assets/cd-until-git/bu.png" alt="bu example" /></p>
<p>I’ve been relatively happy with a combination of these. The problem with both is how “fragile”
they feel. If I <code class="language-plaintext highlighter-rouge">cd</code> to the wrong place, getting back to the right directory is
enough effort to break me out of the flow.</p>
<h2 id="the-automated-way">The Automated Way</h2>
<p>Let the script look “backward” up the parent directories, looking for the first <code class="language-plaintext highlighter-rouge">.git</code> directory it can find:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># cd .. until a .git directory is found</span>
cd_<span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">p</span><span class="o">=</span><span class="nv">$PWD</span>
<span class="k">while</span> <span class="o">[</span> <span class="nv">$p</span> <span class="o">!=</span> <span class="s2">"/"</span> <span class="o">]</span><span class="p">;</span> <span class="k">do
if</span> <span class="o">[</span> <span class="nt">-d</span> <span class="s2">"</span><span class="nv">$p</span><span class="s2">/.git"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
</span><span class="nb">cd</span> <span class="s2">"</span><span class="nv">$p</span><span class="s2">"</span>
<span class="nb">break
</span><span class="k">fi
</span><span class="nv">p</span><span class="o">=</span><span class="k">${</span><span class="nv">p</span>:h<span class="k">}</span>
<span class="k">done</span>
<span class="o">}</span>
<span class="c"># won't cd unless a .git directory is found</span>
<span class="c"># can be used multiple times if multiple .git parents are expected</span></code></pre></figure>
<p>I had already “extended” the <code class="language-plaintext highlighter-rouge">cd</code> command with a wrapper function. For the sake of my muscle memory, I decided to delegate <code class="language-plaintext highlighter-rouge">cd _</code> to <code class="language-plaintext highlighter-rouge">cd_</code>:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># wrap `cd`, add behaviors</span>
<span class="nb">cd</span><span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">dest</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="p">@</span><span class="k">:-</span><span class="s2">"</span><span class="nv">$HOME</span><span class="s2">"</span><span class="k">}</span><span class="s2">"</span>
<span class="k">if</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$dest</span><span class="s2">"</span> <span class="o">=</span> <span class="s2">"_"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
</span>cd_ <span class="s2">"</span><span class="nv">$dest</span><span class="s2">"</span>
<span class="k">return
fi
if</span> <span class="o">[</span> <span class="nt">-f</span> <span class="s2">"</span><span class="nv">$dest</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
</span><span class="nv">dest</span><span class="o">=</span><span class="k">${</span><span class="nv">dest</span>:h<span class="k">}</span>
<span class="k">fi
</span><span class="nb">builtin cd</span> <span class="s2">"</span><span class="nv">$dest</span><span class="s2">"</span>
<span class="o">}</span></code></pre></figure>
<p>both functions are available in <a href="https://github.com/jpalardy/dotfiles/blob/03099574c68edfb56e21ff8fec4bf3da09339ac7/zsh/commands/cd.zsh">cd.zsh</a> from my <a href="https://github.com/jpalardy/dotfiles">dotfiles</a>.</p>
<p>(why <code class="language-plaintext highlighter-rouge">_</code> / underscore? … it seemed unlikely to conflict with a real directory name, is relatively easy to type, and is the thing that came to mind when I thought “base”)</p>
Goodbye Git Master Branch2021-02-06T00:00:00+00:00https://blog.jpalardy.com/posts/goodbye-git-master-branch<p>Switching off the git <code class="language-plaintext highlighter-rouge">master</code> branch was easier than I expected!</p>
<h2 id="changes-are-coming">Changes are coming…</h2>
<p>I had heard about the <a href="https://www.zdnet.com/article/github-to-replace-master-with-alternative-term-to-avoid-slavery-references/">controversy</a>
about git’s <code class="language-plaintext highlighter-rouge">master</code> branch. I didn’t have a strong emotional attachment to the name… so I decided to switch and see what happens.</p>
<p>It seems like <code class="language-plaintext highlighter-rouge">main</code> is <a href="https://github.com/github/renaming#why-main">the consensus</a> for the new branch name:</p>
<ul>
<li>it’s short</li>
<li>it’s easy on muscle memory (has the same first 2 letters)</li>
<li>it translates well to most languages</li>
</ul>
<p>Let’s switch everything to <code class="language-plaintext highlighter-rouge">main</code>.</p>
<h2 id="new-repositories-on-github">New Repositories, on GitHub</h2>
<p><a href="https://github.blog/changelog/2020-10-01-the-default-branch-for-newly-created-repositories-is-now-main/">Since October 2020</a>, GitHub creates new repositories with <code class="language-plaintext highlighter-rouge">main</code> as the default branch.</p>
<p>Done ✅</p>
<h2 id="new-repositories-locally">New Repositories, locally</h2>
<p>Since git 2.28, there’s a <a href="https://github.blog/2020-07-27-highlights-from-git-2-28/">new config setting</a> for the default branch name on
<strong>newly</strong> created repositories:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> git config <span class="nt">--global</span> init.defaultBranch main</code></pre></figure>
<p>Done ✅</p>
<h2 id="existing-repositories">Existing Repositories…</h2>
<p>It took a few tries to find a recipe that I was happy with.</p>
<p>For each repository, pick one clone:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> git branch <span class="nt">-m</span> master main
<span class="o">></span> git push <span class="nt">-u</span> origin main
<span class="o">></span> git symbolic-ref refs/remotes/origin/HEAD refs/remotes/origin/main
<span class="c"># remote is github?</span>
<span class="c"># - visit https://github.com/$username/$project/settings/branches</span>
<span class="c"># it's under "settings" >> "branches"</span>
<span class="c"># - rename default branch</span>
<span class="c"># remote is bare repo?</span>
<span class="c"># - cd to its directory</span>
<span class="o">></span> git symbolic-ref HEAD refs/heads/main
<span class="c"># back to local directory</span>
<span class="o">></span> git push origin <span class="nt">--delete</span> master</code></pre></figure>
<p>and for all your other clones:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> git branch <span class="nt">-m</span> master main
<span class="o">></span> git branch <span class="nt">-u</span> origin/main</code></pre></figure>
<p>I audited all my local repositories and had the chance to practice on over 20 projects. ✅</p>
<p>Then, I remembered to repeat on all my computers. ✅</p>
<h2 id="your-own-local-audit">Your own local audit</h2>
<p>If you’ve been accumulating clones over time, it’s easy to lose track of <em>ALL</em>
the git repositories you have hanging around. This can be a good start:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># from your $HOME</span>
<span class="o">></span> find <span class="nb">.</span> <span class="nt">-name</span> <span class="s1">'config'</span> <span class="nt">-type</span> f 2>/dev/null | <span class="nb">grep</span> <span class="s1">'\.git/config'</span></code></pre></figure>
<p>If you save the output to a file, you can bootstrap the next phase: checking “ownership”</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># output from previous command, saved to "all_repos"</span>
<span class="o">></span> <span class="nb">grep </span>github.com:jpalardy <span class="si">$(</span><span class="nb">cat </span>all_repos<span class="si">)</span></code></pre></figure>
<p>You’ll want to adjust this to your own git remote names, usernames, etc</p>
<h2 id="consequences">Consequences</h2>
<p>Everything <em>just worked</em> … which wasn’t what I was expecting. I was planning on breaking something… researching and fixing problems over time. 😅</p>
<p>Intellectually, I understood that commits matter, but branches are just
convenient names. Of course, branches can have different names locally and remotely … (although I never thought that would be a good idea)</p>
<h2 id="references">References</h2>
<p>Sounds too easy? Not sure? Here are some of the pages that helped me:</p>
<ul>
<li><a href="https://www.hanselman.com/blog/easily-rename-your-git-default-branch-from-master-to-main">Easily rename your Git default branch from master to main</a></li>
<li><a href="https://github.com/github/renaming">github/renaming</a></li>
<li><a href="https://www.git-tower.com/learn/git/faq/git-rename-master-to-main/">How to rename the “master” branch to “main” in Git</a></li>
<li><a href="https://stevenmortimer.com/5-steps-to-change-github-default-branch-from-master-to-main/">5 steps to change GitHub default branch from master to main</a></li>
</ul>
The Best Books I Read in 20202020-12-16T00:00:00+00:00https://blog.jpalardy.com/posts/best-books-i-read-2020<p>I have read many books in 2020; let’s forget most and talk about the good ones.</p>
<p>There is no particular order, but I broke down my recommendations by
categories: <a href="#technical">technical</a>, <a href="#non-fiction">non-fiction</a> and
<a href="#fiction">fiction</a>.</p>
<p>This is a yearly tradition! You can read my book reviews for
<a href="https://blog.jpalardy.com/posts/best-books-i-read-2015/">2015</a>,
<a href="https://blog.jpalardy.com/posts/best-books-i-read-2016/">2016</a>,
<a href="https://blog.jpalardy.com/posts/best-books-i-read-2017/">2017</a>,
<a href="https://blog.jpalardy.com/posts/best-books-i-read-2018/">2018</a> and
<a href="https://blog.jpalardy.com/posts/best-books-i-read-2019/">2019</a>.</p>
<hr />
<h2 id="technical">Technical</h2>
<h3 id="programming-phoenix">Programming Phoenix</h3>
<p><a href="https://www.amazon.com/dp/1680502263/"><img class="book-cover" src="/assets/best-books-2020/1680502263.jpg" alt="Programming Phoenix" /></a></p>
<p>Before reading this book, I felt that I needed to read a
<a href="https://www.amazon.com/dp/1593274351/">bunch</a>
<a href="https://www.amazon.com/dp/1680502999/">of</a>
<a href="https://www.amazon.com/dp/1617295027/">other</a>
<a href="https://www.amazon.com/dp/1680506617/">books</a>
first. 😢 I made the same mistake with Ruby and Rails…</p>
<p>At heart, I’m a web developer and that’s what I care about. Knowing <em>HOW</em> I can use Elixir
to build a webapp was so much more practical than the bottoms-up I took.</p>
<p>I only have good things to say about this book:</p>
<ul>
<li>it’s at the “right level”</li>
<li>it covers what you need, when you need it</li>
<li>it doesn’t go too deep (it doesn’t try to be an introduction <em>AND</em> a reference book)</li>
</ul>
<p>If you follow along, you’ll end up building just enough of a webapp to give you
an idea about whether Phoenix and Elixir are right for you.</p>
<p>Afterwards, I <em>would</em> recommend <a href="https://www.amazon.com/dp/1680502999/">Programming Elixir</a>, which covers the language
in more detail. It’s a great book, and I’m surprised it didn’t make it to my <a href="https://blog.jpalardy.com/posts/best-books-i-read-2018/">2018</a> recommendations… 🤔
(maybe I felt that learning Elixir in a vacuum didn’t make sense…)</p>
<ul>
<li><a href="https://www.amazon.com/dp/1680502263/">amazon</a></li>
<li><a href="https://pragprog.com/titles/phoenix14/programming-phoenix-1-4/">website</a></li>
</ul>
<h3 id="elm-in-action">Elm in Action</h3>
<p><a href="https://www.amazon.com/dp/1617294047/"><img class="book-cover" src="/assets/best-books-2020/1617294047.jpg" alt="Elm in Action" /></a></p>
<p>I had been wanting to learn Elm for a while, and I found <strong>Elm in Action</strong>
to be a gem!</p>
<p>Yes, Elm has a <a href="https://guide.elm-lang.org/">guide</a>, but I cannot say that it
was the easiest to follow or the most organized – especially for someone (me)
going in cold. To be fair, I went through the guide <em>after</em> the book and found
(in retrospect) that it made complete sense (a feeling probably shared with the
authors of the guide).</p>
<p><strong>Elm in Action</strong> is a tutorial in book format. It walks you through a project,
introducing concepts and adding functionality one chapter at a time.
I thought that the pace, order of presentation, and tone were
<em>perfect</em>: each section clarified a previous point, or answered my next
question.</p>
<p>From the preface:</p>
<blockquote>
<p>… I realized I was describing the book I wished I’d had back when I first
set out to learn functional programming. […] I wanted that book to exist so
badly, I volunteered to write it.</p>
</blockquote>
<p>Mission accomplished! I would recommend this book either for Elm enthusiasts,
but also for people wanting to get deeper into functional programming.</p>
<ul>
<li><a href="https://www.amazon.com/dp/1617294047/">amazon</a></li>
<li><a href="https://www.manning.com/books/elm-in-action">website</a></li>
</ul>
<h3 id="game-programming-patterns">Game Programming Patterns</h3>
<p><a href="https://www.amazon.com/dp/0990582906/"><img class="book-cover" src="/assets/best-books-2020/0990582906.jpg" alt="Game Programming Patterns" /></a></p>
<p><strong>Game Programming Patterns</strong> is another no-brainer: I found it
interesting and relevant – even if I’m not a game developer.
(also: I had seen it recommended so many times, it was hard to avoid…)</p>
<p><a href="https://www.amazon.com/dp/0201633612/">Design Patterns</a> is <em>old</em> and not
necessarily the most enjoyable read. And though Amazon says the book is (still) a
best seller, it’s unclear how many people have read it. It’s refreshing to see
an updated take on <em>patterns</em>, with modern examples and opinions about what
to do and what to avoid.</p>
<p>I had the opportunity to apply some of the ideas from the book to a project at work;
it made the code more flexible and simplified the architecture. It’s unclear if the
book was the <em>cause</em>, but it certainly was an inspiration 😄</p>
<p>There are also a few chapters on optimization patterns, of which “data locality” is close to my heart.
Overall, it was a fun read, and gave me ideas to (hopefully) untangle future code messes.</p>
<ul>
<li><a href="https://www.amazon.com/dp/0990582906/">amazon</a></li>
<li><a href="https://gameprogrammingpatterns.com/">website</a></li>
</ul>
<hr />
<h2 id="non-fiction">Non-Fiction</h2>
<h3 id="white-fragility">White Fragility</h3>
<p><a href="https://www.amazon.com/dp/0807047414/"><img class="book-cover" src="/assets/best-books-2020/0807047414.jpg" alt="White Fragility" /></a></p>
<p>What a year 2020 was! During the <a href="https://en.wikipedia.org/wiki/George_Floyd_protests">George Floyd protests</a>,
there were many proposed reading lists, like <a href="https://nymag.com/strategist/article/anti-racist-reading-list.html">this one</a>.
Fighting racism starts with yourself and I thought it would be good to do some
homework.</p>
<p>A few names and titles kept coming back; I ended up reading (in this order):</p>
<ul>
<li><a href="https://www.amazon.com/dp/1580056776/">So You Want to Talk About Race</a></li>
<li><a href="https://www.amazon.com/dp/0525509283/">How to Be an Antiracist</a></li>
<li><a href="https://www.amazon.com/dp/0807047414/">White Fragility</a></li>
</ul>
<p>Let me be clear about this: I enjoyed <em>and</em> recommend all 3 books … but I felt
that <strong>White Fragility</strong> probably hit the hardest.</p>
<p>I tried to gather my thoughts, but I think wikipedia page did a better job:</p>
<blockquote>
<p>DiAngelo coined the term “white fragility” in 2011 to describe any defensive
instincts or reactions that a white person experiences when questioned about
race or made to consider their own race. In White Fragility, DiAngelo views
racism in the United States as systemic and often perpetuated unconsciously
by individuals. She recommends against viewing racism as committed
intentionally by “bad people”.</p>
</blockquote>
<p>If you ever thought “… but I’m not racist!”, this is <strong>exactly</strong> the book for you.</p>
<p>If you need to <em>ease</em> into it, I would start with <a href="https://www.amazon.com/dp/1580056776/">So You Want to Talk About Race</a>.</p>
<ul>
<li><a href="https://www.amazon.com/dp/0807047414/">amazon</a> – all books also linked above ^</li>
<li><a href="https://en.wikipedia.org/wiki/White_Fragility">wikipedia</a></li>
</ul>
<h3 id="never-split-the-difference">Never Split the Difference</h3>
<p><a href="https://www.amazon.com/dp/1847941494/"><img class="book-cover" src="/assets/best-books-2020/1847941494.jpg" alt="Never Split the Difference" /></a></p>
<p>I admit, grudgingly, that <strong>Never Split the Difference</strong> had some good advice.</p>
<p>But, like many non-fiction books, it could have been a much <em>thinner</em> book. And it
could have been called “A Bunch of Stories about my Time as a Hostage Negotiator” …
but that’s not as catchy.</p>
<p>In the end, it’s a book about letting others “help you to help them”: it talks about
mirroring, and variants on “how am I supposed to do that?”. Every time that I’ve remembered
the techniques and put them to use, I was surprised at how well they worked!</p>
<p>This book is an update on classics like <strong>Getting to Yes</strong> (1981) and <strong>Start with NO</strong> (2002).
While not wrong, we’ve learned much since then, and this book contrasts different approaches.</p>
<p>Whether business or interpersonal, even putting this book into practice <em>once</em> will pay the price
of the book multiple times over.</p>
<ul>
<li><a href="https://www.amazon.com/dp/1847941494/">amazon</a></li>
</ul>
<h3 id="science-fictions">Science Fictions</h3>
<p><a href="https://www.amazon.com/dp/1250222699/"><img class="book-cover" src="/assets/best-books-2020/1250222699.jpg" alt="Science Fictions" /></a></p>
<p>I first heard about this book from the article:
<a href="http://nautil.us/blog/you-want-to-see-my-data-i-thought-we-were-friends">You Want to See My Data? I Thought We Were Friends!</a>.
The article, though <strong>unfortunately</strong> not the book, is illustrated by the <a href="https://www.smbc-comics.com/">@smbc</a> guy:</p>
<p><a href="https://www.amazon.com/dp//"><img style="width: 338px" src="/assets/best-books-2020/thought-we-were-friends.png" alt="comic sample" /></a></p>
<p>This book is critical of science (for good reasons) while remaining pro-science:
the way for science to <em>heal</em> itself is <em>through</em> science.
<a href="https://en.wikipedia.org/wiki/Metascience">Metascience</a>, “the use of
scientific methodology to study science itself”, is the backdrop of the
critique.</p>
<p>There are chapters dedicated to the problems of science:</p>
<ul>
<li>fraud</li>
<li>bias</li>
<li>negligence</li>
<li>hype</li>
</ul>
<p>It’s a mess out there, mostly because of how science is currently incentivized.
Don’t lose all hope: the final chapter is called “Fixing Science”, and it
highlights some of the solutions that are possible (though it isn’t meant to be a
comprehensive list).</p>
<ul>
<li><a href="https://www.amazon.com/dp/1250222699/">amazon</a></li>
</ul>
<h3 id="mastery">Mastery</h3>
<p><a href="https://www.amazon.com/dp/0452267560/"><img class="book-cover" src="/assets/best-books-2020/0452267560.jpg" alt="Mastery" /></a></p>
<p>Written in 1992, <strong>Mastery</strong> felt like the “original source” for many other
things that I’ve read over the years. (think 10,000 hours, etc) It’s part of a
certain philosophy about living your life.</p>
<p>Mastery is a journey:</p>
<ul>
<li>good news: you can start today!</li>
<li>bad news: it’s going to take years…</li>
</ul>
<p>The modern world promises quick-and-simple solutions to most problems … it takes
a certain perspective to accept that it’s going to take more time/work.</p>
<p>You also need to know that the journey to mastery will mostly be spent on
plateaus – long periods where progress is stagnant. Depending on your
outlook, you will give up, double-down, coast, or “transcend”. I
don’t think we can (or should) master everything; it becomes a question of
priorities. But it’s helpful to make those decisions consciously.</p>
<p>This is another could-have-been-a-shorter book, but I really enjoyed the first half.</p>
<ul>
<li><a href="https://www.amazon.com/dp/0452267560/">amazon</a></li>
</ul>
<h3 id="how-to-invent-everything">How to Invent Everything</h3>
<p><a href="https://www.amazon.com/dp/0735220158/"><img class="book-cover" src="/assets/best-books-2020/0735220158.jpg" alt="How to Invent Everything" /></a></p>
<p>Oh no: your time machine broke down, and now you’re stuck in the past?! Don’t worry, this guide
will bootstrap your civilization to a relative level of comfort…</p>
<p>This is part history book, part “how to” guide, wrapped in a thin layer of tongue-in-cheek humor.
Written by Ryan North, of <a href="https://qwantz.com/">Dinosaur Comics</a> fame, it’s a decent effort at
accomplishing “a survival guide for the stranded time traveler”.</p>
<p>We live in an ever-increasingly complex society and, proportionally, understand
less and less of it. This creates an anxiety: “if the world came apart, I
wouldn’t know how anything works…” (this is reflected in the popularity of
games like Civilization, Minecraft or Factorio … or other games with
<a href="https://en.wikipedia.org/wiki/Technology_tree">technology trees</a>).
Deep down, most people want to feel self-reliant and feel like they
can manage complexity.</p>
<p>While I sometimes found the humor tedious (especially after a few chapters), I
concede that it might make the book more palatable to a certain audience. I
learned a bunch of things and this is a book that I keep thinking about.</p>
<p><em>(it also made me look for other world-from-scratch books, some of which will probably make
an appearance next year)</em></p>
<ul>
<li><a href="https://www.amazon.com/dp/0735220158/">amazon</a></li>
</ul>
<hr />
<h2 id="fiction">Fiction</h2>
<p>I read a bunch of fiction that I enjoyed this year … but not many books that I could recommend
whole-heartedly. Even this one comes with a disclaimer.</p>
<h3 id="cyteen">Cyteen</h3>
<p><a href="https://www.amazon.com/dp/B007JQNCPI/"><img class="book-cover" src="/assets/best-books-2020/B007JQNCPI.jpg" alt="Cyteen" /></a></p>
<p>If fiction, and especially science fiction, is supposed to “take you places” or
let you (safely) experience things you normally wouldn’t, then <strong>Cyteen</strong> is worth discussing.</p>
<p>Humanity ventured into space and divided due to the long distances. One
possible solution to populate and bootstrap new worlds is cloning.
<strong>Cyteen</strong> is a book exploring cloning and its ramifications:</p>
<ul>
<li>is cloning – for a purpose – slavery?</li>
<li>does a society based on cloning require mind control?</li>
<li>what about cloning specific individuals? (and hoping to reproduce successes?)</li>
<li>what are the benefits/downsides of free will?</li>
</ul>
<p><strong>Cyteen</strong> won the 1989 <a href="https://en.wikipedia.org/wiki/Hugo_Award_for_Best_Novel">Hugo award</a> (that’s how I found it).</p>
<p>The whole book made me feel uncomfortable, claustrophobic and paranoid; living in a completely
different world and spending decades with its main characters. I found the <a href="https://en.wikipedia.org/wiki/Worldbuilding">worldbuilding</a>
impressive, and I’m going to look for more from <a href="https://en.wikipedia.org/wiki/C._J._Cherryh">the author</a>.</p>
<ul>
<li><a href="https://www.amazon.com/dp/B007JQNCPI/">amazon</a></li>
<li><a href="https://en.wikipedia.org/wiki/Cyteen">wikipedia</a></li>
</ul>
<hr />
<h2 id="closing-thoughts">Closing Thoughts</h2>
<p>If you have liked this blog post, please write your own. I want to read your book
reviews and recommendations.</p>
Which Package Name for Go Tests?2020-10-13T00:00:00+00:00https://blog.jpalardy.com/posts/which-package-name-for-go-tests<p>Imagine a simple Go project:</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">.
├── go.mod
├── main.go
└── stats
└── avg.go</code></pre></figure>
<p>No tricks:</p>
<p><a href="/assets/go-test-pkg-name/all-files.png"><img src="/assets/go-test-pkg-name/all-files.png" alt="content of all files" /></a>
<em>click to enlarge</em></p>
<p>You are about to create <code class="language-plaintext highlighter-rouge">stats/avg_test.go</code>: what is the right package name declaration for that file?</p>
<ol>
<li><code class="language-plaintext highlighter-rouge">package stats</code></li>
<li><code class="language-plaintext highlighter-rouge">package stats_test</code></li>
</ol>
<details>
<summary>[reveal the answer]</summary>
<div style="margin: 10px; padding: 10px; border: 2px dotted #bbb">
<p>Both 1 and 2 are correct ⚠️ </p>
<p><em>BUT</em> they will work differently... (keep reading)</p>
</div>
</details>
<h2 id="urgh-what">Urgh… What?</h2>
<p>Here’s what <a href="https://www.amazon.com/dp/1617291781/">Go in Action</a> says:</p>
<blockquote>
<p>The convention for naming your package is to use the name<br />
of the directory containing it. (p.40)</p>
</blockquote>
<p>That’s what most Go programmers would say. Keeping in mind:</p>
<ul>
<li>except for <code class="language-plaintext highlighter-rouge">main</code></li>
</ul>
<p>Here’s another one for you:</p>
<ul>
<li>except for tests: a <code class="language-plaintext highlighter-rouge">_test</code> suffix can be added</li>
</ul>
<p>You could be forgiven for not knowing this rule. In fact, I managed to spend
<em>years</em> with Go without knowing this rule. In denial, I looked all through Go in Action and found:</p>
<blockquote>
<p>[…] the package name also ends with _test. When the package name ends like
this, the test code can only access exported identifiers. (p.226)</p>
</blockquote>
<p>Ouch… never saw that. Or I never realized what it implied.<br />
Here’s another gem from <code class="language-plaintext highlighter-rouge">go help test</code>:</p>
<blockquote>
<p>Test files that declare a package with the suffix “_test” will be compiled
as a separate package, and then linked and run with the main test binary.</p>
</blockquote>
<h2 id="thank-you-testpackage">Thank you: testpackage</h2>
<p>I talked about <a href="https://blog.jpalardy.com/posts/up-your-go-game-with-golangci-lint/">golangci-lint</a>
before. On April 22, 2020, <a href="https://github.com/maratori/testpackage">testpackage</a> was <a href="https://github.com/golangci/golangci-lint/pull/852">added to golangci-lint</a>. If you ran golangci-lint
on this project after you added a test, you would get:</p>
<p><a href="/assets/go-test-pkg-name/use-stats_test.png"><img src="/assets/go-test-pkg-name/use-stats_test.png" alt="testpackage error" /></a></p>
<p>First reaction: my tests run and pass, what do you want from me?!</p>
<p>Second reaction: “use stats_test?” … but that would mean putting the tests in a separate directory (named <code class="language-plaintext highlighter-rouge">stats_test</code>)?! 🤔</p>
<p>As it turns out, you <em>CAN</em> use <code class="language-plaintext highlighter-rouge">package stats_test</code> while keeping the test file in the <code class="language-plaintext highlighter-rouge">stats</code> directory.</p>
<h2 id="implications">Implications</h2>
<p>Why would you want this? Why does <code class="language-plaintext highlighter-rouge">testpackage</code> nag you about this?</p>
<p>It’s a matter of blackbox versus whitebox testing.</p>
<p>Using <code class="language-plaintext highlighter-rouge">stats</code> allows you to test lowercase functions; you’re allowed to reach
for those because your tests live in the same package. In the code above, your
<code class="language-plaintext highlighter-rouge">stats</code> package can invoke <code class="language-plaintext highlighter-rouge">sum</code> even though other packages wouldn’t be allowed
to.</p>
<p>Using <code class="language-plaintext highlighter-rouge">stats_test</code> places your test code <em>outside</em> the <code class="language-plaintext highlighter-rouge">stats</code> package itself;
you’re only allowed to invoke the “public”/uppercase functions. You can test
<code class="language-plaintext highlighter-rouge">stats.Mean</code> (but not just <code class="language-plaintext highlighter-rouge">Mean</code> ⚠️) because that’s what you export and that’s what client code would be
allowed to use.</p>
<p>In both cases, you keep the <code class="language-plaintext highlighter-rouge">_test.go</code> files in the same directory as the rest
of the package. File organization -wise, nothing has changed. Philosophically
speaking, you’re taking a different stance on “the right way to test things”.</p>
<p>If you still want to test lowercase functions but survive linting, you can suffix
your test file with <code class="language-plaintext highlighter-rouge">_internal_test.go</code>…</p>
<p>You can read <a href="https://github.com/maratori/testpackage#motivation">motivation</a>
straight from the testpackage’s README. I recommend the referenced articles if
you need more convincing.</p>
<h2 id="thoughts">Thoughts</h2>
<p>Is this a good thing or not? Personally, I have mixed feelings about this.</p>
<p>I love being able to test internal/helper functions. Other programming
languages don’t necessarily have a great answer for this. Going through the
public interface often makes the tests more complicated and stateful…
(arguably, that’s what blackbox proponents would like to address)</p>
<p>I dislike that’s it’s another exception to package naming, and an obscure one
at that… 😫 Learning about these often feels like preparing for trivia night.
Placing “public package tests” in a separate directory would have accomplished
the same goal without adding an exception.</p>
<p>Also, from the outside, you cannot tell if <code class="language-plaintext highlighter-rouge">_test.go</code> files are
internal/external tests. You have to open the test files to confirm
(probably depending on a per-project testing philosophy).</p>
<p>If this was more explicit, or more frequently discussed… it would be easier
to accept as “another quirk of Go”. In the end, I haven’t decided whether I
would embrace this … or work around it:</p>
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="s">--</span>
<span class="na">linters</span><span class="pi">:</span>
<span class="na">disable</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">testpackage</span></code></pre></figure>
<p>You can <a href="https://stackoverflow.com/questions/19998250/proper-package-naming-for-testing-with-the-go-language">read more on stackoverflow</a>.</p>
Software I like: Skitch2020-10-04T00:00:00+00:00https://blog.jpalardy.com/posts/software-i-like-skitch<p><a href="https://expereb.com/software-que-me-gusta-skitch/">Spanish Translation</a>, thanks to Laura.<br />
<a href="http://lpacode.com/software-i-like-skitch/">Georgian Translation</a>, thanks to Ana.</p>
<h2 id="what-is-skitch">What is Skitch?</h2>
<p><a href="https://evernote.com/products/skitch">Skitch</a> is a screenshot-grabbing and
annotating software. Skitch is free and available for macOS. Here’s a typical
example of how I would use it:</p>
<p><img src="/assets/like-skitch/example-skitch.png" alt="skitch example" /></p>
<h2 id="why-is-skitch-awesome">Why is Skitch awesome?</h2>
<p>It is effortless and “transparent”.</p>
<blockquote>
<p>Fish don’t know they’re in water
(<a href="https://lifehacker.com/fish-dont-know-theyre-in-water-5821126">…</a>)</p>
</blockquote>
<p>That’s how I feel about Skitch… I barely ever <em>think</em> about Skitch, but I use it everyday.
It’s one of the first thing I install when I set up a new computer. I’ve been
a fan even before <a href="https://en.wikipedia.org/wiki/Evernote#Skitch">they joined Evernote</a>.</p>
<p>It is said that “a picture is worth a thousand words”. It’s especially true in a
world of text-based messages, whether email, slack, chat, twitter, or
otherwise. The idea that I would try to explain something in words when,
instead, I can:</p>
<ul>
<li>take a screenshot</li>
<li>draw arrows, boxes and lines on it</li>
<li>overlay some text</li>
</ul>
<p>is <em>almost</em> ridiculous: it’s faster for me and it’s faster for the people on the receiving end.</p>
<p>In fact, I’m annoyed when other people don’t use Skitch… If you’re
about to write me a thousand words, please consider sending me a screenshot
instead 😁</p>
<p>If I have one criticism about Skitch, it’s that it’s not overly “discoverable”. Even
if you hear the name and find the homepage, it looks like this:</p>
<p><img src="/assets/like-skitch/skitch-homepage.png" alt="skitch homepage" /></p>
<p>Is that the right page?! What does Skitch do exactly? Where is the cool demo or
tantalizing screenshots? It feels like a missed opportunity…</p>
<p>I’ve had many people ask “what software do you use to annotate your screenshots?”. It’s unfortunate
that such a cool little app isn’t better known. (I hope this post can help, in a small way.</p>
Chrome Extension: Click to Remove Element2020-08-17T00:00:00+00:00https://blog.jpalardy.com/posts/chrome-extension-click-to-remove-element<p><a href="https://expereb.com/extension-de-chrome-haga-clic-para-eliminar-el-elemento/">Spanish Translation</a>, thanks to Laura.<br />
<a href="http://lpacode.com/chrome-extension-click-to-remove-element/">Georgian Translation</a>, thanks to Ana.</p>
<p>For years, I <em>customized</em> web pages by opening up <code class="language-plaintext highlighter-rouge">Developer Tools</code>, looking
for the right DOM elements and deleting them. Classic cases include:</p>
<ul>
<li>leftover ads</li>
<li>annoying GIF images</li>
<li>modal popups</li>
<li>“Accept cookies” banners</li>
<li>helpers “share” toolbars</li>
<li>excessive headers/footers</li>
</ul>
<h2 id="click-to-remove-element">Click to Remove Element</h2>
<p>I had reached a point where I wished I could “right-click, delete”
on anything on a page. (without opening a console) I went looking and found:
<a href="https://chrome.google.com/webstore/detail/click-to-remove-element/jcgpghgjhhahcefnfpbncdmhhddedhnk">Click to Remove Element</a>.</p>
<p>It does exactly what it says it does 😃</p>
<p>You get a toolbar icon (and a keyboard shortcut: cmd-shift-x) … and if you
click on something, it gets removed. Straight from the website, the examples
are simple but compelling.</p>
<p>Remove images:</p>
<p><img src="/assets/click-to-remove-element/annoying-gif.jpg" alt="example removing an annoying gif" /></p>
<p>Remove navigation:</p>
<p><img src="/assets/click-to-remove-element/annoying-nav.jpg" alt="example removing an annoying nav" /></p>
<p>Remove ads:</p>
<p><img src="/assets/click-to-remove-element/annoying-ad.jpg" alt="example removing an annoying ad" /></p>
<p>Not shown above, but important, there’s a helper popup (browser bottom-right) that shows the HTML path:</p>
<p><img src="/assets/click-to-remove-element/helper-popup.png" alt="helper popup in action" /></p>
<p>As indicated, you can traverse to parent/children with <code class="language-plaintext highlighter-rouge">q</code>/<code class="language-plaintext highlighter-rouge">w</code>, respectively.
You can make a “rough selection” and navigate up to the “container parent”,
which is easy to tell because elements are highlighted on the page.</p>
First Look at Zsh2020-07-29T00:00:00+00:00https://blog.jpalardy.com/posts/first-look-at-zsh<p>I had heard about <a href="https://en.wikipedia.org/wiki/Z_shell">Zsh</a>
before: I thought it had many good things going for itself.
But… is it enough to switch shells?!</p>
<p>I spent years <a href="https://www.amazon.com/dp/0596009658/">reading</a> about,
<a href="https://www.tldp.org/LDP/abs/html/index.html">studying</a>, often <a href="https://blog.jpalardy.com/">writing</a> about Bash.
I didn’t feel like Bash was holding me back. 😐</p>
<p>When Apple announced (2019) that Catalina would use Zsh as the default login
shell… I thought it might be time to, at least, do some due diligence on Zsh.</p>
<h2 id="it-starts-with-a-book">It Starts with a Book</h2>
<p><a href="https://www.amazon.com/dp/1590593766/"><img class="book-cover" src="/assets/first-look-zsh/from-bash-to-z-shell.jpg" alt="book cover of From Bash to Z Shell" /></a></p>
<p>On most systems, you can type <code class="language-plaintext highlighter-rouge">zsh</code> and get started … but that doesn’t mean
you’re done. Few things will make you feel as incompetent as switching tools
unprepared: the jump into the unknown, the loss of muscle memory, all the
memorized trivia…</p>
<p>How does one get started with a new shell? Maybe reading <a href="http://zsh.sourceforge.net/Doc/Release/index.html">the manual</a>?
In my case, I had heard good things about <a href="https://www.amazon.com/dp/1590593766/">From Bash to Z Shell</a>.</p>
<p>It’s a “serious” book (472 pages!) and the beginning is basic (“what is a shell?”),
but it ends up covering both Bash and Zsh, with a bias for Zsh.</p>
<p>Most entertaining book I’ve ever read? No… but I think it did a fair job at building a solid foundation.</p>
<h2 id="oh-my-config">Oh my Config…</h2>
<p>It’s hard to talk about Zsh without talking about <a href="https://en.wikipedia.org/wiki/Z_shell#Oh_My_Zsh">Oh My Zsh</a>…</p>
<p>Personally, I decided not to go that route. I have a strong dislike for the
don’t-worry-about-it and everything-you-could-possibly-want approach to
software customization. I’ve seen it many times in the vim community, there’s a
big difference between:</p>
<ul>
<li>I’m learning vim…</li>
<li>I’m learning vim, using someone else’s config, and I installed 25 plugins…</li>
</ul>
<p>I prefer the bottom-up approach to customization, and that’s what I decided to do.<br />
(<em>maybe</em> I’ll give oh-my-zsh a try at some point, but I’m happy now)</p>
<p>Config-wise, here’s what I did:</p>
<ul>
<li>I started with nothing… a pretty barebone but “fair” experience</li>
<li>I added a few tidbits that I picked up while reading the book</li>
<li>I ported, over <em>MANY</em> days, most of the functionality I had built up over the years in my Bash <a href="https://github.com/jpalardy/dotfiles">dotfiles</a></li>
<li>over the following weeks, I kept an eye for weird (or different) behavior that I might have to tweak</li>
</ul>
<p><a href="https://github.com/jpalardy/dotfiles/blob/main/zsh/zshrc">Here</a> is the final result, if you’re curious. All in all, it amounted to a ton of Googling 😄</p>
<h2 id="my-impressions-so-far">My impressions, so far</h2>
<p>Zsh is an excellent shell, and I would gladly recommend it as a first shell to someone starting out.</p>
<p>For someone who’s already comfortable with Bash… my recommendations would be
more “mixed”. It doesn’t seem clear, especially with the time I spent reading
the book and customizing, that “it’s worth it”.</p>
<p>In fact, maybe it’s worth “skipping” Zsh and going directly to
<a href="https://fishshell.com/">fish</a>? That’s probably something I’ll try soon (and
<a href="https://blog.jpalardy.com/posts/fish-shell-is-awesome/">report back</a>).</p>
Dealing with Non-ASCII Characters2020-07-13T05:09:00+00:00https://blog.jpalardy.com/posts/dealing-with-non-ascii-characters<h2 id="problem">Problem</h2>
<p>Quick: why is this JSON not valid?</p>
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
“user”: {
“username”: “jpalardy”,
“first_name”: “Jonathan”,
“last_name”: “Palardy”
}
}
</code></pre></div></div>
<details>
<summary>[reveal the answer]</summary>
<br />
<p>
<a href="https://typographyforlawyers.com/straight-and-curly-quotes.html">Curly quotes!</a>
</p>
<p>
Trick question? Yes and no... this happened to me and it was difficult to troubleshoot <em>visually</em>.
</p>
</details>
<h2 id="a-class-of-problems">A Class of Problems…</h2>
<p>Many text formats, programming languages and other machine-parsed texts have rules about
what characters are allowed and not.</p>
<p>When in doubt, the lowest common denominator is usually <a href="https://en.wikipedia.org/wiki/ASCII">ASCII</a>:</p>
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>The decimal set:
0 nul 1 soh 2 stx 3 etx 4 eot 5 enq 6 ack 7 bel
8 bs 9 ht 10 nl 11 vt 12 np 13 cr 14 so 15 si
16 dle 17 dc1 18 dc2 19 dc3 20 dc4 21 nak 22 syn 23 etb
24 can 25 em 26 sub 27 esc 28 fs 29 gs 30 rs 31 us
32 sp 33 ! 34 " 35 # 36 $ 37 % 38 & 39 '
40 ( 41 ) 42 * 43 + 44 , 45 - 46 . 47 /
48 0 49 1 50 2 51 3 52 4 53 5 54 6 55 7
56 8 57 9 58 : 59 ; 60 < 61 = 62 > 63 ?
64 @ 65 A 66 B 67 C 68 D 69 E 70 F 71 G
72 H 73 I 74 J 75 K 76 L 77 M 78 N 79 O
80 P 81 Q 82 R 83 S 84 T 85 U 86 V 87 W
88 X 89 Y 90 Z 91 [ 92 \ 93 ] 94 ^ 95 _
96 ` 97 a 98 b 99 c 100 d 101 e 102 f 103 g
104 h 105 i 106 j 107 k 108 l 109 m 110 n 111 o
112 p 113 q 114 r 115 s 116 t 117 u 118 v 119 w
120 x 121 y 122 z 123 { 124 | 125 } 126 ~ 127 del
(courtesy of `man ascii`, a reference never too far)
</code></pre></div></div>
<p>And while “curly quotes” might seem like a made-up problem<sup id="fnref:made-up" role="doc-noteref"><a href="#fn:made-up" class="footnote" rel="footnote">1</a></sup>, there are other insidious examples:</p>
<ul>
<li><a href="https://en.wikipedia.org/wiki/Dash#En_dash">en dash</a>, <a href="https://en.wikipedia.org/wiki/Dash#Em_dash">em dash</a></li>
<li><a href="https://en.wikipedia.org/wiki/Non-breaking_space">non-breaking space</a> and <a href="https://en.wikipedia.org/wiki/Tab_key#Tab_characters">tab</a> (to a lesser extent)</li>
<li><a href="https://en.wikipedia.org/wiki/Carriage_return">carriage return</a> and <a href="https://en.wikipedia.org/wiki/Newline">newline</a></li>
<li>in general: <a href="https://en.wikipedia.org/wiki/Homoglyph">homoglyphs</a> (<a href="https://en.wikipedia.org/wiki/IDN_homograph_attack">other examples</a>)</li>
</ul>
<h2 id="solutions">Solutions</h2>
<p>There is no general solution to all the problems, only an assortment of tricks:</p>
<ul>
<li>“weird spacing” is often flagged or fixed by text editors; details will vary</li>
<li>file formats: can be fixed with <a href="https://waterlan.home.xs4all.nl/dos2unix.html">dos2unix</a> or similar</li>
<li>external linters can be your sanity check:</li>
</ul>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> jq <span class="nb">.</span> invalid.json
parse error: Invalid numeric literal at line 2, column 13
<span class="o">></span>
<span class="c"># better than nothing? 🤔</span></code></pre></figure>
<h3 id="the-non-visible-ascii-regexp-trick">The Non-Visible ASCII regexp Trick</h3>
<p>If what’s allowed is “visible ASCII”, what’s <em>not allowed</em> is “non-visible ASCII”:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[^ -~]
</code></pre></div></div>
<p>described in words: all characters not between “space” and “tilde”<br />
(I don’t remember where I picked up this trick. I would appreciate a link if you know.)</p>
<p>Why does this work? Referring back to the ASCII table from above:</p>
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code> 0 nul 1 soh 2 stx 3 etx 4 eot 5 enq 6 ack 7 bel
8 bs 9 ht 10 nl 11 vt 12 np 13 cr 14 so 15 si
16 dle 17 dc1 18 dc2 19 dc3 20 dc4 21 nak 22 syn 23 etb
24 can 25 em 26 sub 27 esc 28 fs 29 gs 30 rs 31 us
/--- start here
32 sp 33 ! 34 " 35 # 36 $ 37 % 38 & 39 '
40 ( 41 ) 42 * 43 + 44 , 45 - 46 . 47 /
48 0 49 1 50 2 51 3 52 4 53 5 54 6 55 7
56 8 57 9 58 : 59 ; 60 < 61 = 62 > 63 ?
64 @ 65 A 66 B 67 C 68 D 69 E 70 F 71 G
72 H 73 I 74 J 75 K 76 L 77 M 78 N 79 O
80 P 81 Q 82 R 83 S 84 T 85 U 86 V 87 W
88 X 89 Y 90 Z 91 [ 92 \ 93 ] 94 ^ 95 _
96 ` 97 a 98 b 99 c 100 d 101 e 102 f 103 g
104 h 105 i 106 j 107 k 108 l 109 m 110 n 111 o
112 p 113 q 114 r 115 s 116 t 117 u 118 v 119 w
120 x 121 y 122 z 123 { 124 | 125 } 126 ~ 127 del
stop here ---/
</code></pre></div></div>
<p>What is before space? various non-visible characters…<br />
What is after tilde? <code class="language-plaintext highlighter-rouge">del</code>, but also <em>ALL other Unicode characters!</em></p>
<p>Why is this useful? Many text editors can highlight based on regular expressions:</p>
<p><img src="/assets/dealing-with-non-ascii/curlies-in-vim.png" alt="curly quotes highlighted in vim" /><br />
(this is vim; use <code class="language-plaintext highlighter-rouge">:set hlsearch</code> to turn this on)</p>
<p>This trick works everywhere regular expressions work:</p>
<p><img src="/assets/dealing-with-non-ascii/curlies-in-grep.png" alt="curly quotes highlighted in grep" /></p>
<hr />
<p>Footnotes:</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:made-up" role="doc-endnote">
<p>copy-and-paste from Google Doc, Slack … and let’s compare notes 😐 <a href="#fnref:made-up" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
Better Directory Handling in Bash2020-04-10T00:00:00+00:00https://blog.jpalardy.com/posts/better-directory-handling-in-bash<p>When dealing with directories in bash, some obvious things seem like too much work…</p>
<p>Here are 2 quick tips for easier handling of directories in bash.</p>
<h2 id="unix-filename-rubout">unix-filename-rubout</h2>
<p>When dealing with long directory names, you can usually complete forward by pressing <code class="language-plaintext highlighter-rouge">TAB</code>.
But going backward? … that usually means pressing <code class="language-plaintext highlighter-rouge">BACKSPACE</code> repeatedly.</p>
<p>There is a better way: <code class="language-plaintext highlighter-rouge">unix-filename-rubout</code>. Add this to your <code class="language-plaintext highlighter-rouge">~/.inputrc</code>:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="s2">"</span><span class="se">\C</span><span class="s2">-f"</span>: unix-filename-rubout</code></pre></figure>
<p>What does it do?</p>
<p><img src="/assets/directory-handling-in-bash/front-and-back.gif" alt="moving back and forth between directories" /></p>
<p>(<em>TAB forward, ctrl-f backward</em>)</p>
<p>Every time you hit <code class="language-plaintext highlighter-rouge">ctrl-f</code>, it deletes backward until the previous <code class="language-plaintext highlighter-rouge">/</code>: useful!</p>
<p>Why <code class="language-plaintext highlighter-rouge">ctrl-f</code>?</p>
<p>By default, in bash, it’s not mapped to anything. And it’s easy to type. You
can check your bindings with <code class="language-plaintext highlighter-rouge">bind -P</code>:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">bind</span> <span class="nt">-P</span> | <span class="nb">grep </span>rubout
unix-filename-rubout is not bound to any keys
<span class="c"># rest snipped out</span></code></pre></figure>
<p>(detour: use <code class="language-plaintext highlighter-rouge">bind -P</code> to see what is bound to each <code class="language-plaintext highlighter-rouge">ctrl</code> keys)</p>
<h2 id="cd-to-filename">cd to filename</h2>
<p>What <em>usually</em> happens if you try to cd to a filename?</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">cd</span> /etc/hosts
<span class="nt">-bash</span>: <span class="nb">cd</span>: /etc/hosts: Not a directory
<span class="c"># what do you think I meant?! ... 🤬</span></code></pre></figure>
<p>(this happens often when copying-pasting path)</p>
<p>For years, I swore at bash, manually deleted the last part of the path, and
tried again. In fact, that’s what drove me to find <code class="language-plaintext highlighter-rouge">unix-filename-rubout</code> in the
first place.</p>
<p>There’s no built-in fix for this, but it’s not too complicated to wrap the <code class="language-plaintext highlighter-rouge">cd</code> in a function
and add some extra quality-of-life logic in there:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># to add to your bash dotfiles</span>
<span class="c"># wraps `cd`, other behaviors can be added</span>
<span class="nb">cd</span><span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">dest</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="p">@</span><span class="k">:-</span><span class="s2">"</span><span class="nv">$HOME</span><span class="s2">"</span><span class="k">}</span><span class="s2">"</span>
<span class="k">if</span> <span class="o">[</span> <span class="nt">-f</span> <span class="s2">"</span><span class="nv">$dest</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
</span><span class="nv">dest</span><span class="o">=</span><span class="si">$(</span><span class="nb">dirname</span> <span class="s2">"</span><span class="nv">$dest</span><span class="s2">"</span><span class="si">)</span>
<span class="k">fi
</span><span class="nb">builtin cd</span> <span class="s2">"</span><span class="nv">$dest</span><span class="s2">"</span>
<span class="k">return</span> <span class="nv">$?</span>
<span class="o">}</span></code></pre></figure>
<p>in short: if the path is a filename, use the directory containing the file instead!</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">cd</span> /etc/hosts
<span class="o">></span> <span class="nb">pwd</span>
/etc</code></pre></figure>
<h2 id="conclusion">Conclusion</h2>
<p>When using a shell, it’s often the “little things” that turn chores into seamless experiences.</p>
<p>Give those a try and let me know about your own tweaks.</p>
Git: How to Find Modified Files on a Branch2020-02-29T00:00:00+00:00https://blog.jpalardy.com/posts/git-how-to-find-modified-files-on-a-branch<p>(<em>Update 2020-03-01: there’s a better way to do this, see <a href="#addendum">below</a></em>)</p>
<h2 id="situation">Situation</h2>
<p>You’ve been working on a (Git) branch and you need to generate the list of files modified on that branch.</p>
<h2 id="why">Why?</h2>
<p>GitHub shows it: it’s useful to see in a PR. (maybe looking for surprises)</p>
<p>Maybe you need to run tests or a linter but it takes forever to run it for the whole codebase. Running
it on selected files makes it <em>a bit</em> faster. (completely hypothetical! 😄)</p>
<h2 id="how-not-to-do-it">How Not to Do It..</h2>
<p>First, I found all the commits on a branch, <em>manually</em>.</p>
<p>Then I tried to <code class="language-plaintext highlighter-rouge">git log --name-only COMMIT1 COMMIT2</code> … which was kind of close.</p>
<p>The list of files was there … now I needed to grep it out of the output. I also needed to remove duplicates (with <code class="language-plaintext highlighter-rouge">sort | uniq</code> or <a href="https://blog.jpalardy.com/posts/unsorted-uniq/">awk</a>).</p>
<p>Bottom line: long, error-prone, and messy.</p>
<h2 id="a-better-way">A Better Way</h2>
<blockquote>
<p>First, I found all the commits on a branch, <em>manually</em>.</p>
</blockquote>
<p>Let’s skip that step, I’ll come back to it in a moment.</p>
<p>I didn’t know that <code class="language-plaintext highlighter-rouge">git diff</code> also has the <code class="language-plaintext highlighter-rouge">--name-only</code> flag!</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># git diff --name-only <commit> <commit></span>
<span class="o">></span> git diff <span class="nt">--name-only</span> fc5ca53 origin/evaluate-quantile-libs
Gopkg.lock
Makefile
README.md
attack.go
internal/cmd/jsonschema/main.go
lib/metrics.go
lib/metrics_test.go
lib/target.schema.json
lib/targets.go
lib/targets_test.go
<span class="o">></span></code></pre></figure>
<p>Yes! Just the files, without duplicates.</p>
<p>What about “finding all the commits?” I found it at the <a href="https://stackoverflow.com/questions/1549146/git-find-the-most-recent-common-ancestor-of-two-branches">usual place</a>.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># git merge-base <commit> <commit></span>
<span class="o">></span> git merge-base master origin/evaluate-quantile-libs
fc5ca537bf4f01de94b0458729f455289351397e
<span class="o">></span></code></pre></figure>
<p>Let’s double-check <code class="language-plaintext highlighter-rouge">git log --graph</code>:</p>
<p><a href="/assets/branch-modified-files/merge-base.png"><img src="/assets/branch-modified-files/merge-base.png" alt="git log --graph confirms commit SHAs" /></a>
(<em>click to enlarge</em>)</p>
<p>Looks right: <code class="language-plaintext highlighter-rouge">man git-merge-base</code> says:</p>
<blockquote>
<p>Find as good common ancestors as possible for a merge</p>
</blockquote>
<h2 id="all-together">All Together</h2>
<p>With <a href="https://www.gnu.org/software/bash/manual/html_node/Command-Substitution.html">command substitution</a>, it’s possible to combine both commands on one line:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># git diff --name-only $(git merge-base <commit> <commit>) <commit></span>
<span class="o">></span> git diff <span class="nt">--name-only</span> <span class="si">$(</span>git merge-base master origin/evaluate-quantile-libs<span class="si">)</span> origin/evaluate-quantile-libs
Gopkg.lock
Makefile
README.md
attack.go
internal/cmd/jsonschema/main.go
lib/metrics.go
lib/metrics_test.go
lib/target.schema.json
lib/targets.go
lib/targets_test.go
<span class="o">></span></code></pre></figure>
<p>That’s a bit tedious, you can use <code class="language-plaintext highlighter-rouge">HEAD</code> if you’re already on the branch:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># git diff --name-only $(git merge-base <commit> <commit>) <commit></span>
<span class="o">></span> git diff <span class="nt">--name-only</span> <span class="si">$(</span>git merge-base master HEAD<span class="si">)</span> <span class="c"># <- implicit HEAD</span>
<span class="c"># omitted -- but same output!</span>
<span class="o">></span></code></pre></figure>
<p>and if <code class="language-plaintext highlighter-rouge">master</code> is the usual reference branch, you can automate it all:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># define a function:</span>
<span class="o">></span> git-mod-files<span class="o">()</span> <span class="o">{</span>
<span class="o">></span> git diff <span class="nt">--name-only</span> <span class="si">$(</span>git merge-base <span class="k">${</span><span class="nv">1</span><span class="k">:-</span><span class="nv">master</span><span class="k">}</span> HEAD<span class="si">)</span>
<span class="o">></span> <span class="o">}</span>
<span class="o">></span> git-mod-files <span class="c"># defaults to master</span>
<span class="c"># omitted -- but same output!</span>
<span class="o">></span> git-mod-files origin/inline-body <span class="c"># can specify a different branch</span>
.github/CODEOWNERS
Gopkg.lock
Makefile
README.md
attack.go
internal/cmd/jsonschema/main.go
lib/metrics.go
lib/metrics_test.go
lib/target.schema.json
lib/targets.go
lib/targets_test.go
<span class="o">></span></code></pre></figure>
<h2 id="addendum">Addendum</h2>
<p>After I wrote this, I received an email from Nathan who pointed out there’s a simpler way to do this.</p>
<p>As per <code class="language-plaintext highlighter-rouge">man git-diff</code>:</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">git diff [<options>] <commit>...<commit> [--] [<path>...]
This form is to view the changes on the branch containing and up to
the second <commit>, starting at a common ancestor of both
<commit>. "git diff A...B" is equivalent to "git diff $(git
merge-base A B) B". You can omit any one of <commit>, which has the
same effect as using HEAD instead.
Just in case you are doing something exotic, it should be noted that
all of the <commit> in the above description, except in the last two
forms that use ".." notations, can be any <tree>.
For a more complete list of ways to spell <commit>, see "SPECIFYING
REVISIONS" section in gitrevisions(7). However, "diff" is about
comparing two endpoints, not ranges, and the range notations
("<commit>..<commit>" and "<commit>...<commit>") do not mean a range as
defined in the "SPECIFYING RANGES" section in gitrevisions(7).</code></pre></figure>
<p>If your eyes glazed over, the important part is:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>"git diff A...B" is equivalent to "git diff $(git merge-base A B) B".
</code></pre></div></div>
<p>That looks suspiciously close to what I came up with 🤔<br />
At least, I was on the right track…</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># explicit: git diff --name-only master...HEAD</span>
<span class="o">></span> git diff <span class="nt">--name-only</span> master...
Gopkg.lock
Makefile
README.md
attack.go
internal/cmd/jsonschema/main.go
lib/metrics.go
lib/metrics_test.go
lib/target.schema.json
lib/targets.go
lib/targets_test.go
<span class="o">></span></code></pre></figure>
<p>Yep, looks good!</p>
Comments on the Command-Line2020-02-16T00:00:00+00:00https://blog.jpalardy.com/posts/comments-on-the-command-line<p><a href="https://guideslib.com/publications/comments-on-the-command-line/">Bulgarian Translation</a>, thanks to Zlatan.</p>
<p>Most shells support comments with the <code class="language-plaintext highlighter-rouge">#</code> symbol. At first glance, that’s
obvious because shell commands can be turned into shell scripts … and all
programming languages have comments.</p>
<p>But comments are more complicated than they seem; their <a href="https://en.wikipedia.org/wiki/Comment_(computer_programming)#Uses">uses</a>
vary:</p>
<ul>
<li>documentation</li>
<li>explanations</li>
<li>compiler annotations, sometimes</li>
<li>debugging – commenting out code</li>
<li>tagging – TODO, FIXME, etc…</li>
</ul>
<h2 id="what-makes-shell-comments-different">What makes shell comments different?</h2>
<p>Shell comments are part of the shell’s history!</p>
<p>If you type a command and add a trailing comment, it’s going to <code class="language-plaintext highlighter-rouge">history</code>:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">gzip </span>report.html <span class="c"># send to bob later</span>
<span class="o">></span> <span class="nb">history</span> | <span class="nb">tail</span>
<span class="c"># snip</span>
894 <span class="nb">gzip </span>report.html <span class="c"># send to bob later</span>
895 <span class="nb">history</span> | <span class="nb">tail</span>
<span class="o">></span></code></pre></figure>
<p>I was surprised: that’s not how I imagined the shell would behave (by default).</p>
<p>And though that’s a cute use of shell comments, it’s not necessarily something
I would recommend. I wouldn’t build a workflow around this…</p>
<h2 id="how-to-leverage-this">How to leverage this?</h2>
<p>Personally, I use this feature when:</p>
<ul>
<li>I’m in the middle of typing a command</li>
<li>something else happens, I need to pause this…</li>
<li>but I don’t want to lose my “work”</li>
</ul>
<p>It might be because I need to consult a man page, or because I need to do
something else first. In those cases, I go to the beginning of the line (<kbd>ctrl-a</kbd>)
and comment out the whole thing:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="c"># cat data.csv | awk -F, '{print $3}' | sort | uniq -c</span>
<span class="o">></span> <span class="nb">history</span> | <span class="nb">tail</span>
<span class="c"># snip</span>
896 <span class="c"># cat data.csv | awk -F, '{print $3}' | sort | uniq -c</span>
897 <span class="nb">history</span> | <span class="nb">tail</span>
<span class="o">></span></code></pre></figure>
<p>In that case, shell comments can be the <code class="language-plaintext highlighter-rouge">git stash</code> of your command-line session 😄</p>
The Best Books I Read in 20192019-12-25T00:00:00+00:00https://blog.jpalardy.com/posts/best-books-i-read-2019<p>I have read many books in 2019; let’s forget most and talk about the good ones.</p>
<p>There is no particular order, but I broke down my recommendations by
categories: <a href="#technical">technical</a>, <a href="#non-fiction">non-fiction</a> and
<a href="#fiction">fiction</a>.</p>
<p>This is a yearly tradition! You can read my book reviews for
<a href="https://blog.jpalardy.com/posts/best-books-i-read-2015/">2015</a>,
<a href="https://blog.jpalardy.com/posts/best-books-i-read-2016/">2016</a>,
<a href="https://blog.jpalardy.com/posts/best-books-i-read-2017/">2017</a> and
<a href="https://blog.jpalardy.com/posts/best-books-i-read-2018/">2018</a>.</p>
<hr />
<h2 id="technical">Technical</h2>
<h3 id="the-rust-programming-language">The Rust Programming Language</h3>
<p><a href="https://www.amazon.com/dp/1718500440/"><img class="book-cover" src="/assets/best-books-2019/1593278284.jpg" alt="The Rust Programming Language" /></a></p>
<p>Last year, I wrote about <a href="https://blog.jpalardy.com/posts/best-books-i-read-2018/#programming-rust">Programming Rust</a>
and how I felt about the language. This year, I feel the same but stronger. 😄</p>
<p>I’ve given Rust a good effort and did a major chunk of the <a href="https://adventofcode.com/2018">Advent of Code 2018</a>
in Rust. Overall: it’s not perfect but it’s definitely one of my favorite languages.</p>
<p>If I could go back, I would want to read <strong>The Rust Programming Language</strong>
first. It’s well-written, accessible, and friendly. And, good news,
the book is <a href="https://doc.rust-lang.org/book/">available online</a> for free.</p>
<p>I would still read <strong>Programming Rust</strong>, which I feel reinforced the ideas and
presented from a slightly different perspective. I also think it makes for a
better reference book.</p>
<p><strong>The Rust Programming Language</strong> is sometimes just called “the book”. In
practice, I had found the online version without ever connecting it to the
physical book – I didn’t know it was the <em>same</em> book… everything is obvious
(only) in retrospect.</p>
<ul>
<li><a href="https://www.amazon.com/dp/1718500440/">amazon</a></li>
<li><a href="https://doc.rust-lang.org/book/">website</a></li>
</ul>
<h3 id="kubernetes-in-action">Kubernetes in Action</h3>
<p><a href="https://www.amazon.com/dp/1617293725/"><img class="book-cover" src="/assets/best-books-2019/1617293725.jpg" alt="Kubernetes in Action" /></a></p>
<blockquote>
<p>Did you know that Kubernetes means<br />
“complicated” in Greek? </joke></p>
</blockquote>
<p>Seriously, I have mixed feelings about Kubernetes. Yes, it solves a bunch of
problems we had with deployment, redundancy, resource utilization, etc … but
it’s an ecosystem of technologies that is super complicated, and, seemingly,
not getting simpler – or, at least, not yet. It feels like a <a href="https://en.wikipedia.org/wiki/Faust">Faustian pact</a>,
a tradeoff of conveniences and complexities that I’m not always ready to embrace.</p>
<p>The good news is <strong>Kubernetes in Action</strong> is a life-saver. I only have good
things to say about this book. The subject matter itself is dry and it’s not
exactly bedside reading material – but, the book itself is clearly-written,
logically organized, contains most of what you need to know, and serves as a
great reference.</p>
<p>If you work with Kubernetes and need to untangle it all, this is the book for you.</p>
<ul>
<li><a href="https://www.amazon.com/dp/1617293725/">amazon</a></li>
<li><a href="https://www.manning.com/books/kubernetes-in-action">website</a></li>
</ul>
<h3 id="the-linux-programming-interface">The Linux Programming Interface</h3>
<p><a href="https://www.amazon.com/dp/1593272200/"><img class="book-cover" src="/assets/best-books-2019/1593272200.jpg" alt="The Linux Programming Interface" /></a></p>
<p>This one is a tough one….</p>
<p>First of all, it’s a monster of a book: it’s 1552 pages, weighs over 5 lbs (2.3 kg)
and could be used to kill someone.</p>
<p>As for the content, it is generous with sample C code, historical explanations,
and warnings about the (in)compatiblity mess we live in.</p>
<p>This was <em>not</em> a book that I enjoyed reading … and yet, I was happy to have
read it (especially when it was done). I found that other parts of the
programming world suddenly “made sense”, given the proper historical
context. Also, it’s sometimes easier to understand how things work
when you know what lies underneath.</p>
<p>My recommendation: use it as a reference, skip the C code until you need it,
and browse the table of contents first.</p>
<ul>
<li><a href="https://www.amazon.com/dp/1593272200/">amazon</a></li>
<li><a href="https://man7.org/tlpi/">website</a></li>
</ul>
<hr />
<h2 id="non-fiction">Non-Fiction</h2>
<h3 id="why-we-sleep">Why We Sleep</h3>
<p><a href="https://www.amazon.com/dp/1501144316/"><img class="book-cover" src="/assets/best-books-2019/1501144316.jpg" alt="Why We Sleep" /></a></p>
<p>We spend almost a ⅓ of our lives sleeping, but science still doesn’t have clear
answers about the purpose of sleep… until now?</p>
<p>There was a lot of buzz when this book came out, and the author was suddenly
everywhere talking about his book. Bill Gates also <a href="https://www.gatesnotes.com/Books/Why-We-Sleep">reviewed the book</a>
and it made his <a href="https://www.gatesnotes.com/About-Bill-Gates/Holiday-Books-2019">shortlist</a>.</p>
<p>Sleep research has been progressing slowly over time. We have solved some of
the mysteries, and have never known more than we do now. This book acts
as a summary of the research and findings.</p>
<p>I was already a believer in “a good night’s sleep”; hopefully 8 hours during
the week, and more on the weekend. This book didn’t have to do a lot of
convincing… Sleeping is good, and the lack of sleep will ruin your health.</p>
<p>But, I felt that the book was overreaching by the end: “more sleep” had
become the author’s one-solution to all problems. (Bill Gates echoes the feeling)</p>
<ul>
<li><a href="https://www.amazon.com/dp/1501144316/">amazon</a></li>
<li><a href="https://en.wikipedia.org/wiki/Why_We_Sleep">wikipedia</a></li>
</ul>
<h3 id="because-internet">Because Internet</h3>
<p><a href="https://www.amazon.com/dp/0735210934/"><img class="book-cover" src="/assets/best-books-2019/0735210934.jpg" alt="Because Internet" /></a></p>
<p>Languages evolve and the Internet seems to accelerate the process. It also
allows linguists to track trends over time and space in a way that wasn’t
possible before.</p>
<p>For example, the book goes over the (relatively recent) adoption of “hello” as
a greeting. It became popular because it was used to answer the phone.
Previously, people used “hi” or “good morning” as a greeting, while “hello” was used to
attract someone’s attention.</p>
<p>This book is filled with a bunch of those stories like this one, and other historical
perspective on slang, jargon, punctuation, capitalization, emojis, memes, and
so much more.</p>
<p><a href="https://en.wikipedia.org/wiki/The_medium_is_the_message">The medium is the message</a>, and that’s
also a focus of this book: the technologies you use and when you adopted them
will influence <em>how</em> you communicate. It’s interesting stuff especially if
you’ve lived through most of it.</p>
<ul>
<li><a href="https://www.amazon.com/dp/0735210934/">amazon</a></li>
<li><a href="https://gretchenmcculloch.com/book/">website</a></li>
</ul>
<h3 id="envisioning-information">Envisioning Information</h3>
<p><a href="https://www.amazon.com/dp/1930824149/"><img class="book-cover" src="/assets/best-books-2019/1930824149.jpg" alt="Envisioning Information" /></a></p>
<p>The author, <a href="https://en.wikipedia.org/wiki/Edward_Tufte">Edward Tufte</a>,
published a series of beautiful books about information design. This one just
happens to be the first one I picked up.</p>
<p>The whole collection:</p>
<ul>
<li><a href="https://www.amazon.com/dp/1930824130/">The Visual Display of Information</a></li>
<li><a href="https://www.amazon.com/dp/1930824157/">Visual Explanations</a></li>
<li><a href="https://www.amazon.com/dp/1930824149/">Envisioning Information</a></li>
<li><a href="https://www.amazon.com/dp/1930824165/">Beautiful Evidence</a></li>
</ul>
<p>These books are seminal works on information design; the books that other books put
in their bibliographies. It’s easy to read Tufte’s books and claim “it’s
obvious” while forgetting that you’re reading <em>the</em> original source!</p>
<p>Beyond the historical importance, it’s a guided tour through good and bad data
visualizations. What’s the best way to display data? What are the alternatives?
What’s the importance of color, contrast, comparison? How many dimensions can
fit in a 2D visualization?</p>
<p>Beautiful examples from many sources (geographical and historical) are given,
sometimes with explanations, sometimes for you to contemplate.</p>
<ul>
<li><a href="https://www.amazon.com/dp/1930824149/">amazon</a></li>
<li><a href="https://en.wikipedia.org/wiki/Edward_Tufte">wikipedia</a></li>
<li><a href="https://www.edwardtufte.com/tufte/books_ei">website</a></li>
</ul>
<h3 id="the-nature-of-technology">The Nature of Technology</h3>
<p><a href="https://www.amazon.com/dp/1416544062/"><img class="book-cover" src="/assets/best-books-2019/1416544062.jpg" alt="The Nature of Technology" /></a></p>
<p>What is technology?</p>
<p>Is it obvious? Or do you need a moment to pin it down?</p>
<p><strong>The Nature of Technology</strong> is an “academic” book that seemed focused on “the details”,
but that also explained things I hadn’t fully understood before.</p>
<p>The main ideas are summarized <a href="https://www.the-vital-edge.com/the-nature-of-technology/">here</a>. Here’s one example:</p>
<ul>
<li>technologies are usually built from other (sub-)technologies, recursively</li>
<li>“base” technologies are based on exploiting physical phenomena</li>
</ul>
<p>🤔 yeah… I guess.</p>
<p>More importantly (for me): new “superior” technologies often take decades to be
adopted because of the “ecosystem” of technologies necessary to exploit them to
their full extent.</p>
<p>For example: a new programming language might have many
advantages … but until tooling catches up, useful libraries are written, and
enough people know how to use it effectively, the “advantages” might be
balanced by more pragmatic considerations. Eventually, the adoption of that new
language is “obvious” (especially for new projects), but it might take a long
time to get there. Old technologies take a long time to die.</p>
<ul>
<li><a href="https://www.amazon.com/dp/1416544062/">amazon</a></li>
<li><a href="http://tuvalu.santafe.edu/~wbarthur/thenatureoftechnology.htm">website</a></li>
</ul>
<hr />
<h2 id="fiction">Fiction</h2>
<h3 id="commune">Commune</h3>
<p><em>Series: 4 books, complete.</em></p>
<p><a href="https://www.amazon.com/dp/B0771Q2MMN/"><img class="book-cover" src="/assets/best-books-2019/B0771Q2MMN.jpg" alt="Commune" /></a></p>
<p>I think the most interesting aspect of zombie books is how society
disintegrates, almost overnight. What happens afterwards often focuses on
the group dynamics and the drama (*cough* the walking dead *cough*).</p>
<p>There are no zombies in <strong>Commune</strong>, but it definitely falls into that category
of stories. I’ve explained it as a zombie-story-without-zombies; in this case,
it’s a solar storm, followed by a plague.</p>
<p>As usual, everything goes to shit. People divide “the real assholes”,
“psychopaths on the loose” and “good people who must do bad things”… It’s a
bit predictable but it’s an enjoyable and heartbreaking ride.</p>
<p>What made <strong>Commune</strong> special was:</p>
<ul>
<li>narration by <a href="https://en.wikipedia.org/wiki/R._C._Bray">R.C. Bray</a></li>
<li>good character development: the author is going to make you care before making you cry</li>
<li>Gibbs, the foul-mouthed no-nonsense Marine!</li>
</ul>
<p>It’s not high art, but it’s a fun read.</p>
<ul>
<li><a href="https://www.amazon.com/dp/B0771Q2MMN/">amazon</a></li>
<li><a href="https://joshuagayouauthor.com/">website</a></li>
</ul>
<h3 id="ancillary-justice">Ancillary Justice</h3>
<p><em>Series: 3 books, complete.</em></p>
<p><a href="https://www.amazon.com/dp/031624662X/"><img class="book-cover" src="/assets/best-books-2019/031624662X.jpg" alt="Ancillary Justice" /></a></p>
<p>This book had been on my shortlist for a while; it won the Hugo, Nebula, Locus
Awards and a bunch more. The Audible reviews, however, were very critical of the
narration… I did what I wouldn’t normally do: I read it in paper format.</p>
<p>I can understand the narration challenges: the book is narrated from the perspective on an
<a href="https://en.wikipedia.org/wiki/Artificial_intelligence#In_fiction">AI</a> who
inhabits multiple bodies at the same time – making singular and plural
subjective. On top of that, the AI is gender-blind, so the default pronoun for
all characters is “she”. It’s only through indirect clues, or other people
talking, that gender is revealed.</p>
<p>No other book has made me want to wear gloves and drink tea as much as this
one 😄 Although <a href="https://www.amazon.com/dp/B01GIOKBWI/">Ninefox Gambit</a> came
pretty close.</p>
<p>The AI perspective beyond the easy “AI wants to be human” or “AI wants to kill
us all” was refreshing. The “alien” perspective of being a spaceship <em>and</em>
having multiple bodies simultaneously was also interesting. There are a lot of
good ideas in there.</p>
<p>I felt the ending was both disappointing and satisfying at the same time. It
was … an appropriate ending.</p>
<ul>
<li><a href="https://www.amazon.com/dp/031624662X/">amazon</a></li>
<li><a href="https://en.wikipedia.org/wiki/Ancillary_Justice">wikipedia</a></li>
<li><a href="https://annleckie.com/novel/ancillary-justice/">website</a></li>
</ul>
<h3 id="house-of-suns">House of Suns</h3>
<p><a href="https://www.amazon.com/dp/B002MVI14M/"><img class="book-cover" src="/assets/best-books-2019/B002MVI14M.jpg" alt="House of Suns" /></a></p>
<p>I’ve read most of Alastair Reynolds’s books… and picked this one up on a
whim. I prefer longer series but I had run out of material…</p>
<p>(Sidenote: I watched <a href="https://en.wikipedia.org/wiki/Love,_Death_%26_Robots">Love, Death & Robots</a> and was impressed
by <a href="https://en.wikipedia.org/wiki/Zima_Blue_and_Other_Stories">Zima Blue</a>, which I later discovered had been written by Alastair Reynolds.)</p>
<p>I read science fiction to be exposed to “things I had never thought about” …
I try to avoid science fiction as “drama in space”. This book did not disappoint. A sample:</p>
<ul>
<li>what does it mean to live for thousands of years? how do you cope?</li>
<li>what is memory?</li>
<li>what is gender? – also covered in <a href="#ancillary-justice">Ancillary Justice</a></li>
<li>the speed of light and relativity makes everything weird…</li>
</ul>
<p>And when I thought “oh, that’s what this story is about…”, I found that I wasn’t thinking big enough.</p>
<p>However, a warning: Reynolds doesn’t “explain”. His books expect you
to stick around long enough for the details to fall into place. That can be a
jarring experience, like being thrown in the middle of a conversation… I can
vouch that everything <em>eventually</em> makes sense.</p>
<ul>
<li><a href="https://www.amazon.com/dp/B002MVI14M/">amazon</a></li>
<li><a href="https://en.wikipedia.org/wiki/House_of_Suns">wikipedia</a></li>
<li><a href="http://www.alastairreynolds.com/release/house-of-suns-2008/">website</a></li>
</ul>
<h3 id="great-north-road">Great North Road</h3>
<p><a href="https://www.amazon.com/dp/B00AVAQVAW/"><img class="book-cover" src="/assets/best-books-2019/B00AVAQVAW.jpg" alt="Great North Road" /></a></p>
<p>Another book from my “favorite author pile”. <strong>Great North Road</strong> is a
standalone book: it’s a murder investigation, a monster story, a cover-up, and a
space opera.</p>
<p>Imagine a group of corporations ran by clones of the same man. Imagine
investigating the murder of an unidentified clone … across multiple planets.
This one will keep you guessing…</p>
<p>The author, <a href="https://en.wikipedia.org/wiki/Peter_F._Hamilton">Peter F. Hamilton</a>,
is known for his universe-building series and multiple intertwining storylines.
This book is a delicious appetizer hinting at what his other books offer. If
you like this one, there’s a lot more where that came from.</p>
<ul>
<li><a href="https://www.amazon.com/dp/B00AVAQVAW/">amazon</a></li>
<li><a href="https://en.wikipedia.org/wiki/Great_North_Road_(novel)">wikipedia</a></li>
<li><a href="https://www.penguinrandomhouse.ca/books/209161/great-north-road-by-peter-f-hamilton/9780345526670">website</a></li>
</ul>
<hr />
<h2 id="closing-thoughts">Closing Thoughts</h2>
<p>If you have liked this blog post, please write your own. I want to read your book
reviews and recommendations.</p>
What Exec Does in Shell Scripts2019-11-22T00:00:00+00:00https://blog.jpalardy.com/posts/what-exec-does-in-shell-scripts<p>If you think the <a href="https://man7.org/linux/man-pages/man3/exec.3.html">answer</a> is:</p>
<blockquote>
<p>The exec() family of functions replaces the current process image with a new
process image.</p>
</blockquote>
<p>you are only partially right… That’s only true <a href="https://www.computerhope.com/unix/bash/exec.htm#examples">if you don’t redirect</a>:</p>
<blockquote>
<p>Redirections are a special case, and exec does not destroy the current shell
process, but bash will no longer print output to the screen, writing it to
the file instead.</p>
</blockquote>
<p>or from <a href="https://linux.die.net/man/1/bash">man bash</a> itself (scroll down):</p>
<blockquote>
<p>If command is not specified, any redirections take effect in the current shell […]</p>
</blockquote>
<h2 id="seeing-is-believing">Seeing is believing</h2>
<p>Put the following in a file called <code class="language-plaintext highlighter-rouge">test.bash</code></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">echo</span> <span class="s2">"before: </span><span class="nv">$$</span><span class="s2">"</span>
<span class="nb">exec</span> <span class="o">></span> stdout.log
<span class="nb">echo</span> <span class="s2">"after: </span><span class="nv">$$</span><span class="s2">"</span></code></pre></figure>
<p>and run it:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> bash test.bash
before: 78511
<span class="o">></span> <span class="nb">ls
</span>stdout.log test.bash
<span class="o">></span> <span class="nb">cat </span>stdout.log
after: 78511</code></pre></figure>
<p>Comments:</p>
<ul>
<li>the “after” echo ran (after the exec!)</li>
<li>echo was redirected to <code class="language-plaintext highlighter-rouge">stdout.log</code></li>
<li>it shows the same process ID (the <code class="language-plaintext highlighter-rouge">$$</code> variable)</li>
</ul>
<p>There are <a href="https://www.computerhope.com/unix/bash/exec.htm#examples">more examples</a> of what’s possible:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># Open myinfile.txt for reading ("<") on file descriptor 3.</span>
<span class="nb">exec </span>3< myinfile.txt
<span class="c"># "-u 3" tells read to get its data from file descriptor 3</span>
<span class="nb">read</span> <span class="nt">-u</span> 3 mydata
<span class="c"># Close ("&-") the open read descriptor ("<") number 3</span>
<span class="nb">exec </span>3<&-</code></pre></figure>
<h2 id="what-about-the-usual-exec-behavior">What about the usual exec behavior?</h2>
<p>That’s still in effect:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">ls
</span>child.bash parent.bash
<span class="o">></span> <span class="nb">cat </span>parent.bash
<span class="nb">echo</span> <span class="s2">"from parent, before: </span><span class="nv">$$</span><span class="s2">"</span>
<span class="nb">exec </span>bash child.bash
<span class="nb">echo</span> <span class="s2">"from parent, after: </span><span class="nv">$$</span><span class="s2">"</span>
<span class="o">></span> <span class="nb">cat </span>child.bash
<span class="nb">echo</span> <span class="s2">"from child: </span><span class="nv">$$</span><span class="s2">"</span>
<span class="o">></span> bash parent.bash
from parent, before: 78601
from child: 78601</code></pre></figure>
<p>Comments:</p>
<ul>
<li>parent.bash ran until the exec, then it was replaced by child.bash</li>
<li>the process ID is the same – the content of the parent process was replaced by its child process</li>
</ul>
Up your Go Game with golangci-lint2019-11-05T00:00:00+00:00https://blog.jpalardy.com/posts/up-your-go-game-with-golangci-lint<h2 id="another-linter">Another linter?</h2>
<p>If you’re working on Go projects, running
<a href="https://github.com/golangci/golangci-lint">golangci-lint</a> is a no brainer. It
has helped me catch real problems on many different projects.</p>
<p>There are a lot of Go linters out there:</p>
<p><img src="/assets/golangci-lint/other-linters.png" alt="Go linters supported by ALE" /></p>
<p>(<em><a href="https://github.com/dense-analysis/ale/blob/master/supported-tools.md">source: ALE supported tools</a></em>)</p>
<p>but, in the tradition of
<a href="https://github.com/alecthomas/gometalinter">gometalinter</a>, golangci-lint runs
a bunch of other linters for you. Golangci-lint is convenience itself, you can skip:</p>
<ul>
<li>learning about</li>
<li>reviewing</li>
<li>installing</li>
<li>configuring</li>
<li>running</li>
</ul>
<p>each linter separately.</p>
<h2 id="how-to-install-it">How to install it?</h2>
<p>Installing golangci-lint can be as easy as:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> brew <span class="nb">install </span>golangci/tap/golangci-lint</code></pre></figure>
<p>(<em>see <a href="https://github.com/golangci/golangci-lint#install">here</a> for alternative installations</em>)</p>
<h2 id="how-to-run-it">How to run it?</h2>
<p>Go to the base directory of your project and run:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> golangci-lint run</code></pre></figure>
<p>and marvel at everything you’re doing wrong 😃</p>
<p>I personally like to take it to the next level and run:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> golangci-lint run <span class="nt">--enable-all</span></code></pre></figure>
<p>which runs <em>ALL</em> the linters. Which linters are run by default?</p>
<p><img src="/assets/golangci-lint/default-linters.png" alt="Golangci-lint default linters" /></p>
<p>(<em>also <a href="https://github.com/golangci/golangci-lint#enabled-by-default-linters">here</a></em>)</p>
<h2 id="it-hurts">It hurts!</h2>
<p><img class="img-right" style="width: 275px" src="/assets/golangci-lint/chinese-proverb.png" alt="The best time to plant a tree..." /></p>
<p>When is the best time to run a linter?</p>
<blockquote>
<p>back when you started your project…</p>
</blockquote>
<p>The second best time is now.</p>
<p>Douglas Crockford <a href="https://www.jslint.com/help.html#warning">said it best</a>:</p>
<blockquote>
<p>JSLint will hurt your feelings.</p>
</blockquote>
<p>which can be generalized to most linters. But, over time, using a linter makes
you internalize the rules. As you learn the rules, you learn to avoid breaking
them. Eventually, your project is clean and there’s nothing to feel bad about.</p>
<p>However, in the meantime…</p>
<h2 id="taking-it-one-step-at-a-time">Taking it one step at a time</h2>
<p>If you’re getting too many warnings, there are ways to cope. You can put a
config file (called <code class="language-plaintext highlighter-rouge">.golangci.yml</code>) at the base of your project. Here’s mine,
for example:</p>
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="nn">---</span>
<span class="na">run</span><span class="pi">:</span>
<span class="na">skip-dirs</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">generated</span>
<span class="na">linters</span><span class="pi">:</span>
<span class="na">disable</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">wsl</span></code></pre></figure>
<p>(<em><a href="https://github.com/golangci/golangci-lint#config-file">all config options</a></em>)</p>
<p>Why?</p>
<ul>
<li>skip the <code class="language-plaintext highlighter-rouge">generated</code> directory: the code generator I use doesn’t comply 😢</li>
<li><code class="language-plaintext highlighter-rouge">wsl</code> is a newer addition and I haven’t had time to review its complaints (I’m only human)</li>
</ul>
<p>That highlights a good-news/bad-news aspect of linters: they are moving
targets. When you update them, you get newer and better rules, but new “problems” might
be flagged in your previously clean project.</p>
<p>Use common sense to decide if/when you want to update your linter, and whether
the new rules make sense for your project (or go too far, for your liking, into crazy).
Linters are meant to cheaply catch problems, not just make you feel bad.</p>
QR Codes on the Command-Line2019-09-20T00:00:00+00:00https://blog.jpalardy.com/posts/qr-codes-on-the-command-line<h2 id="why-qr-codes">Why QR codes?</h2>
<p><img class="book-cover" src="/assets/qr-codes/hello-world.png" alt="hello world" /></p>
<p><a href="https://en.wikipedia.org/wiki/QR_code">QR codes</a> aren’t magical, they boil down to:</p>
<ul>
<li>2d encoding of information<br />
<em>(<a href="https://en.wikipedia.org/wiki/QR_code#Storage">1000s of characters</a> depending on encoding and error correction)</em></li>
<li>machine readability<br />
<em>(especially with the supercomputing cameras in our pockets)</em></li>
</ul>
<p>but, for my own personal uses, it’s just one of the fastest way to securely
transfer small amount of data, (<em>e.g. URLs or passwords</em>), from my computer to my phone.</p>
<p>To use QR codes, you don’t even need networking! It’s too bad that <a href="https://en.wikipedia.org/wiki/QR_code#WiFi_network_login">wifi network login</a>
was never widely adopted, it’s one of the most useful use cases for QR out there.</p>
<blockquote>
<p>As of January 2018, iPhones have this feature (wifi network login) built into the camera app under iOS 11.x […]</p>
</blockquote>
<p>Whoah… that wasn’t true when I started playing with this!<br />
I just checked and it works 🤯</p>
<h2 id="on-the-command-line">On the command-line</h2>
<p>Get the <a href="https://fukuchi.org/works/qrencode/index.html.en">qrencode</a> package:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> brew <span class="nb">install </span>qrencode <span class="c"># or use another package manager</span></code></pre></figure>
<p>To generate a QR code, dumping it to <code class="language-plaintext highlighter-rouge">stdout</code> as <a href="https://en.wikipedia.org/wiki/ASCII_art#Unicode">ASCII art</a>:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> qrencode <span class="s1">'hello world'</span> <span class="nt">-t</span> ANSIUTF8 <span class="nt">-o</span> -</code></pre></figure>
<p><img src="/assets/qr-codes/qrencode-example.png" alt="qrencode example" /></p>
<p>but that’s a lot to type; put it in your dotfiles as a function:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> qr<span class="o">()</span> <span class="o">{</span> qrencode <span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span> <span class="nt">-t</span> ANSIUTF8 <span class="nt">-o</span> -<span class="p">;</span> <span class="o">}</span>
<span class="c"># call as: qr 'hello world'</span></code></pre></figure>
<p>There’s a bunch of output formats: PNG, PNG32, EPS, SVG, XPM, ANSI, ANSI256,
ASCII, ASCIIi, UTF8, ANSIUTF8. Feel free to experiment 😃</p>
<h2 id="on-the-phone">On the phone</h2>
<p>I’m on iOS and I’ve been using <a href="https://apps.apple.com/us/app/barcode/id522354642">Barcode</a>. It gives you a contextual menu when it finds a QR code:</p>
<ul>
<li>text: copy to clipboard, share</li>
<li>URL: copy to clipboard, share, open in safari/chrome</li>
</ul>
<p>I hadn’t realized that QR codes can be read directly in the iOS camera app…
thank you Apple! It’s a bit finicky (try it, you’ll see what I mean) but it works.</p>
One-liners to Remove Empty Lines from Text Files2019-07-25T00:00:00+00:00https://blog.jpalardy.com/posts/one-liners-to-remove-empty-lines-from-text-files<p>There are multiple scenarios where removing empty lines might be necessary:</p>
<ul>
<li>reformatting source code</li>
<li>cleaning up some data</li>
<li>simplifying command-line output</li>
<li>…</li>
</ul>
<p>but what’s the easiest way of automating this?</p>
<h2 id="aside-what-is-an-empty-line">Aside: What is an empty line?</h2>
<p>For the sake of this post, I’ll make a distinction between “empty” and
“blank” lines:</p>
<ul>
<li>empty line: contains no characters at all</li>
<li>blank line: might contain whitespace (spaces, tabs, nbsp, <a href="https://en.wikipedia.org/wiki/Whitespace_character">etc…</a>)</li>
</ul>
<p>Blank lines <em>usually</em> don’t have a different meaning than empty lines.
Converting blank lines into empty lines makes subsequent handling simpler; it
normalizes your file and transforms 2 cases into just one.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># replace lines with only whitespace with "nothing"</span>
<span class="o">></span> <span class="nb">sed</span> <span class="nt">-E</span> <span class="nt">-e</span> <span class="s1">'s/^\s+$//'</span> FILENAME</code></pre></figure>
<p>In turn, a blank line is just a special case of a line with trailing
whitespace. Trailing whitespace can be fixed:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># replace end-of-line whitespace with "nothing"</span>
<span class="o">></span> <span class="nb">sed</span> <span class="nt">-E</span> <span class="nt">-e</span> <span class="s1">'s/\s+$//'</span> FILENAME</code></pre></figure>
<p>(Use the <code class="language-plaintext highlighter-rouge">sed</code> flag <code class="language-plaintext highlighter-rouge">-i</code>/<code class="language-plaintext highlighter-rouge">--in-place</code> as needed, but responsibly)</p>
<h2 id="how-to-remove-all-empty-lines">How to remove all empty lines?</h2>
<p>The easiest thing to type is <code class="language-plaintext highlighter-rouge">grep .</code>:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">cat </span>sample.txt
3
4
5
6
7
8
9
10
11
12
13
14
<span class="o">></span> <span class="nb">cat </span>sample.txt | <span class="nb">grep</span> <span class="nb">.</span>
3
4
5
6
7
8
9
10
11
12
13
14</code></pre></figure>
<p>The dot (<code class="language-plaintext highlighter-rouge">.</code>) matches <em>any</em> character … but empty lines, by definition, do not contain
anything.</p>
<p>Another solution, for <a href="/posts/why-learn-awk/">awk enthusiasts</a>:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">cat </span>sample.txt | <span class="nb">awk </span>NF
3
4
5
6
7
8
9
10
11
12
13
14</code></pre></figure>
<p>which has the added advantage of dealing uniformly with both empty and blank
lines. <code class="language-plaintext highlighter-rouge">NF</code>, or “number of fields”, is non-zero (i.e. true), for lines containing non-whitespace characters.</p>
<h2 id="how-to-collapse-multiple-empty-lines">How to collapse multiple empty lines?</h2>
<p>Empty lines have their uses: they separate paragraphs. But is there a difference
between one empty line and multiple empty lines? Probably not … in most cases, multiple empty lines are redundant.</p>
<p>Although I could think of multiple ways to automate this (most of them involving AWK), I didn’t have a quick one-liner until recently:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">cat </span>sample.txt | <span class="nb">uniq
</span>3
4
5
6
7
8
9
10
11
12
13
14</code></pre></figure>
<p>It dawned on me that, in most text files, two identical lines following each other would not occur (or would be a mistake). But repeated empty lines fit that definition.</p>
<p>Is it cheating? Does it work as a side-effect? Maybe 😃</p>
<p>I haven’t had a case <em>yet</em> where it removed duplicate non-empty lines that I needed. (and if it did, I would be a short git command away from undoing it)</p>
<p>Did I miss anything? Is there a better-and-easier way to handle this?</p>
Go Slice Gotcha2019-06-11T00:00:00+00:00https://blog.jpalardy.com/posts/go-slice-gotcha<p>Go is, <em>mostly</em>, a straightforward and unsurprising language.</p>
<p>One thing that caught me recently was the behavior of slices.</p>
<h2 id="the-setup">The setup</h2>
<figure class="highlight"><pre><code class="language-go" data-lang="go"><span class="n">nums</span> <span class="o">:=</span> <span class="p">[]</span><span class="kt">int</span><span class="p">{</span><span class="m">3</span><span class="p">,</span> <span class="m">4</span><span class="p">,</span> <span class="m">5</span><span class="p">,</span> <span class="m">6</span><span class="p">,</span> <span class="m">7</span><span class="p">,</span> <span class="m">8</span><span class="p">,</span> <span class="m">9</span><span class="p">,</span> <span class="m">10</span><span class="p">,</span> <span class="m">11</span><span class="p">,</span> <span class="m">12</span><span class="p">,</span> <span class="m">13</span><span class="p">,</span> <span class="m">14</span><span class="p">}</span>
<span class="n">chunk</span> <span class="o">:=</span> <span class="n">nums</span><span class="p">[</span><span class="m">3</span><span class="o">:</span><span class="m">6</span><span class="p">]</span>
<span class="n">fmt</span><span class="o">.</span><span class="n">Println</span><span class="p">(</span><span class="s">"chunk:"</span><span class="p">,</span> <span class="n">chunk</span><span class="p">)</span> <span class="c">// chunk: [6 7 8]</span></code></pre></figure>
<p>A slice and a subslice; no surprises yet.</p>
<figure class="highlight"><pre><code class="language-go" data-lang="go"><span class="n">chunk</span> <span class="o">=</span> <span class="nb">append</span><span class="p">(</span><span class="n">chunk</span><span class="p">,</span> <span class="m">999</span><span class="p">)</span></code></pre></figure>
<p>What are the consequences of this <code class="language-plaintext highlighter-rouge">append</code>? <strong>Spoilers ahead!</strong></p>
<h2 id="oh-no">Oh no…</h2>
<figure class="highlight"><pre><code class="language-go" data-lang="go"><span class="n">fmt</span><span class="o">.</span><span class="n">Println</span><span class="p">(</span><span class="s">"nums: "</span><span class="p">,</span> <span class="n">nums</span><span class="p">)</span> <span class="c">// nums: [3 4 5 6 7 8 999 10 11 12 13 14]</span>
<span class="n">fmt</span><span class="o">.</span><span class="n">Println</span><span class="p">(</span><span class="s">"chunk:"</span><span class="p">,</span> <span class="n">chunk</span><span class="p">)</span> <span class="c">// chunk: [6 7 8 999]</span></code></pre></figure>
<p><code class="language-plaintext highlighter-rouge">chunk</code> contains what you would expect.</p>
<p>What about that <code class="language-plaintext highlighter-rouge">999</code> after 8 in the original slice?!</p>
<h2 id="slice-capacity">Slice Capacity</h2>
<p>The answer is not the <code class="language-plaintext highlighter-rouge">len</code>, check the <code class="language-plaintext highlighter-rouge">cap</code></p>
<figure class="highlight"><pre><code class="language-go" data-lang="go"><span class="nb">len</span><span class="p">(</span><span class="n">nums</span><span class="p">[</span><span class="m">3</span><span class="o">:</span><span class="m">6</span><span class="p">])</span> <span class="c">// 3</span>
<span class="nb">cap</span><span class="p">(</span><span class="n">nums</span><span class="p">[</span><span class="m">3</span><span class="o">:</span><span class="m">6</span><span class="p">])</span> <span class="c">// 9</span></code></pre></figure>
<p>Why 9? Because the extra capacity of the original slice was carried over when
slicing. There were 9 elements from index 3 to the end of the slice:</p>
<figure class="highlight"><pre><code class="language-go" data-lang="go"><span class="n">fmt</span><span class="o">.</span><span class="n">Println</span><span class="p">(</span><span class="n">nums</span><span class="p">[</span><span class="m">3</span><span class="o">:</span><span class="p">])</span> <span class="c">// [6 7 8 9 10 11 12 13 14]</span>
<span class="nb">len</span><span class="p">(</span><span class="n">nums</span><span class="p">[</span><span class="m">3</span><span class="o">:</span><span class="p">])</span> <span class="c">// 9</span></code></pre></figure>
<h2 id="the-solution">The Solution</h2>
<p>Go has a syntax to limit the <code class="language-plaintext highlighter-rouge">cap</code> when subslicing:</p>
<figure class="highlight"><pre><code class="language-go" data-lang="go"><span class="nb">len</span><span class="p">(</span><span class="n">nums</span><span class="p">[</span><span class="m">3</span><span class="o">:</span><span class="m">6</span><span class="o">:</span><span class="m">6</span><span class="p">])</span> <span class="c">// 3</span>
<span class="nb">cap</span><span class="p">(</span><span class="n">nums</span><span class="p">[</span><span class="m">3</span><span class="o">:</span><span class="m">6</span><span class="o">:</span><span class="m">6</span><span class="p">])</span> <span class="c">// 3</span></code></pre></figure>
<p>It’s worth reading <a href="https://golang.org/ref/spec#Slice_expressions">Slice expressions</a>; especially the “Full slice expressions” subsection.</p>
<blockquote>
<p>a[low : high : max]</p>
</blockquote>
<p>I found that <a href="https://www.amazon.com/Go-Action-William-Kennedy/dp/1617291781/">Go in Action</a> (section 4.2.3) had a great explanation
on this:</p>
<blockquote>
<p>The purpose [of the third index] is not to increase capacity, but to restrict the capacity.</p>
</blockquote>
<p>You’re still sharing the underlying array, within the valid indices. If you try to append, it’s a <a href="https://en.wikipedia.org/wiki/Copy-on-write">copy on write</a>.</p>
<h2 id="conclusion">Conclusion</h2>
<p>I’m just <em>surprised</em> this syntax doesn’t work the other way around:</p>
<figure class="highlight"><pre><code class="language-go" data-lang="go"><span class="o">-</span> <span class="n">nums</span><span class="p">[</span><span class="m">3</span><span class="o">:</span><span class="m">6</span><span class="p">]</span> <span class="c">// gets only what you asked for?</span>
<span class="o">-</span> <span class="n">nums</span><span class="p">[</span><span class="m">3</span><span class="o">:</span><span class="m">6</span><span class="o">:</span><span class="nb">cap</span><span class="p">(</span><span class="n">nums</span><span class="p">)</span> <span class="o">-</span> <span class="m">3</span><span class="p">]</span> <span class="c">// gets you that extra capacity?</span></code></pre></figure>
<p>Ugly? Yes. But I like syntactic vinegar when doing things that aren’t straightforward.</p>
Watch ANY Video Faster2019-05-13T00:00:00+00:00https://blog.jpalardy.com/posts/watch-any-video-faster<p><a href="https://prodocs24.com/articles/watch-any-video-faster/">Azerbaijanian Translation</a>, thanks to Amir.<br />
<a href="http://expereb.com/mire-cualquier-video-mas-rapido/">Spanish Translation</a>, thanks to Laura.</p>
<p>I posted <a href="/posts/watch-youtube-faster/">Watch YouTube Faster</a> a few years ago.</p>
<p>My opinion hasn’t changed: for <em>some</em> videos, it’s WAY better to watch faster.</p>
<p>I have already saved hours (days?) of my life.
Watching faster also means that I watched things I wouldn’t have watched otherwise. (too long, lack of patience…)</p>
<p>What changed is what I use.</p>
<h2 id="video-speed-controller">Video Speed Controller</h2>
<p>There’s a Chrome extension called <a href="https://chrome.google.com/webstore/detail/video-speed-controller/nffaoalbilbmmfgbnbgppjihopabppdk">Video Speed Controller</a>.
It’s written by the awesome <a href="https://www.igvita.com/">Ilya Grigorik</a> and the <a href="https://github.com/igrigorik/videospeed">code is on Github</a>.</p>
<p><img src="/assets/videospeed/videospeed.png" alt="videospeed screenshot" /></p>
<p>What’s best about this is that it works with:</p>
<ul>
<li>youtube</li>
<li>netflix</li>
<li>coursera, edx</li>
<li>…</li>
</ul>
<p>but also with any HTML5 video out there! If it’s video and inside a web page,
chances are that it’s going to work for it.</p>
<p>You can control the speed in increments of 0.1x, and there’s a bunch of keyboard shortcuts (as shown in the screenshot).</p>
<p>This Chrome Extension is one of the first thing I install on any computer I use. It’s hard to go back once you get used to it.</p>
tmux Synchronized Panes2019-04-19T00:00:00+00:00https://blog.jpalardy.com/posts/tmux-synchronized-panes<p>tmux can send the text you type to multiple panes:</p>
<p><a href="https://sanctum.geek.nz/arabesque/sync-tmux-panes/"><img src="/assets/tmux-sync-panes/synchronize-panes.gif" alt="demo of tmux synchronized panes" /></a><br />
(<em><a href="https://sanctum.geek.nz/arabesque/sync-tmux-panes/">screenshot source</a></em>)</p>
<p>Even though I’ve used tmux for years, I never realized that it could do that
until my colleague (thanks <a href="https://twitter.com/laurence_man">@laurence_man</a>!)
told me about it.</p>
<p>It’s similar to <a href="https://github.com/duncs/clusterssh">clusterssh</a> <em>without</em> having to ssh anywhere. 😄</p>
<p>Call it from the tmux command line:</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text"># toggle
:setw synchronize-panes
# or, explicit on/off
:setw synchronize-panes on
:setw synchronize-panes off</code></pre></figure>
<h2 id="make-it-easy-key-binding">Make it easy: key binding</h2>
<p>Having to type <code class="language-plaintext highlighter-rouge">:setw synchronize-panes</code> is too much work. I added this to tmux.conf:</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text"># toggle synchronize-panes
bind C-x setw synchronize-panes
# that's prefix + ctrl-x ^^</code></pre></figure>
<p>I don’t necessarily recommend that key binding, pick something that works for
you. <a href="https://stackoverflow.com/questions/16325449/how-to-send-a-command-to-all-panes-in-tmux">Other people</a>
use <code class="language-plaintext highlighter-rouge">ctrl-x</code> (without prefix) but that conflicts with some bash/vim bindings.</p>
<h2 id="make-it-obvious-window-status">Make it obvious: window status</h2>
<p>If the panes are synchronized, I want an easy way to tell. I added this to tmux.conf:</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">setw -g window-status-current-format '#{?pane_synchronized,#[bg=red],}#I:#W'
setw -g window-status-format '#{?pane_synchronized,#[bg=red],}#I:#W'</code></pre></figure>
<p>The <code class="language-plaintext highlighter-rouge">pane_synchronized</code> variable can be used for conditional coloring in the window status. (<a href="https://superuser.com/questions/710498/tmux-visual-indication-if-panes-are-synchronized">inspiration</a>)</p>
<p><img src="/assets/tmux-sync-panes/window-status-red.png" alt="synchronized pane highlighted in red" /></p>
<p>Guess which window has synchronized panes ^</p>
<p>Feel free to check <a href="https://github.com/jpalardy/dotfiles/blob/main/tmux.conf">my tmux.conf</a> and grab the parts you like.</p>
Automatically Show Exit Codes in Bash2019-03-15T00:00:00+00:00https://blog.jpalardy.com/posts/automatically-show-exit-codes-in-bash<p>When software stops running, it produces an <a href="https://en.wikipedia.org/wiki/Exit_status">exit code</a>. In your shell (bash, zsh…),
you can show the exit code using the <code class="language-plaintext highlighter-rouge">$?</code> variable:</p>
<p><img src="/assets/exit-code-bash/terminal-with-exit-code.png" alt="terminal with exit code" /></p>
<figure class="highlight"><pre><code class="language-text" data-lang="text"># or, usually, as a one-liner...
$ echo something; echo $?</code></pre></figure>
<p>0 is good, <a href="https://www.tldp.org/LDP/abs/html/exit-status.html">any other value</a> means something went
wrong. Different values have different meanings, but it’s <a href="https://stackoverflow.com/questions/1101957/are-there-any-standard-exit-status-codes-in-linux">software specific</a>.</p>
<p>That works … but it’s awkward. If you need the check the exit code, you need
to remember running it <em>immediately</em> after the command you’re interested in.
Otherwise, <code class="language-plaintext highlighter-rouge">$?</code> will be set to the exit code of whatever you ran after.</p>
<h2 id="showing--automatically">Showing $? Automatically</h2>
<p>It doesn’t take too much Googling to find solutions to show the exit code after a command exits:</p>
<p><img src="/assets/exit-code-bash/terminal-with-loud-exit-code.png" alt="terminal with loud exit code" /></p>
<p>Here’s the code for your <code class="language-plaintext highlighter-rouge">.bashrc</code>:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># capture escape sequences for colors...</span>
<span class="nv">COLOR_RED</span><span class="o">=</span><span class="si">$(</span>tput setaf 1<span class="si">)</span>
<span class="nv">ATTR_RESET</span><span class="o">=</span><span class="si">$(</span>tput sgr0<span class="si">)</span>
<span class="nb">export </span><span class="nv">PROMPT_COMMAND</span><span class="o">=</span>show_exit_code
show_exit_code<span class="o">()</span> <span class="o">{</span>
<span class="nb">local exit</span><span class="o">=</span><span class="nv">$?</span>
<span class="k">if</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$exit</span><span class="s2">"</span> <span class="nt">-ne</span> 0 <span class="o">]</span><span class="p">;</span> <span class="k">then
</span><span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="k">${</span><span class="nv">COLOR_RED</span><span class="k">}</span><span class="s2">exit: </span><span class="k">${</span><span class="nv">exit</span><span class="k">}${</span><span class="nv">ATTR_RESET</span><span class="k">}</span><span class="s2">"</span>
<span class="k">fi</span>
<span class="o">}</span></code></pre></figure>
<p><a href="https://tldp.org/HOWTO/Bash-Prompt-HOWTO/x264.html">PROMPT_COMMAND</a>:</p>
<blockquote>
<p>Bash provides an environment variable called PROMPT_COMMAND.
The contents of this variable are executed as a regular Bash command just before Bash displays a prompt.</p>
</blockquote>
<p>What I like about this solution is that it doesn’t mess with <code class="language-plaintext highlighter-rouge">PS1</code> (unlike
<a href="https://gist.github.com/weibeld/f3b6e6187029924a9b3d">other</a>
<a href="https://stackoverflow.com/questions/16715103/bash-prompt-with-last-exit-code">solutions</a>)
and it doesn’t show anything when the exit code is 0. I’ve never been a fan of
complicated <code class="language-plaintext highlighter-rouge">PS1</code> with a lot of fancy colors and statuses.</p>
Dealing with Large Files in Vim2019-03-04T00:00:00+00:00https://blog.jpalardy.com/posts/dealing-with-large-files-in-vim<p>Vim is fast, that’s one of its main selling points. In general, it can deal with very large files easily.</p>
<p>But there are ways to ruin your Vim experience, and it’s not especially hard:</p>
<ul>
<li>opening LARGE files, with syntax highlighting</li>
<li>opening generated files (uglified JavaScript, for example), and letting <a href="https://github.com/w0rp/ale">ALE</a> run a linter on the contents</li>
</ul>
<p>That second case is the one that has burned me recently: it pegged my CPU to 100% for a few minutes and completely <em>blocked</em> Vim while
it tried to flag everything. The file itself wasn’t particularly large, it was less than 250KB.</p>
<p>The solution for these cases is to turn off <strong><em>everything</em></strong>: highlighting and plugins:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>vim <span class="nt">-u</span> NONE filename</code></pre></figure>
<p><code class="language-plaintext highlighter-rouge">man vim</code> says:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-u {vimrc} Use the commands in the file {vimrc} for initializations.
All the other initializations are skipped. Use this to
edit a special kind of files. It can also be used to skip
all initializations by giving the name "NONE". See ":help
initialization" within vim for more details.
</code></pre></div></div>
<p>If Vim is stuck, I haven’t found a way to fix that directly; <code class="language-plaintext highlighter-rouge">ctrl-c</code> or
<code class="language-plaintext highlighter-rouge">ctrl-\</code> won’t help in this case. I’ve had to <code class="language-plaintext highlighter-rouge">kill</code> Vim, although <code class="language-plaintext highlighter-rouge">15</code>/<code class="language-plaintext highlighter-rouge">TERM</code> is enough, to
make it stop.</p>
The Best Books I Read in 20182018-12-28T00:00:00+00:00https://blog.jpalardy.com/posts/best-books-i-read-2018<p>I have read many books in 2018; some good and some not so good.</p>
<p>Here are the books that deserve to be acknowledged. There is no particular
order.<br />
I broke down my recommendations by categories: <a href="#technical">technical</a>,
<a href="#non-fiction">non-fiction</a> and <a href="#fiction">fiction</a>.</p>
<p>I have similar posts for <a href="https://blog.jpalardy.com/posts/best-books-i-read-2015/">2015</a>, <a href="https://blog.jpalardy.com/posts/best-books-i-read-2016/">2016</a> and <a href="https://blog.jpalardy.com/posts/best-books-i-read-2017/">2017</a>.</p>
<hr />
<h2 id="technical">Technical</h2>
<h3 id="redis-in-action">Redis in Action</h3>
<p><a href="https://www.amazon.com/dp/1617290858/"><img class="book-cover" src="/assets/best-books-2018/1617290858.jpg" alt="Redis in Action" /></a></p>
<p>I feel that Redis is <em>misunderstood</em>. What can it do? What is it for? What about persistence?</p>
<p>Redis has a few in-memory data structures. Each data structure supports a few
operations. Most tutorials will cover those and send you on your way. It’s easy
to dismiss Redis as too simple.</p>
<p>But if you’re building distributed systems – and what modern application isn’t to some extent? –
you want to lean on people who have “been-there, done-that” before.</p>
<p><strong>Redis in Action</strong> reads like a bunch of war stories on how <em>not</em> to implement
systems in Redis. Then it walks you through the problems and the “right way” to
do things. I loved <a href="https://blog.jpalardy.com/posts/best-books-i-read-2017/#designing-data-intensive-applications">Designing Data-Intensive Applications</a>
last year, and <strong>Redis in Action</strong> reminded of that book. I can’t get enough of
stories on how to build real systems.</p>
<p>The book is available for free online, see below.</p>
<ul>
<li><a href="https://www.amazon.com/dp/1617290858/">amazon</a></li>
<li><a href="https://redislabs.com/community/ebook/">website</a></li>
</ul>
<h3 id="programming-rust">Programming Rust</h3>
<p><a href="https://www.amazon.com/dp/1491927283/"><img class="book-cover" src="/assets/best-books-2018/1491927283.jpg" alt="Programming Rust" /></a></p>
<blockquote>
<p>A language that doesn’t affect the way you think about programming, is not worth knowing.</p>
</blockquote>
<p><a href="https://en.wikiquote.org/wiki/Alan_Perlis#Epigrams_on_Programming,_1982">Alan Perlis</a></p>
<p>Rust: the language everybody is talking about. Or that’s how it feels. I wanted
to know what the big deal was, so I picked up this book to get an overview.</p>
<p>Maybe this isn’t an endorsement of this book as much as the Rust programming
language itself. Wow… most languages seem like a mix of good and bad
things, but Rust seems polished. The syntax makes sense, the standard library is full
of useful things, its designers have thought about things I hadn’t.</p>
<p>The “big idea” behind Rust is about creating a modern language <em>without</em> garbage collection.
It’s a compromise between usability and performance, but I think Rust really nailed it.
The compiler keeps track of what code <em>owns</em> a piece of memory, so it knows when it can deallocate it.
This is all done automatically, at compile time.</p>
<p>If you’re intrigued … you might want to go read up what all the fuss is about.</p>
<ul>
<li><a href="https://www.amazon.com/dp/1491927283/">amazon</a></li>
<li><a href="https://shop.oreilly.com/product/0636920040385.do">website</a></li>
</ul>
<hr />
<h2 id="non-fiction">Non-Fiction</h2>
<h3 id="toyota-kata">Toyota Kata</h3>
<p><a href="https://www.amazon.com/dp/0071635238/"><img class="book-cover" src="/assets/best-books-2018/0071635238.jpg" alt="Toyota Kata" /></a></p>
<p>I have read many books about Toyota and I always find them interesting. But,
with so many books about Toyota, what else is there to talk about?</p>
<p>Many companies read about Toyota, implement some of the solutions but they
don’t necessarily work. The main insight is that people focus on what Toyota
does (kanban, for example) without trying to understand how Toyota came up with
that solution. The “Toyota Way” is a way of thinking rather than a series of
practices.</p>
<p>One story in the book struck me, consultants being upset at Toyota employees:</p>
<blockquote>
<p>…a few years ago you were doing X, but now you’re doing Y?! Which one is right?</p>
</blockquote>
<p>The solutions change <em>because</em> the problems change.</p>
<p>No wonder the Japanese can open their factories to visitors… the secret isn’t
<em>what</em> they do, it’s <em>how</em> they decide what they do.</p>
<ul>
<li><a href="https://www.amazon.com/dp/0071635238/">amazon</a></li>
<li><a href="https://en.wikipedia.org/wiki/Toyota_Kata">wikipedia</a></li>
<li><a href="http://www-personal.umich.edu/~mrother/Homepage.html">website</a></li>
</ul>
<h3 id="we-have-no-idea">We Have No Idea</h3>
<p><a href="https://www.amazon.com/dp/0735211523/"><img class="book-cover" src="/assets/best-books-2018/0735211523.jpg" alt="We Have No Idea" /></a></p>
<p><a href="https://www.youtube.com/user/voxdotcom">I</a>
<a href="https://www.youtube.com/user/CGPGrey">love</a>
“<a href="https://www.youtube.com/user/Kurzgesagt">explainers</a>”; the people who explain
topics in a way that anybody can understand. There’s too much to learn and not
enough time. If someone can do the research and give me a good
overview in a fraction of the time, that’s awesome.</p>
<p>It also means that these overviews are often “shallow” and “oversimplified”… but I
think that’s a reasonable compromise. You can always dig deeper if you find it interesting.</p>
<p><strong>We Have No Idea</strong> focuses on physics and tries to explain:</p>
<ul>
<li>dark matter</li>
<li>dark energy</li>
<li>elementary particles</li>
<li>space</li>
<li>time</li>
<li>and more…</li>
</ul>
<p>There is <em>so much</em> we don’t know … and I wasn’t fully aware of that until I
read this book. If you thought we had physics “solved” and the main questions
answered, this book will give you a fresh perspective.</p>
<p>Here’s an example of what’s covered in the book:</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/4GbWfNHtHRg" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<p><br /></p>
<ul>
<li><a href="https://www.amazon.com/dp/0735211523/">amazon</a></li>
<li><a href="https://phdcomics.com/noidea/">website</a></li>
</ul>
<h3 id="sapiens">Sapiens</h3>
<p><a href="https://www.amazon.com/dp/0062316095/"><img class="book-cover" src="/assets/best-books-2018/0062316095.jpg" alt="Sapiens" /></a></p>
<p><strong>Sapiens</strong> … the unavoidable? At one point, it was everywhere and it seems like
everybody was reading the book.</p>
<p>In a way, Yuval Noah Harari is another “explainer”. This book is <em>FILLED</em> with
interesting ideas … about things I never thought about.</p>
<p>It’s a history book; it describes what happened but also covers <em>why</em>. It also
has an overarching theory of everything which is very satisfying. Suddenly, a lot of things make sense.</p>
<p>My complaint is the same I have with other “explainers”… when it came to
topics I am already familiar with, I found the coverage to be superficial (other
books did it better!) and misleading (it’s not the full story!).</p>
<ul>
<li><a href="https://www.amazon.com/dp/0062316095/">amazon</a></li>
<li><a href="https://en.wikipedia.org/wiki/Sapiens:_A_Brief_History_of_Humankind">wikipedia</a></li>
<li><a href="https://www.ynharari.com/book/sapiens/">website</a></li>
</ul>
<hr />
<h2 id="fiction">Fiction</h2>
<h3 id="altered-carbon">Altered Carbon</h3>
<p><em>Series: 3 books, complete.</em></p>
<p><a href="https://www.amazon.com/dp/B0009I6MC8/"><img class="book-cover" src="/assets/best-books-2018/B0009I6MC8.jpg" alt="Altered Carbon" /></a></p>
<p>When I searched for “recommended science fiction”, <strong>Altered Carbon</strong> often made the short list.
When I heard that it was coming to Netflix, I decided to read it first.</p>
<p>In a way, it’s a classic “detective story”. There’s been a murder, nobody can
figure it out, an “expert” is brought in … you follow along and the
conclusions are shocking. Maybe you beat yourself up for not figuring it out
yourself. (it was so obvious in retrospect!) Since this is science fiction,
there is an interesting twist: the client is looking for an explanation for why
<em>he</em> was murdered a few days before.</p>
<p>Although the story follows familiar detective novel lines, I think the appeal
of the book is the exploration of “what does it mean to be <em>you</em>?” In the world
of <strong>Altered Carbon</strong>, you can transfer your conciousness to different bodies:</p>
<ul>
<li>what does it mean to be a man/woman when you can transfer bodies?</li>
<li>what does <em>race</em> mean when you can transfer bodies?</li>
<li>what does it mean to live forever by jumping from an older to a younger body?</li>
<li>who’s the <em>real</em> you if your conciousness exists in multiple bodies at the same time?</li>
<li>what is death if you can wake up in another body after a few “lost” hours?</li>
</ul>
<p><strong>Altered Carbon</strong> was written in 2002, and it feels a bit dated now. Most of these
themes have been picked up in other books. I think it was more
shocking/revolutionary when it came out.</p>
<p>There are 2 other books in the series, but I think <strong>Altered Carbon</strong> stands by itself.
I don’t think it’s necessary to read the sequels.</p>
<ul>
<li><a href="https://www.amazon.com/dp/B0009I6MC8/">amazon</a></li>
<li><a href="https://en.wikipedia.org/wiki/Altered_Carbon">wikipedia</a></li>
</ul>
<h3 id="112263">11.22.63</h3>
<p><a href="https://www.amazon.com/dp/B0064HJNQK/"><img class="book-cover" src="/assets/best-books-2018/B0064HJNQK.jpg" alt="11.22.63" /></a></p>
<p>I have a history with Stephen King … some of the first “serious” books I read were from him.
Even though I enjoy his writing, it’s not a style I automatically reach for.</p>
<p>Many people mentioned <strong>11.22.63</strong>, so I decided to give it a try. It had been a
while since I had read Stephen King and I had forgotten how … <em>rich</em> the
universes he constructs are. More than most authors I read, I really feel the
characters and places he describes. He made me feel nostalgia for 1958!</p>
<p>If you could travel back in time, could you save Kennedy? Without giving away
too much – what if time itself were the “bad guy”?</p>
<p>Most horror leaves me feeling hollow: many people died, some survived …
it’s often disappointing. This book is no different … but the writing is so
rich and full that it makes me forgive.</p>
<ul>
<li><a href="https://www.amazon.com/dp/B0064HJNQK/">amazon</a></li>
<li><a href="https://en.wikipedia.org/wiki/11/22/63">wikipedia</a></li>
</ul>
<h3 id="children-of-time">Children of Time</h3>
<p><a href="https://www.amazon.com/dp/B06ZXTHNSJ/"><img class="book-cover" src="/assets/best-books-2018/B06ZXTHNSJ.jpg" alt="Children of Time" /></a></p>
<p>The reviews looked good, but it was just one book. If I didn’t like it, it would only be one book.
But if I liked it, it would still be only one book… (a typical fiction-picking dilemma)</p>
<p>In the far future, the last humans and <a href="https://en.wikipedia.org/wiki/Uplift_(science_fiction)">uplifted</a>
spiders come head to head in a conflict for survival.</p>
<ul>
<li>do humans deserve a second chance?</li>
<li>is human survival worth destroying the chance of other species?</li>
</ul>
<p>Similar to other books covering extremely long timelines, there is plenty of
interesting ground to cover. I enjoyed reading even though I wasn’t sure where it
was headed.</p>
<p>Also, I found the ending satisfying and that’s worth a lot.</p>
<ul>
<li><a href="https://www.amazon.com/dp/B06ZXTHNSJ/">amazon</a></li>
<li><a href="https://en.wikipedia.org/wiki/Children_of_Time_(novel)">wikipedia</a></li>
</ul>
<h3 id="ball-lightning">Ball Lightning</h3>
<p><a href="https://www.amazon.com/dp/B07F9ZBBVW/"><img class="book-cover" src="/assets/best-books-2018/B07F9ZBBVW.jpg" alt="Ball Lightning" /></a></p>
<p>After <a href="https://blog.jpalardy.com/posts/best-books-i-read-2015/#the-three-body-problem">The Three-Body Problem</a>
series, I was willing to give anything Cixin Liu writes a go. This is a
self-standing book, but also a sort of (optional) <em>prequel</em> to the Three-Body
series.</p>
<p>While reading the first part of the book, I was reminded of the feelings I had when I read Cixin Liu before:</p>
<ul>
<li>this is boring…</li>
<li>where is this going?</li>
</ul>
<p>But, like before, even the irrelevant parts all came together into “oh shit!” moments later on. Some fiction
is enjoyable even if predictable … in some ways, this is the opposite. You go along for the ride because it
takes you places you never thought about.</p>
<p>This one isn’t for everybody. I would start with Three-Body and pick this one up if you’re hungry for more.</p>
<ul>
<li><a href="https://www.amazon.com/dp/B07F9ZBBVW/">amazon</a></li>
<li><a href="https://en.wikipedia.org/wiki/Ball_Lightning_(novel)">wikipedia</a></li>
<li><a href="https://www.tor.com/2016/09/13/cover-reveal-ball-lightning-cixin-liu/">website</a></li>
</ul>
<h3 id="all-systems-red">All Systems Red</h3>
<p><em>Series: 4 books, complete.</em></p>
<p><a href="https://www.amazon.com/dp/B076X98RLL/"><img class="book-cover" src="/assets/best-books-2018/B076X98RLL.jpg" alt="All Systems Red" /></a></p>
<p>Ah… the Murderbot Diaries. If robots that have sex are “sexbots”, robots
that murder would be “murderbots”.</p>
<p>Imagine that you are a security cyborg, that you hacked your own software so
you could make your own decisions. You kill (or protect) people for a living,
but would rather watch soap operas. There’s <em>a bit</em> more to it than that.</p>
<p>The books are short and hard to put down once started. They explore different
non-human perspectives on being alive. This is a hard one <em>not</em> to recommend.</p>
<ul>
<li><a href="https://www.amazon.com/dp/B076X98RLL/">amazon</a></li>
<li><a href="https://en.wikipedia.org/wiki/All_Systems_Red">wikipedia</a></li>
<li><a href="https://www.marthawells.com/murderbot.htm">website</a></li>
</ul>
<hr />
<h2 id="closing-thoughts">Closing Thoughts</h2>
<p>If you have liked this blog post, please write your own. I want to read your book
reviews and recommendations.</p>
The Family Emoji2018-10-28T00:00:00+00:00https://blog.jpalardy.com/posts/the-family-emoji<p>I was reading <a href="https://exploringjs.com/impatient-js/index.html">Impatient JS</a> and came up on a <a href="http://exploringjs.com/impatient-js/ch_strings.html#caveat-grapheme-clusters">surprising part</a>:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="o">></span> <span class="p">...[</span><span class="dl">'</span><span class="s1">👨👩👦</span><span class="dl">'</span><span class="p">]</span>
<span class="p">[</span> <span class="dl">'</span><span class="s1">👨</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1"></span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">👩</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1"></span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">👦</span><span class="dl">'</span> <span class="p">]</span> <span class="c1">// cool: man, woman, and boy emojis...</span></code></pre></figure>
<p>but when my friend <a href="https://twitter.com/jzhang729">Jordan</a> tried it:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="o">></span> <span class="p">[...</span><span class="dl">'</span><span class="s1">👪</span><span class="dl">'</span><span class="p">]</span>
<span class="p">[</span> <span class="dl">'</span><span class="s1">👪</span><span class="dl">'</span> <span class="p">]</span></code></pre></figure>
<p>What… why?! Is the book wrong?</p>
<h1 id="there-are-2-family-emojis">There are 2 “family” emojis…</h1>
<p>If you already know the answer, the book is clear… When you search for the
“family emoji” and end up on a <a href="https://unicode.org/emoji/charts/full-emoji-list.html#family">page like this</a>,
you’ll see what’s going on:</p>
<p><img src="/assets/emoji-family/two-families.png" alt="two family graphemes" /></p>
<p>There’s one codepoint that’s already the family emoji. But the same result can be obtained by combining a series of codepoints:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="o">></span> <span class="p">[...</span><span class="dl">'</span><span class="s1">👪</span><span class="dl">'</span><span class="p">].</span><span class="nx">map</span><span class="p">(</span><span class="nx">c</span> <span class="o">=></span> <span class="nx">c</span><span class="p">.</span><span class="nx">codePointAt</span><span class="p">(</span><span class="mi">0</span><span class="p">))</span>
<span class="p">[</span> <span class="mi">128106</span> <span class="p">]</span>
<span class="o">></span> <span class="p">[...</span><span class="dl">'</span><span class="s1">👨👩👦</span><span class="dl">'</span><span class="p">].</span><span class="nx">map</span><span class="p">(</span><span class="nx">c</span> <span class="o">=></span> <span class="nx">c</span><span class="p">.</span><span class="nx">codePointAt</span><span class="p">(</span><span class="mi">0</span><span class="p">))</span>
<span class="p">[</span> <span class="mi">128104</span><span class="p">,</span> <span class="mi">8205</span><span class="p">,</span> <span class="mi">128105</span><span class="p">,</span> <span class="mi">8205</span><span class="p">,</span> <span class="mi">128102</span> <span class="p">]</span>
<span class="c1">// or, in hex</span>
<span class="o">></span> <span class="p">[...</span><span class="dl">'</span><span class="s1">👪</span><span class="dl">'</span><span class="p">].</span><span class="nx">map</span><span class="p">(</span><span class="nx">c</span> <span class="o">=></span> <span class="nx">c</span><span class="p">.</span><span class="nx">codePointAt</span><span class="p">(</span><span class="mi">0</span><span class="p">).</span><span class="nx">toString</span><span class="p">(</span><span class="mi">16</span><span class="p">))</span>
<span class="p">[</span> <span class="dl">'</span><span class="s1">1f46a</span><span class="dl">'</span> <span class="p">]</span>
<span class="o">></span> <span class="p">[...</span><span class="dl">'</span><span class="s1">👨👩👦</span><span class="dl">'</span><span class="p">].</span><span class="nx">map</span><span class="p">(</span><span class="nx">c</span> <span class="o">=></span> <span class="nx">c</span><span class="p">.</span><span class="nx">codePointAt</span><span class="p">(</span><span class="mi">0</span><span class="p">).</span><span class="nx">toString</span><span class="p">(</span><span class="mi">16</span><span class="p">))</span>
<span class="p">[</span> <span class="dl">'</span><span class="s1">1f468</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">200d</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">1f469</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">200d</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">1f466</span><span class="dl">'</span> <span class="p">]</span></code></pre></figure>
<p>These are the same hexadecimal values as the screenshot above. Everything is right with the world :-)</p>
<p>The U+200d character is the <a href="https://en.wikipedia.org/wiki/Zero-width_joiner">ZERO WIDTH JOINER</a>, abbreviated to ZWJ.
The <a href="https://en.wikipedia.org/wiki/Emoji#Joining">joining section</a> from the emoji Wikipedia page gives context that
will feel very familiar (pun intended).</p>
<h1 id="there-are-all-kinds-of-families">There are all kinds of families</h1>
<p>The example above is just one of the many emoji families available:</p>
<div style="font-size: 300%">
👨👩👦 👨👩👧 👨👩👧👦 👨👩👦👦 👨👩👧👧
</div>
<div style="font-size: 300%">
👨👨👦 👨👨👧 👨👨👧👦 👨👨👦👦 👨👨👧👧
</div>
<div style="font-size: 300%">
👩👩👦 👩👩👧 👩👩👧👦 👩👩👦👦 👩👩👧👧
</div>
<div style="font-size: 300%">
👨👦 👨👦👦 👨👧 👨👧👦 👨👧👧
</div>
<div style="font-size: 300%">
👩👦 👩👦👦 👩👧 👩👧👦 👩👧👧
</div>
<h1 id="additional-resources">Additional Resources</h1>
<p>I had heard of these things but hadn’t fully grasped them until I played with them myself. While
I was trying to figure this out, I found a bunch of good resources:</p>
<ul>
<li>Wes Bos has a <a href="https://twitter.com/wesbos/status/769228067780825088?lang=en">tweet with cool examples</a></li>
<li>unicode.org’s <a href="https://unicode.org/emoji/charts/full-emoji-list.html">full list of emojis</a></li>
<li>Good intro to <a href="https://manishearth.github.io/blog/2017/01/14/stop-ascribing-meaning-to-unicode-code-points/">code points vs grapheme clusters</a></li>
<li>More <a href="https://manishearth.github.io/blog/2017/01/15/breaking-our-latin-1-assumptions/#emoji">emoji fun</a></li>
<li>Cry a little: <a href="https://eev.ee/blog/2015/09/12/dark-corners-of-unicode/">Dark Corners of Unicode</a></li>
<li><a href="https://github.com/jagracey/Awesome-Unicode">Awesome Unicode</a>, which I’m just starting to read</li>
</ul>
JavaScript tooling: prettier2018-09-26T00:00:00+00:00https://blog.jpalardy.com/posts/javascript-tooling-prettier<p><a href="https://prettier.io/">Prettier</a> is an awesome code formatter.</p>
<p>I was first exposed to code formatters when I was learning Golang. I thought
<a href="https://golang.org/cmd/gofmt/">gofmt</a> was one of the most interesting aspects
of the language. Finally, we could all format our code the same way and skip
the endless discussions about style and where things should go. Prettier aims
to do the same for JavaScript, and a <a href="https://github.com/prettier/prettier#opinionated-code-formatter">bunch of other
languages</a>.</p>
<p>Prettier allows you to type messy code, press a key, and <em>probably</em> get it in a
better shape than you would have done yourself.</p>
<p>Using their own example (but slightly shortened), you can transform:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="c1">// from:</span>
<span class="nx">foo</span><span class="p">(</span><span class="nx">reallyLongArg</span><span class="p">(),</span> <span class="nx">omgSoManyParameters</span><span class="p">(),</span> <span class="nx">IShouldRefactorThis</span><span class="p">());</span>
<span class="c1">// to:</span>
<span class="nx">foo</span><span class="p">(</span>
<span class="nx">reallyLongArg</span><span class="p">(),</span>
<span class="nx">omgSoManyParameters</span><span class="p">(),</span>
<span class="nx">IShouldRefactorThis</span><span class="p">()</span>
<span class="p">);</span></code></pre></figure>
<h1 id="except-when-it-doesnt">Except when it doesn’t…</h1>
<p>Prettier doesn’t have too many <a href="https://prettier.io/docs/en/options.html">options</a>, and the idea of
configuring Prettier seems to go against the <a href="https://prettier.io/docs/en/option-philosophy.html">philosophy</a> of the tool itself.</p>
<p>The setting that has the most significant impact on the final result is <a href="https://prettier.io/docs/en/options.html#print-width">printWidth</a>.
The documentation itself warns about setting the value too high:</p>
<blockquote>
<p>Prettier, on the other hand, strives to fit the most code into every line. With
the print width set to 120, prettier may produce overly compact, or otherwise
undesirable code.</p>
</blockquote>
<p><code class="language-plaintext highlighter-rouge">printWidth</code> defaults to 80, which felt claustrophobic… but higher values, ironically, will transform:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="c1">// from:</span>
<span class="nx">foo</span><span class="p">(</span>
<span class="nx">reallyLongArg</span><span class="p">(),</span>
<span class="nx">omgSoManyParameters</span><span class="p">(),</span>
<span class="nx">IShouldRefactorThis</span><span class="p">()</span>
<span class="p">);</span>
<span class="c1">// back to:</span>
<span class="nx">foo</span><span class="p">(</span><span class="nx">reallyLongArg</span><span class="p">(),</span> <span class="nx">omgSoManyParameters</span><span class="p">(),</span> <span class="nx">IShouldRefactorThis</span><span class="p">());</span></code></pre></figure>
<p>I found a <em>workaround</em>: if you add a single line comment, you force Prettier into a situation
where it cannot inline your code without getting rid of the comment (which it won’t):</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="c1">// locked by the comment:</span>
<span class="nx">foo</span><span class="p">(</span>
<span class="nx">reallyLongArg</span><span class="p">(),</span> <span class="c1">// ⏎</span>
<span class="nx">omgSoManyParameters</span><span class="p">(),</span>
<span class="nx">IShouldRefactorThis</span><span class="p">()</span>
<span class="p">);</span></code></pre></figure>
<p>I use a ⏎ (Unicode character: <a href="https://en.wikipedia.org/wiki/Carriage_return">carriage return</a>) as a convention in my own code, but what
you put in the comment is up to you.</p>
<h1 id="prettier-is-not-a-linter">Prettier is not a linter</h1>
<p>Prettier will fix the <em>format</em> of your code, but it doesn’t have opinions about
the <em>quality</em> of your code.</p>
<p>For example, Prettier would not complain about:</p>
<ul>
<li>global or missing variables</li>
<li>the use of <code class="language-plaintext highlighter-rouge">var</code> and other obsolete language features</li>
<li>unused variables</li>
<li>and other horrors…</li>
</ul>
<p>(<a href="https://prettier.io/docs/en/comparison.html">read more</a>)</p>
<p>Yes, you will probably still need <a href="https://eslint.org/">ESLint</a>. You will need to find a config that’s compatible with Prettier’s rules.
I use <a href="https://www.npmjs.com/package/eslint-config-airbnb-base">airbnb-base</a> with some <a href="https://github.com/jpalardy/dotfiles/blob/056f78f561ebb71193751ef0499f82fa0364e3cf/eslintrc">tweaks</a>.
You can also check <a href="https://prettier.io/docs/en/eslint.html">Integrating with ESLint</a>.</p>
<h1 id="getting-started">Getting Started</h1>
<p>Check <a href="https://prettier.io/docs/en/editors.html">Editor Integration</a> for details.</p>
<p>Most integrations have 2 modes:</p>
<ul>
<li>manual: a keyboard shortcut reformats the code</li>
<li>automatic: saving a file reformats the code</li>
</ul>
<p>Depending on the state of the codebase you’re in, I find <code class="language-plaintext highlighter-rouge">manual</code>
mode makes more sense; I don’t always want to reformat every file I touch.</p>
JavaScript tooling: madge2018-08-23T00:00:00+00:00https://blog.jpalardy.com/posts/javascript-tooling-madge<p>In large projects, the way files depend on each other is not always clear.</p>
<h1 id="dependencies">Dependencies</h1>
<p>What files does <em>this</em> file depend on?</p>
<blockquote>
<p>Easy: look for the <code class="language-plaintext highlighter-rouge">require</code> or <code class="language-plaintext highlighter-rouge">import</code> statements.</p>
</blockquote>
<p>What files depend on <em>this</em> file?</p>
<blockquote>
<p>Uh… use <code class="language-plaintext highlighter-rouge">grep</code>?</p>
</blockquote>
<p>You can get answers, but it’s the <em>micro</em> view. What about the big picture?<br />
Let’s look at ALL files and ALL dependencies at the same time.</p>
<h1 id="madge">Madge</h1>
<p><a href="https://github.com/pahen/madge">Madge</a></p>
<blockquote>
<p>Create graphs from your CommonJS, AMD or ES6 module dependencies</p>
</blockquote>
<p>Short version:</p>
<ul>
<li>step into a project</li>
<li>run <code class="language-plaintext highlighter-rouge">madge</code>, feed it the source files</li>
<li>look at the generated image</li>
</ul>
<p>Here’s what happens when you run Madge on itself:</p>
<p><img src="/assets/surviving-unfamiliar/madge-on-madge.svg" alt="madge on madge" /></p>
<p>(you might need to install <a href="https://www.graphviz.org/">graphviz</a>, follow the <a href="https://github.com/pahen/madge#graphviz-optional">instructions</a>)</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># HOW TO ^^^</span>
git clone https://github.com/pahen/madge.git
<span class="nb">cd </span>madge/
npm <span class="nb">install
</span>node bin/cli.js bin <span class="nt">-i</span> madge-on-madge.svg</code></pre></figure>
<p>But what about a bigger project? Here’s <a href="https://github.com/facebook/immutable-js/">immutable.js</a> (click to enlarge):</p>
<p><a href="/assets/surviving-unfamiliar/immutable.png"><img src="/assets/surviving-unfamiliar/immutable.png" alt="immutable" /></a></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># HOW TO ^^^</span>
git clone https://github.com/facebook/immutable-js.git
<span class="nb">cd </span>immutable-js/
madge src <span class="nt">-i</span> immutable.png <span class="c"># assumes: `npm install -g madge`</span></code></pre></figure>
<p>If you want to focus on a part of the graph, you can start at a specific file (Seq.js):</p>
<p><a href="/assets/surviving-unfamiliar/seq.png"><img src="/assets/surviving-unfamiliar/seq.png" alt="part of immutable" /></a></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># HOW TO ^^^</span>
madge src/Seq.js <span class="nt">-i</span> seq.png</code></pre></figure>
<p>What do the colors mean?</p>
<ul>
<li>blue files have dependencies</li>
<li>green files DO NOT have dependencies</li>
<li>red files have circular dependencies (!)</li>
</ul>
<h1 id="how-to-use">How to use?</h1>
<p>The <a href="https://github.com/pahen/madge#cli">README</a> has a bunch of examples.</p>
<p>You can:</p>
<ul>
<li>run madge only on specific files, directories, or combinations of both</li>
<li>show dependencies</li>
<li>show circular dependencies</li>
<li>show “orphans” – files not required by any other files (!)</li>
</ul>
<p>It also supports many formats:</p>
<ul>
<li>text</li>
<li>JSON</li>
<li>dot (graphviz)</li>
<li>image (png, svg, …)</li>
</ul>
<p>You can even exclude files:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">madge <span class="nt">--exclude</span> <span class="s1">'^(foo|bar)\.js$'</span> path/src/app.js</code></pre></figure>
<p>which is useful for non-interesting files (i.e. configuration files, or tests).</p>
Surviving Unfamiliar JavaScript Projects: dependencies2018-06-25T00:00:00+00:00https://blog.jpalardy.com/posts/surviving-unfamiliar-javascript-projects-dependencies<p>The chances to find a JavaScript project with all dependencies correct and
up-to-date is almost zero. For some dependencies, just stepping out for coffee
<em>almost</em> puts you at risk of falling behind.</p>
<p>It’s not a question of “falling behind”, that’s a given. It’s a question
of figuring out what packages are outdated and by how much.</p>
<h1 id="npm-outdated">npm outdated</h1>
<p>A good first step is built-in npm: when you run <code class="language-plaintext highlighter-rouge">npm outdated</code>, you get an output similar to this:</p>
<p><img src="/assets/surviving-unfamiliar/npm-outdated.png" alt="npm outdated output" /></p>
<p>“Package” is obvious, but let’s clarify the others:</p>
<ul>
<li>Current: what’s currently installed locally in <code class="language-plaintext highlighter-rouge">node_modules</code></li>
<li>Wanted: the maximum version compatible with the semver in your <code class="language-plaintext highlighter-rouge">package.json</code></li>
<li>Latest: the most up-to-date, the latest, version in npm</li>
</ul>
<p>You need to be responsible with when and how you upgrade. I enjoy
the lets-see-what-breaks approach to upgrading… it’s often faster and more
definitive than researching CHANGELOGs and READMEs for clues.</p>
<p>If you want to jump to the latest version of everything, let’s talk about <code class="language-plaintext highlighter-rouge">ncu</code>.</p>
<h1 id="ncu">ncu</h1>
<p><code class="language-plaintext highlighter-rouge">ncu</code>, from the <a href="https://github.com/tjunnone/npm-check-updates">npm-check-updates</a> package, also tries to evaluate your
<code class="language-plaintext highlighter-rouge">package.json</code> against the package versions out there:</p>
<p><img src="/assets/surviving-unfamiliar/ncu.png" alt="ncu output" /></p>
<p>The output might be a <em>little</em> easier to parse compared to <code class="language-plaintext highlighter-rouge">npm outdated</code>. But
the real value happens if you run <code class="language-plaintext highlighter-rouge">ncu -u</code>: it will update your <code class="language-plaintext highlighter-rouge">package.json</code>
with the versions from the 2nd column.</p>
<p>Even if you don’t want all the changes from <code class="language-plaintext highlighter-rouge">ncu -u</code>, I find it’s easier to run
it and revert lines from the <code class="language-plaintext highlighter-rouge">package.json</code> manually. It certainly feels faster
than copy-pasting version numbers in <code class="language-plaintext highlighter-rouge">package.json</code>.</p>
<h1 id="nsp-and-npm-audit">nsp and npm audit</h1>
<p>If you’re not ready to upgrade, you might still want to check if you’re relying on packages
with known security problems. <a href="https://github.com/nodesecurity/nsp">nsp</a> (Node Security Platform) provides
a tool to audit the packages you depend on:</p>
<p><a href="/assets/surviving-unfamiliar/nsp.png"><img src="/assets/surviving-unfamiliar/nsp.png" alt="nsp output" /></a>
(output truncated ^)</p>
<p>As mentioned in the <a href="https://github.com/nodesecurity/nsp">nsp README</a>:</p>
<blockquote>
<p>The Node Security Platform has been acquired by npm</p>
</blockquote>
<p>As of npm version 6, you can get a similar output to <code class="language-plaintext highlighter-rouge">nsp</code> with <code class="language-plaintext highlighter-rouge">npm audit</code>:</p>
<p><a href="/assets/surviving-unfamiliar/npm-audit.png"><img src="/assets/surviving-unfamiliar/npm-audit.png" alt="npm audit output" /></a>
(output truncated ^)</p>
<h1 id="depcheck">depcheck</h1>
<p><a href="https://github.com/depcheck/depcheck">depcheck</a> does not <em>trust</em> what your
<code class="language-plaintext highlighter-rouge">package.json</code> says, it goes out into the project’s code and <em>verifies</em>. For
each dependency, it checks that it is <code class="language-plaintext highlighter-rouge">require</code>d. Conversely, it checks that
each <code class="language-plaintext highlighter-rouge">require</code> is present in the <code class="language-plaintext highlighter-rouge">package.json</code>.</p>
<p><img src="/assets/surviving-unfamiliar/depcheck.png" alt="depcheck" /></p>
<p>It’s not perfect, but it’s better than the alternative (hoping everything is
ok?). There are ways to configure it to ignore false positives too. Check the
<a href="https://github.com/depcheck/depcheck">README</a> for more details.</p>
<h1 id="searching-for-packages">Searching for packages</h1>
<p>Whenever I encounter a package I’m not familiar with, I check the Github’s
README to give me an idea of what it’s used for.</p>
<p>For example, what is <a href="https://github.com/kentcdodds/cross-env">cross-env</a>?</p>
<p>There are more direct ways to find a project’s repo than Googling it.
Open a terminal anywhere (whether in a project or not) and type:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>npm home cross-env</code></pre></figure>
<p>That will open your browser to the <code class="language-plaintext highlighter-rouge">homepage</code> entry of a package’s <code class="language-plaintext highlighter-rouge">package.json</code>.</p>
<p>You can also search package’s name on <a href="https://www.npmjs.com/">npmjs.com</a>
(here: <a href="https://www.npmjs.com/search?q=cross-env">cross-env</a>)</p>
<p>But I found that using <a href="https://yarnpkg.com/en/">yarnpkg.com</a> (here:
<a href="https://yarnpkg.com/en/packages?q=cross-env">cross-env</a>) is the best
experience. You don’t have to use yarn to enjoy their search engine.</p>
<p>The number of downloads in the last 30 days gives you an idea about
how popular different projects are:</p>
<p><img src="/assets/surviving-unfamiliar/cross-env.png" alt="cross-env" /></p>
Surviving Unfamiliar JavaScript Projects: package.json2018-06-17T00:00:00+00:00https://blog.jpalardy.com/posts/surviving-unfamiliar-javascript-projects-package-json<p>This isn’t your code or project: you <code class="language-plaintext highlighter-rouge">git clone</code> and step into the directory…</p>
<p>Where do you start? What’s important? How are things organized?</p>
<h1 id="packagejson">package.json</h1>
<p>A project’s <code class="language-plaintext highlighter-rouge">package.json</code> will tell you <a href="https://docs.npmjs.com/files/package.json">a lot</a>.
All sections are informative, but some are better starting points than others.</p>
<p>Is that too obvious? You can play along with the <code class="language-plaintext highlighter-rouge">package.json</code> from real projects. The more unfamiliar the project, the better:</p>
<ul>
<li><a href="https://github.com/visionmedia/debug/blob/master/package.json">debug</a></li>
<li><a href="https://github.com/trentm/node-bunyan/blob/master/package.json">bunyan</a></li>
<li><a href="https://github.com/lodash/lodash/blob/master/package.json">lodash</a></li>
<li><a href="https://github.com/zeit/ms/blob/master/package.json">ms</a></li>
<li><a href="https://github.com/moment/moment/blob/develop/package.json">moment</a></li>
</ul>
<p>Any surprises in there? My answers/spoilers <a href="#surprises-from-real-packagejson">below</a>.</p>
<h2 id="section-main">section: main</h2>
<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="nl">"main"</span><span class="p">:</span><span class="w"> </span><span class="s2">"index.js"</span></code></pre></figure>
<p>A good place to start … if you’re going to jump in and read some code, this
is the entry point of this project; this is what a <code class="language-plaintext highlighter-rouge">require</code> would read.</p>
<h2 id="section-bin">section: bin</h2>
<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="nl">"bin"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"nyc"</span><span class="p">:</span><span class="w"> </span><span class="s2">"./bin/nyc.js"</span><span class="w">
</span><span class="p">}</span></code></pre></figure>
<p>Does this project provide “executables”? If so, that’s another kind of entry point
into the code. It comes from a different perspective: doing something useful and specific.</p>
<p>Sometimes it’s obvious what’s in there, sometimes not. It’s usually worth a look.</p>
<h2 id="section-scripts">section: scripts</h2>
<p><em>this snippet was trimmed down to simplify</em></p>
<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="nl">"scripts"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"test"</span><span class="p">:</span><span class="w"> </span><span class="s2">"npm run clean && npm run build && ..."</span><span class="p">,</span><span class="w">
</span><span class="nl">"clean"</span><span class="p">:</span><span class="w"> </span><span class="s2">"rimraf ./.nyc_output ./node_modules/.cache ..."</span><span class="p">,</span><span class="w">
</span><span class="nl">"build"</span><span class="p">:</span><span class="w"> </span><span class="s2">"node ./build-tests"</span><span class="w">
</span><span class="err">...</span><span class="w">
</span><span class="p">}</span></code></pre></figure>
<p>Other languages have Makefiles, Rakefiles, or something else. The <code class="language-plaintext highlighter-rouge">scripts</code> section ends up
being a dumping ground for useful “commands” and “recipes”.</p>
<p>Look for the unexpected.</p>
<h2 id="section-dependencies">section: *dependencies</h2>
<p>People don’t write code anymore, people <a href="https://www.wisdomandwonder.com/link/2110/why-mit-switched-from-scheme-to-python">glue code</a> together until it’s useful.</p>
<p>Yes, <code class="language-plaintext highlighter-rouge">dependencies</code> and <code class="language-plaintext highlighter-rouge">devDependencies</code>
(<a href="https://docs.npmjs.com/files/package.json#optionaldependencies">and</a> <a href="https://docs.npmjs.com/files/package.json#peerdependencies">others</a>)
contain “everything”, but it doesn’t tell you where and why.</p>
<p>Also, it’s probably not up to date:</p>
<ul>
<li>old versions</li>
<li>versions with security flaws</li>
<li>unused dependencies: not used but found in <code class="language-plaintext highlighter-rouge">package.json</code></li>
<li>missing dependencies: used but not found in <code class="language-plaintext highlighter-rouge">package.json</code></li>
</ul>
<p>The good news is that you don’t have to do this alone, <a href="/posts/surviving-unfamiliar-javascript-projects-dependencies/">there are tools to help</a>.</p>
<h1 id="surprises-from-real-packagejson">Surprises from real package.json</h1>
<p>When I read a <code class="language-plaintext highlighter-rouge">package.json</code>, I look for the unexpected. Here are some things that caught my eye:</p>
<p><a href="https://github.com/visionmedia/debug/blob/master/package.json">debug</a></p>
<ul>
<li>What’s the <a href="https://docs.npmjs.com/files/package.json#browser">browser</a> section?</li>
</ul>
<p><a href="https://github.com/trentm/node-bunyan/blob/master/package.json">bunyan</a></p>
<ul>
<li>bunyan has an executable?! <a href="https://github.com/trentm/node-bunyan#cli-usage">What</a> does it do?</li>
</ul>
<p><a href="https://github.com/lodash/lodash/blob/master/package.json">lodash</a></p>
<ul>
<li><code class="language-plaintext highlighter-rouge">lodash.js</code> is the <code class="language-plaintext highlighter-rouge">main</code> entry point…</li>
<li>but lodash allows you to require only what you need, maybe just one function: <code class="language-plaintext highlighter-rouge">require("lodash/compact")</code></li>
<li>check the <a href="https://github.com/lodash/lodash">base</a> of the project to see ALL the code…</li>
</ul>
<p><a href="https://github.com/zeit/ms/blob/master/package.json">ms</a></p>
<ul>
<li>No <code class="language-plaintext highlighter-rouge">dependencies</code>! It’s a cute little library.</li>
</ul>
<p><a href="https://github.com/moment/moment/blob/develop/package.json">moment</a></p>
<ul>
<li>Also no <code class="language-plaintext highlighter-rouge">dependencies</code>… but a ton of code.</li>
<li>There’s a bunch of non-standard sections in the package.json</li>
</ul>
8 Years of Reading, Visualized2018-03-02T00:00:00+00:00https://blog.jpalardy.com/posts/8-years-of-reading-visualized<h2 id="the-data">The Data</h2>
<p><img style="float: right; margin-left: 10px" src="/assets/8-years-reading/data-sample.png" alt="Sample data" /></p>
<p>I’ve been <a href="https://bookpiles.ca/jonathan/books?pile=done">keeping track</a> of
what I read for years. I occasionally pull the data out and try to see
what insights I can extract out of it. The <a href="/assets/8-years-reading/books.csv">data</a>
consists of the date I finished reading a book. There’s one date per
book, and some dates have more than one book.</p>
<p>I’ve tried a bunch of <a href="https://blog.jpalardy.com/posts/reject-summary-statistics/">summary statistics</a>,
various counts and averages, but it wasn’t very enlightening. I
wondered what it would look like on different plots.</p>
<h2 id="the-plots">The Plots</h2>
<p>I started with the <em>easiest</em> thing, each event as a dot:</p>
<p><a href="/assets/8-years-reading/dots.png"><img src="/assets/8-years-reading/dots.png" alt="events as dots" /></a></p>
<p>At this level, it’s clear that some periods of time have more/less events
happening. I’m also using transparency to reveal overlapping dots. But dots on
a timeline makes it hard to interpret progress, let’s make it cumulative:</p>
<p><a href="/assets/8-years-reading/geom_step.png"><img src="/assets/8-years-reading/geom_step.png" alt="events as geom_step" /></a></p>
<p>There are a few problems with this. The y-axis doesn’t have a lot of distinct values
(there are only so many books I can read in a year), which means a fair bit of
overlap between years. Book progression might be more linear, if
I’m 60% of the way through a book, that’s 60% of the way to +1. A “line” graph
might be fairer:</p>
<p><a href="/assets/8-years-reading/geom_line.png"><img src="/assets/8-years-reading/geom_line.png" alt="events as geom_line" /></a></p>
<p>This still suffers from overcrowding in the early months … it all starts
at zero and diverges from there. I decided to separate each year:</p>
<p><a href="/assets/8-years-reading/facet.png"><img src="/assets/8-years-reading/facet.png" alt="events as geom_line, faceted" /></a></p>
<p>It makes each trend clearer, but it also makes it harder to compare between the years.</p>
<h2 id="discussion">Discussion</h2>
<p>A “book read” isn’t a great unit of measurement in the first place. What is a
book? How many pages does it have? Books come in different sizes with different
font sizes and amount of whitespace.</p>
<p>Data-wise, there are a few observations:</p>
<ul>
<li>2017 isn’t like the other years – I’m not sure what I did differently</li>
<li>the end of each year seems to be <em>flatter</em> than the usual trend</li>
<li>apart from 2017, other years look similar, especially given the above caveat about the choice of unit</li>
</ul>
<p>To make date events share the same x-axis, I had to normalize the years so that
all dates would share the same year. To ensure that all dates were available, I
picked 2016 because it was a leap year.</p>
<p>I also had to create “zero events” on January 1st and December 31st so that
charts would be anchored. Without these anchors, the lines would start on the
year’s first event and stop on the year’s last event.</p>
<p>All the details are in the <a href="/assets/8-years-reading/report.html">R Markdown report</a>.</p>
Mocha and Istanbul in 5 minutes2018-02-19T00:00:00+00:00https://blog.jpalardy.com/posts/mocha-and-istanbul-in-5-minute<h2 id="who-is-this-for">Who is this for?</h2>
<p>This post is for you if:</p>
<ul>
<li>you want to get started with tests and test coverage but<br />
feel overwhelmed by the details</li>
</ul>
<p>or</p>
<ul>
<li>you already know all this… but will appreciate a “checklist”<br />
next time you need to do it all again</li>
</ul>
<p>I was in the second situation recently and decided to write everything down while it’s still fresh.</p>
<h2 id="context">Context</h2>
<p>You have a Node.js project, but no tests (yet), and you need <em>just enough</em> to get started.</p>
<ul>
<li><a href="https://mochajs.org/">Mocha</a> is test framework.</li>
<li><a href="https://istanbul.js.org/">Istanbul</a> is a test coverage tool.</li>
<li><code class="language-plaintext highlighter-rouge">nyc</code> is the command-line client for Istanbul.</li>
</ul>
<p>I put all the files you’ll need on <a href="https://github.com/jpalardy/mocha-istanbul-in-5m">github</a>.</p>
<p>This is not a tutorial on how to write tests or how to use Mocha/Istanbul … but there are some pointers at the end of the post.</p>
<h2 id="step-by-step">Step by step</h2>
<h3 id="1-install-your-dependencies">1. Install your dependencies</h3>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>npm <span class="nb">install</span> <span class="nt">--save-dev</span> mocha nyc</code></pre></figure>
<h3 id="2-create-the-test-directory">2. Create the test directory</h3>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">mkdir test</span>
<span class="nv">$ </span><span class="c"># paste the following code in test/example.js</span></code></pre></figure>
<p><a href="https://github.com/jpalardy/mocha-istanbul-in-5m/blob/master/test/example.js">test/example.js</a></p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="kd">const</span> <span class="nx">assert</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">assert</span><span class="dl">'</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">pluralize</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">../lib/example</span><span class="dl">'</span><span class="p">).</span><span class="nx">pluralize</span><span class="p">;</span>
<span class="nx">describe</span><span class="p">(</span><span class="dl">'</span><span class="s1">example</span><span class="dl">'</span><span class="p">,</span> <span class="kd">function</span> <span class="p">()</span> <span class="p">{</span>
<span class="nx">describe</span><span class="p">(</span><span class="dl">'</span><span class="s1">pluralize</span><span class="dl">'</span><span class="p">,</span> <span class="kd">function</span> <span class="p">()</span> <span class="p">{</span>
<span class="nx">it</span><span class="p">(</span><span class="dl">'</span><span class="s1">keeps singular when count is 1</span><span class="dl">'</span><span class="p">,</span> <span class="kd">function</span> <span class="p">()</span> <span class="p">{</span>
<span class="nx">assert</span><span class="p">.</span><span class="nx">strictEqual</span><span class="p">(</span><span class="nx">pluralize</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="dl">'</span><span class="s1">cat</span><span class="dl">'</span><span class="p">),</span> <span class="dl">'</span><span class="s1">1 cat</span><span class="dl">'</span><span class="p">);</span>
<span class="p">});</span>
<span class="p">});</span>
<span class="p">});</span></code></pre></figure>
<h3 id="3-create-a-file-to-test">3. Create a file to test</h3>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">mkdir </span>lib
<span class="nv">$ </span><span class="c"># paste the following code in lib/example.js</span></code></pre></figure>
<p><a href="https://github.com/jpalardy/mocha-istanbul-in-5m/blob/master/lib/example.js">lib/example.js</a></p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="nx">exports</span><span class="p">.</span><span class="nx">pluralize</span> <span class="o">=</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">count</span><span class="p">,</span> <span class="nx">singular</span><span class="p">,</span> <span class="nx">plural</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">count</span> <span class="o">===</span> <span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="s2">`</span><span class="p">${</span><span class="nx">count</span><span class="p">}</span><span class="s2"> </span><span class="p">${</span><span class="nx">singular</span><span class="p">}</span><span class="s2">`</span><span class="p">;</span>
<span class="p">}</span>
<span class="nx">plural</span> <span class="o">=</span> <span class="nx">plural</span> <span class="o">||</span> <span class="s2">`</span><span class="p">${</span><span class="nx">singular</span><span class="p">}</span><span class="s2">s`</span><span class="p">;</span>
<span class="k">return</span> <span class="s2">`</span><span class="p">${</span><span class="nx">count</span><span class="p">}</span><span class="s2"> </span><span class="p">${</span><span class="nx">plural</span><span class="p">}</span><span class="s2">`</span><span class="p">;</span>
<span class="p">};</span></code></pre></figure>
<h3 id="4-edit-scripts-in-packagejson">4. Edit scripts in package.json</h3>
<p>Open your <code class="language-plaintext highlighter-rouge">package.json</code> and make sure the <code class="language-plaintext highlighter-rouge">scripts</code> section looks like:</p>
<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="nl">"scripts"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"test"</span><span class="p">:</span><span class="w"> </span><span class="s2">"mocha --reporter dot"</span><span class="p">,</span><span class="w">
</span><span class="nl">"coverage"</span><span class="p">:</span><span class="w"> </span><span class="s2">"nyc --reporter html --reporter text npm test"</span><span class="w">
</span><span class="p">}</span></code></pre></figure>
<h3 id="5-run-your-tests">5. Run your tests</h3>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>npm <span class="nb">test</span></code></pre></figure>
<h3 id="6-run-your-test-coverage">6. Run your test coverage</h3>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>npm run coverage
<span class="nv">$ </span><span class="c"># open coverage/index.html</span></code></pre></figure>
<h2 id="overtime">Overtime</h2>
<p>We have tests, we can run them, and we have test coverage: done!</p>
<h3 id="comments">Comments:</h3>
<p>Test coverage is poor, on purpose; to show you how useful Istanbul can be:</p>
<p><img src="/assets/mocha-and-istanbul-in-5m/coverage-example-js.png" alt="code coverage for example.js" /></p>
<p>In this case, you would need more tests:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="c1">// covers `else`</span>
<span class="nx">it</span><span class="p">(</span><span class="dl">'</span><span class="s1">uses explicit plural when given</span><span class="dl">'</span><span class="p">,</span> <span class="kd">function</span> <span class="p">()</span> <span class="p">{</span>
<span class="nx">assert</span><span class="p">.</span><span class="nx">strictEqual</span><span class="p">(</span><span class="nx">pluralize</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="dl">'</span><span class="s1">mouse</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">mice</span><span class="dl">'</span><span class="p">),</span> <span class="dl">'</span><span class="s1">3 mice</span><span class="dl">'</span><span class="p">);</span>
<span class="p">});</span>
<span class="c1">// covers line 6, after `||`</span>
<span class="nx">it</span><span class="p">(</span><span class="dl">'</span><span class="s1">uses implicit plural otherwise</span><span class="dl">'</span><span class="p">,</span> <span class="kd">function</span> <span class="p">()</span> <span class="p">{</span>
<span class="nx">assert</span><span class="p">.</span><span class="nx">strictEqual</span><span class="p">(</span><span class="nx">pluralize</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="dl">'</span><span class="s1">cat</span><span class="dl">'</span><span class="p">),</span> <span class="dl">'</span><span class="s1">3 cats</span><span class="dl">'</span><span class="p">);</span>
<span class="p">});</span></code></pre></figure>
<p>What’s <code class="language-plaintext highlighter-rouge">describe</code> and <code class="language-plaintext highlighter-rouge">it</code>? Refer to the excellent <a href="https://mochajs.org/">mocha documentation</a>.</p>
<p>Mocha, by default, runs tests from <code class="language-plaintext highlighter-rouge">test/*.js</code>, that’s why we didn’t need to specify test files or directories.</p>
<p>I kept things simple and used Node.js’s <a href="https://nodejs.org/api/assert.html">assert</a> library.</p>
<p>When things get complicated, you’ll probably need <a href="https://sinonjs.org/">sinon</a>. Here’s a good <a href="https://semaphoreci.com/community/tutorials/best-practices-for-spies-stubs-and-mocks-in-sinon-js">tutorial</a>.</p>
ISO 8601 and Date Arithmetic on the Command-Line2018-01-15T00:00:00+00:00https://blog.jpalardy.com/posts/iso8601-and-date-arithmetic-on-the-command-line<h2 id="why">Why?</h2>
<ul>
<li>typing full correct dates is painful</li>
<li>date arithmetic is useful – especially in bash scripts</li>
</ul>
<h2 id="iso-8601-format">ISO 8601 Format</h2>
<p>When you use <code class="language-plaintext highlighter-rouge">date</code>, you might not realize its full potential:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">date
</span>Mon Jan 15 18:43:31 PST 2018</code></pre></figure>
<p>it’s better if you provide a format:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">date</span> +%Y-%m-%d <span class="c"># iso8601, of course</span>
2018-01-15
<span class="nv">$ </span><span class="nb">date</span> +%F <span class="c"># same: iso8601 has its own short format</span>
2018-01-15</code></pre></figure>
<p>Check <code class="language-plaintext highlighter-rouge">man date</code> for all the formats you need. But why wouldn’t you use
<a href="https://en.wikipedia.org/wiki/ISO_8601">ISO 8601</a>?</p>
<h2 id="specify-an-absolute-date">Specify an Absolute Date</h2>
<p>You can specify a date to format with <code class="language-plaintext highlighter-rouge">-d</code>, it recognizes a bunch of formats:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">date</span> +%F <span class="nt">-d</span> 2018-02-25 <span class="c"># silly: iso8601 to iso8601...</span>
2018-02-25
<span class="nv">$ </span><span class="nb">date</span> +%F <span class="nt">-d</span> <span class="s2">"February 25 2018"</span> <span class="c"># full date</span>
2018-02-25
<span class="nv">$ </span><span class="nb">date</span> +%F <span class="nt">-d</span> <span class="s2">"Feb 25 2018"</span> <span class="c"># short month</span>
2018-02-25
<span class="nv">$ </span><span class="nb">date</span> +%F <span class="nt">-d</span> <span class="s2">"Feb 25"</span> <span class="c"># missing year, current year implied</span>
2018-02-25</code></pre></figure>
<h2 id="specify-a-relative-date">Specify a Relative Date</h2>
<p>The <code class="language-plaintext highlighter-rouge">-d</code> flag also accepts relative dates:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">date</span> +%F <span class="nt">-d</span> <span class="s2">"next Saturday"</span>
2018-01-20
<span class="nv">$ </span><span class="nb">date</span> +%F <span class="nt">-d</span> <span class="s2">"next Sat"</span> <span class="c"># short day name</span>
2018-01-20
<span class="nv">$ </span><span class="nb">date</span> +%F <span class="nt">-d</span> <span class="s2">"3 sat"</span> <span class="c"># 3 Saturdays from today</span>
2018-02-03
<span class="nv">$ </span><span class="nb">date</span> +%F <span class="nt">-d</span> <span class="s2">"2 months"</span>
2018-03-15
<span class="nv">$ </span><span class="nb">date</span> +%F <span class="nt">-d</span> <span class="s2">"3 weeks sat"</span> <span class="c"># Saturday after 3 weeks</span>
2018-02-10
<span class="nv">$ </span><span class="nb">date</span> +%F <span class="nt">-d</span> <span class="s2">"last month"</span>
2017-12-15
<span class="nv">$ </span><span class="nb">date</span> +%F <span class="nt">-d</span> <span class="s2">"-5 days"</span> <span class="c"># also: "5 days ago"</span>
2018-01-10</code></pre></figure>
<p>The documentation says:</p>
<ul>
<li>“last” means -1</li>
<li>“this” means 0</li>
<li>“next” means 1</li>
<li>“ago”, as a suffix, means negative</li>
<li>“now” and “today” mean 0</li>
</ul>
<p>There is support for units (in singular and plural):</p>
<ul>
<li>year</li>
<li>month</li>
<li>week</li>
<li>day</li>
<li>hour</li>
<li>min(ute)</li>
<li>sec(ond)</li>
</ul>
<h2 id="date-arithmetic">Date Arithmetic!</h2>
<p>Yes, you can combine both absolute and relative dates:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">date</span> +%F <span class="nt">-d</span> <span class="s2">"today + 3 days"</span>
2018-01-18
<span class="nv">$ </span><span class="nb">date</span> +%F <span class="nt">-d</span> <span class="s2">"+ 3 days"</span> <span class="c"># shorter</span>
2018-01-18
<span class="nv">$ </span><span class="nb">date</span> +%F <span class="nt">-d</span> <span class="s2">"3 days"</span> <span class="c"># shorter still</span>
2018-01-18
<span class="nv">$ </span><span class="nb">date</span> +%F <span class="nt">-d</span> <span class="s2">"sep 1 - 3 weeks"</span>
2018-08-11
<span class="nv">$ </span><span class="nb">date</span> +%F <span class="nt">-d</span> <span class="s2">"5 days - 3 weeks"</span>
2017-12-30</code></pre></figure>
<p>Some of these are useful, others not so much… it depends what you need.</p>
<p>Think about your edge cases carefully:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">date</span> +%F <span class="nt">-d</span> <span class="s2">"jan 27 + 1 month"</span>
2018-02-27
<span class="nv">$ </span><span class="nb">date</span> +%F <span class="nt">-d</span> <span class="s2">"jan 28 + 1 month"</span>
2018-02-28
<span class="nv">$ </span><span class="nb">date</span> +%F <span class="nt">-d</span> <span class="s2">"jan 29 + 1 month"</span> <span class="c"># careful!</span>
2018-03-01
<span class="nv">$ </span><span class="nb">date</span> +%F <span class="nt">-d</span> <span class="s2">"feb 1 + 1 month"</span> <span class="c"># same...</span>
2018-03-01
<span class="nv">$ </span><span class="nb">date</span> +%F <span class="nt">-d</span> <span class="s2">"jan 30 + 1 month"</span> <span class="c"># careful!</span>
2018-03-02
<span class="nv">$ </span><span class="nb">date</span> +%F <span class="nt">-d</span> <span class="s2">"jan 31 + 1 month"</span> <span class="c"># careful!</span>
2018-03-03</code></pre></figure>
<p>This <a href="https://www.gnu.org/software/coreutils/manual/html_node/Relative-items-in-date-strings.html#Relative-items-in-date-strings">page</a> (bottom) discusses
this gotcha and possible workarounds.</p>
<p>The subpages of <a href="https://www.gnu.org/software/coreutils/manual/html_node/Date-input-formats.html">Date input formats</a>
contain all the details from above and more… For example, I didn’t even try to discuss times, timezones and all their
terrifying complexity.</p>
<h2 id="my-iso8601-helper">My <code class="language-plaintext highlighter-rouge">iso8601</code> Helper</h2>
<p>Because I use ISO 8601 dates in a bunch of command-line contexts, I wrote a <a href="https://github.com/jpalardy/dotfiles/blob/a25af1bd7a06b0a2bfda978ffa48afcd5d66a9e4/bin/iso8601">helper</a>.
Dump the script (call it <code class="language-plaintext highlighter-rouge">iso8601</code>) in your PATH:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c">#!/bin/bash</span>
<span class="nb">date</span> +%F <span class="nt">-d</span> <span class="s2">"</span><span class="nv">$*</span><span class="s2">"</span></code></pre></figure>
<p>The <code class="language-plaintext highlighter-rouge">$*</code> sequence means you don’t need to (but still can) quote your inputs:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>iso8601
2018-01-15
<span class="nv">$ </span>iso8601 3 days
2018-01-18
<span class="nv">$ </span>iso8601 mar 15
2018-03-15
<span class="nv">$ </span>iso8601 jun 6 + 3 weeks
2018-06-27
<span class="nv">$ </span>iso8601 <span class="s2">"jun 6 + 3 weeks"</span> <span class="c"># use quotes, if you want...</span>
2018-06-27</code></pre></figure>
<p>I use this script mainly from inside Vim. I type <code class="language-plaintext highlighter-rouge">!!iso8601 <bla bla></code> on a blank line to insert a date.</p>
The Best Books I Read in 20172017-12-23T00:00:00+00:00https://blog.jpalardy.com/posts/best-books-i-read-2017<p>I have read many books in 2017, but it wasn’t all good. Some books, however,
deserve to be acknowledged. I broke down my recommendations by categories:
<a href="#technical">technical</a>, <a href="#non-fiction">non-fiction</a> and <a href="#fiction">fiction</a>.
There is no particular order.</p>
<p>I have similar posts for <a href="https://blog.jpalardy.com/posts/best-books-i-read-2015/">2015</a> and <a href="https://blog.jpalardy.com/posts/best-books-i-read-2016/">2016</a>.</p>
<hr />
<h2 id="technical">Technical</h2>
<h3 id="how-to-measure-anything">How to Measure Anything</h3>
<p><a href="https://www.amazon.com/dp/1118539273/"><img class="book-cover" src="/assets/best-books-2017/1118539273.jpg" alt="How to Measure Anything" /></a></p>
<p>“Anything can be measured.” is how the book begins, the rest of the book
explains how to go about it.</p>
<p>When people say “it can’t be measured”, they often mean:</p>
<ul>
<li>we don’t know how to measure it…</li>
<li>we think it would cost too much to measure…</li>
<li>we can’t pinpoint its <em>exact</em> value…<br />
(<em>in other words: there’s always uncertainty</em>)</li>
</ul>
<p>The first part of the book tries to convince you that measurements are
worthwhile. There is a way to measure what you need, it’s cheaper than you
expect, and it gets you “close enough” to the truth.</p>
<p>The book also covers “what’s a measurement?”, what you can measure <em>versus</em> what you should
measure, how to reduce uncertainty, the cost/value of more information, a
long list of cognitive biases to consider, etc.</p>
<p>If you’re familiar with <a href="https://en.wikipedia.org/wiki/Fermi_problem">Fermi problems</a>, this book
will cover and expand the technique:</p>
<ul>
<li>estimate something: give it a range (rather than <a href="https://blog.jpalardy.com/posts/reject-summary-statistics/">the alternative</a>)</li>
<li>adjust the range so that you’re 90% confident it contains the real value – the book gives you techniques to do that</li>
<li>assume a specific distribution (linear, normal, other?)</li>
<li>each estimate chains into the next</li>
</ul>
<p>If you simulate the above, you get the <a href="https://en.wikipedia.org/wiki/Monte_Carlo_method">Monte Carlo method</a>.
It doesn’t have to be complicated; you can run this in a spreadsheet. The <a href="https://www.howtomeasureanything.com/">book’s website</a> provides
spreadsheet templates to get you started.</p>
<p>Don’t accept “it can’t be measured” from people who haven’t read this book.</p>
<ul>
<li><a href="https://www.amazon.com/dp/1118539273/">amazon</a></li>
<li><a href="https://www.howtomeasureanything.com/how-to-measure-anything-third-edition/">website</a></li>
</ul>
<h3 id="high-performance-browser-networking">High Performance Browser Networking</h3>
<p><a href="https://www.amazon.com/dp/1449344763/"><img class="book-cover" src="/assets/best-books-2017/1449344763.jpg" alt="High Performance Browser Networking" /></a></p>
<p>It’s easy to take for granted all the technologies required to make web
development possible. We learn what we need, pick up more over time, but we
rarely resort to the depths provided by this book.</p>
<p>The book is divided into parts:</p>
<ul>
<li>low level: latency, bandwidth, TCP, UDP, TLS</li>
<li>wireless networks: WiFi and mobile networks</li>
<li>HTTP: 1.x, 2.x and optimizations</li>
<li>APIs and protocols: XHR, SSE, WebSocket, WebRTC</li>
</ul>
<p>It covers <em>a lot</em>. Most of it was a review, but I was grateful to
get it all in one place. The author’s style is clear; it was a joy to read.
I think this book has a place in a modern technical bookshelf.</p>
<ul>
<li><a href="https://www.amazon.com/dp/1449344763/">amazon</a></li>
<li><a href="https://hpbn.co/">website</a></li>
</ul>
<h3 id="designing-data-intensive-applications">Designing Data-Intensive Applications</h3>
<p><a href="https://www.amazon.com/dp/1449373321/"><img class="book-cover" src="/assets/best-books-2017/1449373321.jpg" alt="Designing Data-Intensive Applications" /></a></p>
<p>There was a lot of buzz when this book came out. It’s all justified.</p>
<p>This is a survey of theories, tools, and techniques. It takes so many topics and
explains them so well – it would take months of research to get the content
from other sources. Case in point: there’s an extensive bibliography for <em>each chapter</em>!</p>
<p>Whenever I’ve had to talk about system design since I’ve read it, I was
grateful to have read this book. There is no right answer, but you develop a
sense of the trade-offs involved in different solutions. You get a better idea
of what questions to ask and what to look for.</p>
<p>It also gave me an appreciation for all the ways that things can go wrong
with distributed systems. It’s a wonder that anything works…</p>
<ul>
<li><a href="https://www.amazon.com/dp/1449373321/">amazon</a></li>
<li><a href="https://dataintensive.net/">website</a></li>
</ul>
<h3 id="r-for-data-science">R for Data Science</h3>
<p><a href="https://www.amazon.com/dp/1491910399/"><img class="book-cover" src="/assets/best-books-2017/1491910399.jpg" alt="R for Data Science" /></a></p>
<p>This is the book I wish I had when I started out learning R.</p>
<p>Back then, there were <em>few</em> books and <em>fewer</em> good books. Nowadays, the
situation has improved significantly. However…</p>
<p>There is “old R” and “new R” out there. Old R is what we’ve had
since the beginning, usually denoted with <code class="language-plaintext highlighter-rouge">*apply</code> functions and the base
graphics package. There’s nothing wrong with old R … but it’s neither
easy to learn nor easy to use. When authors talk about it, the explanations are full of apologies…</p>
<p>Meanwhile, the <a href="https://www.tidyverse.org/">tidyverse</a> is what most people
are using now. It’s a modern twist on R coming from the open-source community.
The tidyverse is just an opinionated list of packages including:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">ggplot2</code> – the “killer app” of R, in my opinion</li>
<li><code class="language-plaintext highlighter-rouge">dplyr</code> – select/filter/order functions for data frames / tibbles</li>
<li><code class="language-plaintext highlighter-rouge">tibble</code> – a friendly replacement for data frames</li>
<li><code class="language-plaintext highlighter-rouge">magrittr</code> – like <code class="language-plaintext highlighter-rouge">|</code> on the command-line</li>
<li><a href="https://www.tidyverse.org/packages/">and more…</a></li>
</ul>
<p>Instead of choosing and loading these packages separately, loading <code class="language-plaintext highlighter-rouge">tidyverse</code>
will load them all for you. If you tell me you use the tidyverse, I know
exactly what packages you use.</p>
<p>On top of walking you through the tidyverse, you also get plenty of advices on
how to organize and analyze data for your “data science” projects – but really
any project with data.</p>
<ul>
<li><a href="https://www.amazon.com/dp/1491910399/">amazon</a></li>
<li><a href="https://r4ds.had.co.nz/">website</a></li>
</ul>
<hr />
<h2 id="non-fiction">Non-Fiction</h2>
<h3 id="the-dictators-handbook">The Dictator’s Handbook</h3>
<p><a href="https://www.amazon.com/dp/1610391845/"><img class="book-cover" src="/assets/best-books-2017/1610391845.jpg" alt="The Dictator's Handbook" /></a></p>
<p>A powerful, if depressing, book.</p>
<p>This was a <a href="https://en.wikipedia.org/wiki/CGP_Grey">CGP grey</a> recommendation at the end of
his excellent video series called: <a href="https://www.youtube.com/watch?v=rStL7niR7gs&list=PLqs5ohhass_QPOfhvhIzxas3Vr9k31Vaz">Rules for Rulers</a>.
Good news: the videos alone will give you 90% of the ideas from the book.</p>
<p>If you accept the premise of the book, you get a framework to understand political behaviors:</p>
<ul>
<li>why revolutionaries end up betraying each other</li>
<li>why corruption makes sense</li>
<li>why foreign aid is <em>complicated</em></li>
</ul>
<p>This book provides tremendous explaining power.</p>
<p>Personally, I went from “politics doesn’t make sense” to understanding why
politicians do what they do.</p>
<p>Not everybody agrees with the political theory of the book. But I think theories
can be judged based on <em>how much</em> they can explain simply.</p>
<ul>
<li><a href="https://www.amazon.com/dp/1610391845/">amazon</a></li>
<li><a href="https://en.wikipedia.org/wiki/The_Dictator%27s_Handbook">wikipedia</a></li>
<li><a href="https://www.hachettebookgroup.com/titles/bruce-bueno-de-mesquita/the-dictators-handbook/9781610390453/">website</a></li>
</ul>
<h3 id="weapons-of-math-destruction">Weapons of Math Destruction</h3>
<p><a href="https://www.amazon.com/dp/0553418815/"><img class="book-cover" src="/assets/best-books-2017/0553418815.jpg" alt="Weapons of Math Destruction" /></a></p>
<p><em>Weapons of Math Destruction</em> is a sobering critique of the (careless) use of algorithms.</p>
<p>Earlier this year, I also read <a href="https://www.amazon.com/dp/0465065708/">The Master Algorithm</a>
which reads more like a love story. Short version: The Master Algorithm didn’t
make this list. I thought it was techno-optimism: a celebration of current and
coming achievements with a don’t-worry-about-it perspective about possible
shortcomings.</p>
<p>The book documents instances of systematic discrimination coming from “black
box” algorithms. We know that <a href="https://en.wikipedia.org/wiki/List_of_cognitive_biases">humans are biased</a>
… but <a href="https://www.youtube.com/watch?v=_2u_eHHzRto">algorithms are biased too</a>.
Trust in algorithms we don’t understand, applied wholesale to a variety of
problems, without recourse when things go wrong, provide a dystopian view of
things to come.</p>
<p>Whether statisticians or programmers, it is up to us to push back on misguided
applications of algorithms, big data, and machine learning. If we don’t, the
coming backlash might be justified.</p>
<ul>
<li><a href="https://www.amazon.com/dp/0553418815/">amazon</a></li>
<li><a href="https://en.wikipedia.org/wiki/Weapons_of_Math_Destruction">wikipedia</a></li>
<li><a href="https://weaponsofmathdestructionbook.com/">website</a></li>
</ul>
<hr />
<h2 id="fiction">Fiction</h2>
<h3 id="we-are-legion">We Are Legion</h3>
<p><em>Series: 3 books, complete.</em></p>
<p><a href="https://www.amazon.com/dp/B01L082SCI/"><img class="book-cover" src="/assets/best-books-2017/B01L082SCI.jpg" alt="We Are Legion" /></a></p>
<p>“We are legion. We are Bob”, really? I didn’t even judge the book by its cover;
the uninspiring title was enough for me. But I read the glowing reviews and decided to give it a try.</p>
<p>I loved the beginning: it reminded me of <a href="https://www.amazon.com/dp/0765348276/">Old Man’s War</a>;
you discover the story, and the rules, at the same time the main character does.</p>
<p>I thought the second and third books weren’t as good, maybe even episodic. There
was too much to keep track of. And the ending wasn’t satisfying… But, despite my
complaints, I couldn’t put it down until the end.</p>
<ul>
<li><a href="https://www.amazon.com/dp/B01L082SCI/">amazon</a></li>
<li><a href="https://dennisetaylor.org/legion/">website</a></li>
</ul>
<h3 id="expeditionary-force">Expeditionary Force</h3>
<p><em>Series: 5 books, ongoing.</em></p>
<p><a href="https://www.amazon.com/dp/B01MQR08XA/"><img class="book-cover" src="/assets/best-books-2017/B01MQR08XA.jpg" alt="Expeditionary Force" /></a></p>
<p>This falls under the category of “military sci-fi”, a place I hadn’t spent a
lot of time in and wasn’t planning on visiting soon. Once again, the reviews pulled me in.</p>
<p>This isn’t a series that takes itself seriously. Is
military-goofy sci-fi a genre? Earth is in danger, aliens are involved, our
enemies aren’t who we think they are, etc. How can this be a comedy?!
I don’t want to spoil anything, pick it up and get back to me.</p>
<p>Is it a masterpiece? No. Will I buy the next book when it comes out? Yes,
definitely.</p>
<ul>
<li><a href="https://www.amazon.com/dp/B01MQR08XA/">amazon</a></li>
<li><a href="https://www.craigalanson.com/books/">website</a></li>
</ul>
<h3 id="fear-the-sky">Fear the Sky</h3>
<p><em>Series: 3 books, complete.</em></p>
<p><a href="https://www.amazon.com/dp/B00S8FPDQA/"><img class="book-cover" src="/assets/best-books-2017/B00S8FPDQA.jpg" alt="Fear the Sky" /></a></p>
<p>In contrast to the books above, the <em>Fear Saga</em> series is a darker
type of sci-fi. It has <em>some</em> aspects of the <a href="https://www.amazon.com/dp/0765377063/">Three-Body Problem</a>
which I <a href="https://blog.jpalardy.com/posts/best-books-i-read-2015/#the-three-body-problem">reviewed previously</a>.</p>
<p>Aliens are coming and … it’s complicated. Let me spare you the spoilers.
The first book is a thriller; there is so much at stake, so much to fear, and so many opportunities to fail.</p>
<p>The second and third books didn’t have quite the tension of the first, but they
were enjoyable nonetheless. I was able to digest the <a href="https://en.wikipedia.org/wiki/Deus_ex_machina">deus ex machina</a>
ending because the reviews had warned me in advance.</p>
<p>Not perfect, but pretty good.</p>
<ul>
<li><a href="https://www.amazon.com/dp/B00S8FPDQA/">amazon</a></li>
<li><a href="http://www.thefearsaga.com/">website</a></li>
</ul>
<hr />
<h2 id="closing-thoughts">Closing Thoughts</h2>
<p>If you have liked this blog post, please write your own. I want to read your book
reviews and recommendations.</p>
Throwaway MySQL Servers with Docker2017-11-02T00:00:00+00:00https://blog.jpalardy.com/posts/throwaway-mysql-servers-with-docker<h2 id="mysql-without-the-mess">MySQL without the Mess</h2>
<p>Sometimes, I need MySQL<sup><a href="#fn1">1</a></sup> but:</p>
<ul>
<li>I don’t want to install MySQL on macOS (homebrew or otherwise).</li>
<li>I don’t want to spin up a virtual machine (locally or in the cloud).</li>
<li>I don’t want to install it, administer it, upgrade it, or clean it up.</li>
</ul>
<p>With Docker, I can pull down the <a href="https://hub.docker.com/_/mysql/">official MySQL image</a>
and run <em>both</em> server and client from it.</p>
<h2 id="what-to-install">What to Install</h2>
<p>Get <a href="https://www.docker.com/docker-mac">Docker for Mac</a>, or equivalent. Once installed,
you get the menu bar whale icon and the <code class="language-plaintext highlighter-rouge">docker</code> command will be available on
the command-line.</p>
<p>Pull down the mysql image:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>docker pull mysql</code></pre></figure>
<p>or, if you don’t want latest, check the <a href="https://hub.docker.com/_/mysql/">available versions</a>.</p>
<p>This isn’t a Docker tutorial; you know <a href="https://www.google.com/search?q=docker+tutorial">where</a> to find one if needed :-)</p>
<h2 id="how-to-run-the-server">How to Run the Server</h2>
<p>Run this command in its own terminal. This is where you’ll control the
MySQL server and see its logs.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>docker run <span class="nt">-e</span> <span class="nv">MYSQL_ALLOW_EMPTY_PASSWORD</span><span class="o">=</span><span class="nb">yes</span> <span class="nt">-p</span> 3306:3306 <span class="nt">--rm</span> mysql
<span class="c"># docker run -- runs an image</span>
<span class="c"># -e MYSQL_ALLOW_EMPTY_PASSWORD=yes -- sets an environment variable</span>
<span class="c"># -p 3306:3306 -- exposes container port 3306 as local port 3306</span>
<span class="c"># --rm -- cleans up container after it exits</span>
<span class="c"># mysql -- the name of the image to run</span></code></pre></figure>
<p>The use of <code class="language-plaintext highlighter-rouge">-e MYSQL_ALLOW_EMPTY_PASSWORD=yes</code> might raise an eyebrow, but
we’re not deploying to production. This is <em>meant</em> to be run locally and to be
trashed. Convenience is often the opposite of security.</p>
<p>Note: <code class="language-plaintext highlighter-rouge">ctrl-c</code> doesn’t kill the server – but <code class="language-plaintext highlighter-rouge">ctrl-\</code> does.</p>
<h2 id="how-to-run-the-client">How to Run the Client</h2>
<p>Run this command in another terminal. This is where you’ll run the client and
connect to the MySQL server. This is where you’ll type your commands.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>docker run <span class="nt">-it</span> <span class="nt">-v</span> <span class="nv">$PWD</span>:/data <span class="nt">-w</span> /data <span class="nt">--rm</span> mysql mysql <span class="nt">-h</span> <span class="nv">$LOCAL_IP</span> <span class="nt">-u</span> root <span class="nt">-p</span>
<span class="c"># docker run -- runs an image</span>
<span class="c"># -i -- keeps STDIN open</span>
<span class="c"># -t -- allocates a pseudo-TTY</span>
<span class="c"># -v $PWD:/data -- binds $PWD to /data inside the container</span>
<span class="c"># -w /data -- changes the PWD of the container to /data</span>
<span class="c"># --rm -- cleans up container after it exits</span>
<span class="c"># mysql -- the name of the image to run</span>
<span class="c"># mysql -h $LOCAL_IP -u root -p -- the command to run inside the container</span></code></pre></figure>
<p>We configured the server so the root account doesn’t have a password; press ENTER when prompted.</p>
<p>Don’t use <code class="language-plaintext highlighter-rouge">127.0.0.1</code> for <code class="language-plaintext highlighter-rouge">$LOCAL_IP</code>, it only works with your real IP address.</p>
<h2 id="this-mysql-server-is-empty-now-what">This MySQL Server is Empty… Now What?</h2>
<p>This is by design: you’re spinning up a brand new MySQL server with nothing in it.</p>
<p>There are two ways to initialize the server with your data:</p>
<ul>
<li>loading data from the client</li>
<li>starting the server with the data</li>
</ul>
<h2 id="loading-data-from-the-client">Loading Data From the Client</h2>
<p>The way we configured the client, all your $PWD files are available from the
client container. In the MySQL client, source the relevant SQL dump to bootstrap the server.</p>
<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="o">></span> <span class="k">source</span> <span class="n">dump</span><span class="p">.</span><span class="k">sql</span></code></pre></figure>
<h2 id="starting-the-server-with-the-data">Starting the Server with the Data</h2>
<p>The <a href="https://hub.docker.com/_/mysql/">documentation</a> says:</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">[the container] will execute files with extensions .sh, .sql and .sql.gz
that are found in /docker-entrypoint-initdb.d. Files will be executed in
alphabetical order.</code></pre></figure>
<p>Yes, it’s a bit magical but, if you put .sql (etc) files in <code class="language-plaintext highlighter-rouge">/docker-entrypoint-initdb.d</code>,
they will be automatically loaded into the server.</p>
<p>In practice, add <code class="language-plaintext highlighter-rouge">-v</code> to the server command from above:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>docker run <span class="nt">-e</span> <span class="nv">MYSQL_ALLOW_EMPTY_PASSWORD</span><span class="o">=</span><span class="nb">yes</span> <span class="nt">-p</span> 3306:3306 <span class="se">\</span>
<span class="nt">-v</span> <span class="nv">$PWD</span>:/docker-entrypoint-initdb.d <span class="nt">--rm</span> mysql</code></pre></figure>
<p><code class="language-plaintext highlighter-rouge">$PWD</code> assumes you start the server in the directory where your .sql files are. If that’s not the case, replace <code class="language-plaintext highlighter-rouge">$PWD</code>
with the directory that contains your SQL dump.</p>
<h2 id="discussion">Discussion</h2>
<p>It’s possible to create an image with your data already sitting in the
<code class="language-plaintext highlighter-rouge">/docker-entrypoint-initdb.d</code> directory. It could be convenient to use and
distribute this image rather than following these instructions.</p>
<p>With the right docker images, it’s easy to run multiple versions of MySQL.
Use <code class="language-plaintext highlighter-rouge">mysql:8.0.3</code> (for example) instead of <code class="language-plaintext highlighter-rouge">mysql</code> for the image name.</p>
<p>The image’s <a href="https://hub.docker.com/_/mysql/">DockerHub page</a> contains a <em>LOT</em>
more options. If you’re going to use the image, I recommend going over its documentation.</p>
<p>If you’re curious, all the details are contained in the <a href="https://github.com/docker-library/mysql/blob/883703dfb30d9c197e0059a669c4bb64d55f6e0d/5.7/Dockerfile">Dockerfile</a>.</p>
<hr />
<p><a name="fn1">1</a>: I’m taking a MOOC, it’s a long story.</p>
Reject Summary Statistics2017-09-26T00:00:00+00:00https://blog.jpalardy.com/posts/reject-summary-statistics<p>A summary statistic is what you get when you reduce a bunch of numbers down to a single number:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="c1"># the heights of 10 people you know (cm):</span><span class="w">
</span><span class="m">180</span><span class="w"> </span><span class="m">175</span><span class="w"> </span><span class="m">165</span><span class="w"> </span><span class="m">176</span><span class="w"> </span><span class="m">183</span><span class="w"> </span><span class="m">173</span><span class="w"> </span><span class="m">186</span><span class="w"> </span><span class="m">175</span><span class="w"> </span><span class="m">180</span><span class="w"> </span><span class="m">181</span><span class="w">
</span><span class="c1"># the "average" of those heights:</span><span class="w">
</span><span class="m">177.4</span></code></pre></figure>
<p>But it doesn’t have to be the <em>mean</em>, it could be:</p>
<ul>
<li>median / percentile / IQR</li>
<li>mode</li>
<li>min / max / range</li>
<li>variance / standard deviation</li>
<li>etc…</li>
</ul>
<p><a href="https://en.wikipedia.org/wiki/Summary_statistics">Wikipedia</a> says:</p>
<blockquote>
<p>[…] summary statistics are used to summarize a set of observations, in order to communicate the largest amount of information as simply as possible.</p>
</blockquote>
<h2 id="why-are-summary-statistics-bad">Why are summary statistics bad?</h2>
<p>Summary statistics are extremely <a href="https://en.wikipedia.org/wiki/Lossy_compression">lossy</a>.</p>
<p>They take <em>all</em> the data – full of nuances, patterns, outliers, and special cases – and
collapse it down to a single point.</p>
<p>What if the people you know included a basketball player?</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="c1"># the heights of 10 people you know (cm):</span><span class="w">
</span><span class="m">180</span><span class="w"> </span><span class="m">175</span><span class="w"> </span><span class="m">165</span><span class="w"> </span><span class="m">176</span><span class="w"> </span><span class="m">183</span><span class="w"> </span><span class="m">173</span><span class="w"> </span><span class="m">198</span><span class="w"> </span><span class="m">175</span><span class="w"> </span><span class="m">180</span><span class="w"> </span><span class="m">181</span><span class="w">
</span><span class="c1"># ^^^ ... changed 1 value</span><span class="w">
</span><span class="c1"># the "average" of those heights:</span><span class="w">
</span><span class="m">178.6</span></code></pre></figure>
<p>Yes: <code class="language-plaintext highlighter-rouge">178.6</code> is bigger than <code class="language-plaintext highlighter-rouge">177.4</code> … but it doesn’t begin to tell the real story:</p>
<p><img src="/assets/reject-summary-statistics/whoah.png" alt="plot of heights" /></p>
<h2 id="if-you-dont-look-at-the-data-you-dont-understand-the-data">If you don’t look at the data, you don’t understand the data…</h2>
<p>It’s almost a cliché, but <a href="https://en.wikipedia.org/wiki/Anscombe%27s_quartet">Anscombe’s quartet</a> clearly shows
the limits of summary statistics:</p>
<p><img src="/assets/reject-summary-statistics/anscombes-quartet.png" alt="Anscombe's quartet" /></p>
<p>Those four data sets have the same:</p>
<ul>
<li>mean (both x and y)</li>
<li>variance (both x and y)</li>
<li>correlation</li>
<li>linear regression</li>
</ul>
<p>A more recent and striking example is the <a href="https://blog.revolutionanalytics.com/2017/05/the-datasaurus-dozen.html">datasaurus dozen</a>:</p>
<p><img src="/assets/reject-summary-statistics/datasaurus.gif" alt="datasaurus data set" /></p>
<h2 id="a-terrible-example-average-request-duration">A terrible example: “average” request duration</h2>
<p>If you’ve ever been asked:</p>
<blockquote>
<p>What’s the average request duration?</p>
</blockquote>
<p>for an HTTP endpoint, you know there’s only one good answer:</p>
<blockquote>
<p>urgh…</p>
</blockquote>
<p>The truth is that “it’s complicated” and the question itself is based on many wrong assumptions:</p>
<ul>
<li>the distribution is <a href="https://en.wikipedia.org/wiki/Normal_distribution">normal</a> … it’s <a href="https://stats.stackexchange.com/questions/25709/what-distribution-is-most-commonly-used-to-model-server-response-time">not</a></li>
<li>the distribution has <a href="https://en.wikipedia.org/wiki/Unimodality">one mode</a> … probably <a href="https://en.wikipedia.org/wiki/Multimodal_distribution">not</a>, depending on caching, or the specific if-then-else handling, etc…</li>
<li>the mean is “exact” … rather than using a confidence interval</li>
</ul>
<p>Here is a real duration graph distribution:</p>
<p><img src="/assets/reject-summary-statistics/density-durations.png" alt="sample durations distribution" /></p>
<p>Details:</p>
<ul>
<li>this is a <a href="https://ggplot2.tidyverse.org/reference/geom_density.html">density</a> plot, a “smoothed” histogram</li>
<li>yes, this is from a real production system</li>
<li>it spans thousands of requests over a 1-hour window</li>
<li>durations were cropped <= 250ms</li>
<li>the mean duration is 62.4ms (56ms for the cropped data) – <a href="/assets/reject-summary-statistics/duration.csv">raw data</a></li>
</ul>
<p>The mean duration sits exactly <strong>nowhere</strong> interesting or representative.</p>
<h2 id="a-failure-of-communication">A failure of communication</h2>
<p>Every time I hear/read the word “average”, I assume the worst. The average, or
any summary statistics, obscures the real data, incompetently at best or maliciously
at worst.</p>
<p>Summary statistics <em>feel</em> like information, but they are usually sound bites.</p>
<p>As a society, we need to strive for facts, for understanding, and for objective data:</p>
<ul>
<li>REJECT summary statistics: ask for the data if it’s missing</li>
<li>refuse conclusions until the data and methodology are produced and reviewed</li>
<li>raise the bar: ask for visualizations – but visualizations do NOT replace the data, they complement the data</li>
</ul>
Bash Aliases, Functions and Commands2017-08-10T00:00:00+00:00https://blog.jpalardy.com/posts/bash-aliases-functions-and-commands<p>The first thing you type on a command-line is either an alias, a function or a command.
If you haven’t written it yourself, it might not be obvious <em>what</em> exactly you’re calling.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># defining an alias</span>
<span class="nv">$ </span><span class="nb">alias </span><span class="nv">x1</span><span class="o">=</span><span class="s2">"echo hello world"</span>
<span class="c"># defining a function</span>
<span class="nv">$ </span>x2<span class="o">()</span> <span class="o">{</span> <span class="nb">echo </span>hello world<span class="p">;</span> <span class="o">}</span> <span class="c"># ; is important, see below</span>
<span class="c"># defining a command</span>
<span class="nv">$ </span><span class="nb">cat</span> <span class="o">></span> /usr/local/bin/x3 <span class="o"><<</span><span class="no">END</span><span class="sh">
#!/bin/bash
echo hello world
</span><span class="no">END
</span><span class="nv">$ </span><span class="nb">chmod</span> +x /usr/local/bin/x3</code></pre></figure>
<p>If you’re curious about <code class="language-plaintext highlighter-rouge">;</code> in the function definition, read <a href="/posts/one-liner-bash-functions/">One-liner Bash Functions</a>.</p>
<p>They all do the same thing, but you can’t tell what they are:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>x1
hello world
<span class="nv">$ </span>x2
hello world
<span class="nv">$ </span>x3
hello world</code></pre></figure>
<p>How can you tell? Use <a href="/posts/type-a/">type -a</a>:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">type</span> <span class="nt">-a</span> x1
x1 is aliased to <span class="s1">'echo hello world'</span>
<span class="nv">$ </span><span class="nb">type</span> <span class="nt">-a</span> x2
x2 is a <span class="k">function
</span>x2 <span class="o">()</span>
<span class="o">{</span>
<span class="nb">echo </span>hello world
<span class="o">}</span>
<span class="nv">$ </span><span class="nb">type</span> <span class="nt">-a</span> x3
x3 is /usr/local/bin/x3</code></pre></figure>
<h2 id="so-what">So what?</h2>
<p>It becomes important if you chain them. Worst case: an alias for a function that calls a command … all with the same name:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">alias </span><span class="nv">x</span><span class="o">=</span><span class="s2">"echo from the alias; x"</span>
<span class="nv">$ </span>x<span class="o">()</span> <span class="o">{</span>
<span class="nb">echo</span> <span class="s2">"from the function"</span>
<span class="nb">command </span>x <span class="c"># To prevent recursion. There's also a 'builtin' override.</span>
<span class="o">}</span>
<span class="nv">$ </span><span class="nb">cat</span> <span class="o">></span> /usr/local/bin/x <span class="o"><<</span><span class="no">END</span><span class="sh">
#!/bin/bash
echo from the command
echo hello world
</span><span class="no">END
</span><span class="nv">$ </span><span class="nb">chmod</span> +x /usr/local/bin/x
<span class="c"># calling x...</span>
<span class="nv">$ </span>x
from the <span class="nb">alias
</span>from the <span class="k">function
</span>from the <span class="nb">command
</span>hello world
<span class="c"># what am I calling?</span>
<span class="nv">$ </span><span class="nb">type</span> <span class="nt">-a</span> x
x is aliased to <span class="s1">'echo from the alias; x'</span>
x is a <span class="k">function
</span>x <span class="o">()</span>
<span class="o">{</span>
<span class="nb">echo</span> <span class="s2">"from the function"</span><span class="p">;</span>
<span class="nb">command </span>x
<span class="o">}</span>
x is /usr/local/bin/x</code></pre></figure>
<p>The order returned by <code class="language-plaintext highlighter-rouge">type -a</code> is the evaluation order:</p>
<ul>
<li>aliases</li>
<li>functions</li>
<li>commands</li>
</ul>
<h2 id="why-would-you-do-that">Why would you do that?</h2>
<p>The <code class="language-plaintext highlighter-rouge">ls</code> command is often aliased. (… is <code class="language-plaintext highlighter-rouge">ls</code> a command or builtin? <code class="language-plaintext highlighter-rouge">type -a ls</code>).</p>
<p>Or you might do fancy things with <code class="language-plaintext highlighter-rouge">cd</code>.</p>
<p>The point is that you <em>can</em> and it’s <em>sometimes</em> useful.</p>
<h2 id="when-to-use-which">When to use which?</h2>
<p>It’s <a href="https://unix.stackexchange.com/questions/30925/in-bash-when-to-alias-when-to-script-and-when-to-write-a-function">complicated</a>. Some guidelines:</p>
<h3 id="aliases">aliases</h3>
<p>For trivial things, when arguments (if any) come at the end.</p>
<h3 id="functions">functions</h3>
<p>For more complicated things:</p>
<ul>
<li>positional arguments</li>
<li>multi-line</li>
<li>all the Bash you want</li>
<li>“faster” – stored in the shell’s memory</li>
<li>can modify current shell – you <em>can’t</em> implement <code class="language-plaintext highlighter-rouge">cd</code> as a command</li>
</ul>
<h3 id="commands">commands</h3>
<p>For everything else:</p>
<ul>
<li>don’t have to use Bash</li>
<li>easily installed / shared – put somewhere in PATH</li>
<li>can be used by other programs, subshells…</li>
</ul>
SKIP grep, use AWK2017-07-03T00:00:00+00:00https://blog.jpalardy.com/posts/skip-grep-use-awk<p>Over the years, I’ve seen many people use this pattern (filter-map):</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="o">[</span>data is generated] | <span class="nb">grep </span>something | <span class="nb">awk</span> <span class="s1">'{print $2}'</span></code></pre></figure>
<p>but it can be shortened to:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="o">[</span>data is generated] | <span class="nb">awk</span> <span class="s1">'/something/ {print $2}'</span></code></pre></figure>
<h2 id="you-probably-dont-need-grep">You (probably) don’t need grep</h2>
<p>Following this logic, you can replace a simple grep with:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="o">[</span>data is generated] | <span class="nb">awk</span> <span class="s1">'/something/'</span></code></pre></figure>
<p>This will <em>implicitly</em> print lines that match the regular expression.</p>
<p>If feel lost, I’ve got an illuminating
<a href="/posts/why-learn-awk/">series</a> of
<a href="/posts/awk-tutorial-part-1/">posts</a> about
<a href="/posts/awk-tutorial-part-2/">awk</a> for
<a href="/posts/awk-tutorial-part-3/">you</a>.</p>
<h2 id="why-would-i-do-this">Why would I do this?</h2>
<p>I can think of 4 reasons:</p>
<ul>
<li>it’s shorter to type</li>
<li>it spawns one less process</li>
<li>awk uses modern (read “Perl”) regular expressions, by default – like <code class="language-plaintext highlighter-rouge">grep -E</code></li>
<li>it’s ready to “augment” with more awk</li>
</ul>
<h2 id="but-grep--v-is-ok">But “grep -v” is OK…</h2>
<p>It’s <em>possible</em> to emulate <code class="language-plaintext highlighter-rouge">grep -v</code> with awk, but it’s not a good idea:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="o">[</span>data is generated] | <span class="nb">awk</span> <span class="s1">'/something/ {next} 1'</span></code></pre></figure>
<ul>
<li>it’s uglier</li>
<li>it’s longer than <code class="language-plaintext highlighter-rouge">grep -v</code></li>
<li>“what’s does that even do?!” – it requires a deeper understanding of awk</li>
</ul>
<p><strong>UPDATE:</strong></p>
<p>Many people have pointed out that “grep -v” can be done more consicely with:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="o">[</span>data is generated] | <span class="nb">awk</span> <span class="s1">'! /something/'</span></code></pre></figure>
<p>which isn’t too bad at all.</p>
From iptables to UFW: 5 things to note2017-06-11T00:00:00+00:00https://blog.jpalardy.com/posts/from-iptables-to-ufw-5<p>When I migrated a <a href="https://www.digitalocean.com/">droplet</a> from Ubuntu 14.04 to
16.04, the <a href="https://www.digitalocean.com/community/tutorials">tutorials</a> clearly implied
that I should <a href="https://www.digitalocean.com/community/tutorials/how-to-set-up-a-firewall-with-ufw-on-ubuntu-16-04">use UFW</a>.</p>
<p>UFW, the <a href="https://en.wikipedia.org/wiki/Uncomplicated_Firewall">Uncomplicated Firewall</a>, lives up to its name.
From what I read, it does everything I need, in a much simpler way than iptables.</p>
<p>What surprised me was everything those tutorials did NOT say…</p>
<h2 id="no-configuration-files">No configuration files?</h2>
<p>Tutorials will encourage you to open some ports:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">sudo </span>ufw allow 22
<span class="nv">$ </span><span class="nb">sudo </span>ufw allow 80
<span class="nv">$ </span><span class="nb">sudo </span>ufw allow 443 <span class="c"># you know, the usual...</span></code></pre></figure>
<p>and turn on UFW:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">sudo </span>ufw <span class="nb">enable</span></code></pre></figure>
<p>How does that even work?</p>
<p>It turns out that every command reaches into UFW’s configuration files,
under <code class="language-plaintext highlighter-rouge">/etc/ufw/*.rules</code>, and adjusts them accordingly. You wouldn’t normally edit
those files manually.</p>
<h2 id="yes-your-config-will-survive-a-reboot">Yes, your config will survive a reboot.</h2>
<p>Once you enable UFW (above), and you understand that the config files were
written for you, it’s not overly surprising to see why your firewall will come
back up after a reboot.</p>
<p>In fact, this is a refreshing change from <a href="https://www.google.com/search?q=iptables+load+on+reboot">the bad old days</a>.</p>
<h2 id="check-your-rules--before-its-too-late">Check your rules … before it’s too late.</h2>
<p><em>After</em> you enable UFW, if you haven’t locked yourself out, you can review its rules:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">sudo </span>ufw status numbered
Status: active
To Action From
<span class="nt">--</span> <span class="nt">------</span> <span class="nt">----</span>
<span class="o">[</span> 1] 22 ALLOW IN Anywhere
<span class="o">[</span> 2] 80 ALLOW IN Anywhere
<span class="o">[</span> 3] 443 ALLOW IN Anywhere
<span class="o">[</span> 4] 22 <span class="o">(</span>v6<span class="o">)</span> ALLOW IN Anywhere <span class="o">(</span>v6<span class="o">)</span>
<span class="o">[</span> 5] 80 <span class="o">(</span>v6<span class="o">)</span> ALLOW IN Anywhere <span class="o">(</span>v6<span class="o">)</span>
<span class="o">[</span> 6] 443 <span class="o">(</span>v6<span class="o">)</span> ALLOW IN Anywhere <span class="o">(</span>v6<span class="o">)</span></code></pre></figure>
<p>But what about <em>before</em> turning everything on?</p>
<p>I had to go look for it – this <a href="https://bugs.launchpad.net/ufw/+bug/987784">feature</a> was added later on:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">sudo </span>ufw show added
Added user rules <span class="o">(</span>see <span class="s1">'ufw status'</span> <span class="k">for </span>running firewall<span class="o">)</span>:
ufw allow 22
ufw allow 80
ufw allow 443</code></pre></figure>
<h2 id="new-rules-are-applied-live">New rules are applied live.</h2>
<p>If you change the rules:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">sudo </span>ufw delete 3 <span class="c"># close down port 443, see above</span></code></pre></figure>
<p>it’s going to be applied <em>now</em>. You won’t have to restart UFW. As always, the
configuration files will also be updated.</p>
<h2 id="application-provide-their-own-ufw-profiles">Application provide their own UFW profiles</h2>
<p>If you install NGINX, it will drop a file in <code class="language-plaintext highlighter-rouge">/etc/ufw/applications.d</code>:</p>
<figure class="highlight"><pre><code class="language-txt" data-lang="txt">[Nginx HTTP]
title=Web Server (Nginx, HTTP)
description=Small, but very powerful and efficient web server
ports=80/tcp
[Nginx HTTPS]
title=Web Server (Nginx, HTTPS)
description=Small, but very powerful and efficient web server
ports=443/tcp
[Nginx Full]
title=Web Server (Nginx, HTTP + HTTPS)
description=Small, but very powerful and efficient web server
ports=80,443/tcp</code></pre></figure>
<p>and you can see the inventory of profiles:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">sudo </span>ufw app list
Available applications:
Nginx Full
Nginx HTTP
Nginx HTTPS
OpenSSH</code></pre></figure>
<p>You can use a profile to configure UFW:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">sudo </span>ufw allow <span class="s2">"Nginx Full"</span></code></pre></figure>
<p>There are 2 advantages to this:</p>
<ul>
<li>you don’t have to clone-and-own that knowledge into your favorite provisioning tool</li>
<li>modularity means that multiple packages don’t fight for ownership of the firewall configuration files</li>
</ul>
Handling Broken JSON with jq2017-04-19T00:00:00+00:00https://blog.jpalardy.com/posts/handling-broken-json-with-jq<h2 id="problem">Problem</h2>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">cat </span>broken.json
<span class="o">{</span>
<span class="s2">"color"</span>: <span class="s2">"red"</span>
<span class="o">}</span>
<span class="o">{</span>
<span class="s2">"color"</span>: <span class="s2">"green"</span>
<span class="o">{</span>
<span class="s2">"color"</span>: <span class="s2">"blue"</span>
<span class="o">}</span></code></pre></figure>
<p><em>There’s a missing <code class="language-plaintext highlighter-rouge">}</code> after green.</em></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">cat </span>broken.json | jq <span class="nb">.</span>
<span class="o">{</span>
<span class="s2">"color"</span>: <span class="s2">"red"</span>
<span class="o">}</span>
parse error: Expected separator between values at line 6, column 1</code></pre></figure>
<p>When jq hits invalid JSON, it completely <em>stops</em> processing the stream.<br />
That’s not always great.</p>
<h2 id="some-unhelpful-solutions">Some Unhelpful Solutions</h2>
<p>People will be quick to “fix” your problem:</p>
<blockquote>
<p>why don’t you fix the JSON at the source?</p>
</blockquote>
<p>If you can do this, that’s the cleanest way out of this.</p>
<p>But… real life is messy. You don’t always control the JSON you have to process.<br />
I ran into this recently: I extracted JSON logs from a system that decided to
truncate <em>some</em> lines, <em>some</em> of the time.</p>
<blockquote>
<p>why don’t you “check” your JSON before you … “check” your JSON?</p>
</blockquote>
<p>Yes – you <em>could</em> do some minimal regex-based checks, possibly with AWK or grep.<br />
But you know what’s already great at handling JSON? jq.</p>
<h2 id="using---seq">Using <code class="language-plaintext highlighter-rouge">--seq</code></h2>
<p>That’s what you’ll find if you <a href="https://www.google.com/search?q=jq+broken+json">keep searching</a>.
The <a href="https://stedolan.github.io/jq/manual/">documentation</a> says:</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text"> --seq:
Use the application/json-seq MIME type scheme for separating JSON texts in jq’s
input and output. This means that an ASCII RS (record separator) character is
printed before each value on output and an ASCII LF (line feed) is printed
after every output. Input JSON texts that fail to parse are ignored (but warned
about), discarding all subsequent input until the next RS. This mode also
parses the output of jq without the --seq option.</code></pre></figure>
<p>but an example might be clearer:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">cat </span>broken-with-rs.json
<RS>
<span class="o">{</span>
<span class="s2">"color"</span>: <span class="s2">"red"</span>
<span class="o">}</span>
<RS>
<span class="o">{</span>
<span class="s2">"color"</span>: <span class="s2">"green"</span>
<RS>
<span class="o">{</span>
<span class="s2">"color"</span>: <span class="s2">"blue"</span>
<span class="o">}</span></code></pre></figure>
<p>Just put <code class="language-plaintext highlighter-rouge">RS</code> (ASCII character 0x1e) in front of each record. See below for <a href="#an-example">an example</a>.<br />
(<em>or check the <a href="https://tools.ietf.org/html/draft-ietf-json-text-sequence-09">internet draft</a> for more details</em>)</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">cat </span>broken-with-rs.json | jq <span class="nt">--seq</span> <span class="nb">.</span>
<span class="o">{</span>
<span class="s2">"color"</span>: <span class="s2">"red"</span>
<span class="o">}</span>
<span class="o">{</span>
<span class="s2">"color"</span>: <span class="s2">"blue"</span>
<span class="o">}</span></code></pre></figure>
<p><em>The broken “green” entry is skipped…</em></p>
<h2 id="why-isnt-there-a-better-solution">Why isn’t there a better solution?</h2>
<p>When a JSON parser finds a problem, what’s the <em>best</em> solution?</p>
<p>I don’t think there is one.</p>
<p>If the JSON is invalid … how much of it needs to be thrown out?</p>
<ul>
<li>only the current field? – <code class="language-plaintext highlighter-rouge">"color": "green"</code></li>
<li>only the current record? inside the closest curlies? – <code class="language-plaintext highlighter-rouge">{ ... }</code></li>
</ul>
<p>The answer is probably application-specific. And wouldn’t it be <em>worse</em> if jq
silently skipped invalid JSON? How long would it take to debug that?!</p>
<p>With <code class="language-plaintext highlighter-rouge">RS</code> delimiters, you’re explicitly boxing failures: on a parse error, it
skips the current record and forwards to the next <code class="language-plaintext highlighter-rouge">RS</code>.</p>
<h2 id="an-example">An Example</h2>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">cat </span>broken.json | <span class="nb">sed</span> <span class="nt">-e</span> <span class="s1">'s/^{/'</span><span class="si">$(</span><span class="nb">printf</span> <span class="s2">"</span><span class="se">\x</span><span class="s2">1e"</span><span class="si">)</span><span class="s1">'{/'</span> | jq <span class="nt">--seq</span> <span class="nb">.</span>
<span class="o">{</span>
<span class="s2">"color"</span>: <span class="s2">"red"</span>
<span class="o">}</span>
<span class="o">{</span>
<span class="s2">"color"</span>: <span class="s2">"blue"</span>
<span class="o">}</span></code></pre></figure>
<p><em>Catch lines beginning with <code class="language-plaintext highlighter-rouge">{</code> and insert a <code class="language-plaintext highlighter-rouge">RS</code> there.</em></p>
<p><strong>DISCLAIMER</strong> again: the broken “green” entry is skipped… in this case it’s silent,
but I’ve seen other broken cases where a message is shown on STDERR. Use responsibly.</p>
Problem Solving with MiniZinc2017-04-05T00:00:00+00:00https://blog.jpalardy.com/posts/problem-solving-with-minizinc<p><a href="https://www.coursera.org/learn/modeling-discrete-optimization"><img style="float: right" src="/assets/minizinc-and-declarative-programming/modeling-discrete-optimization.png" alt="modeling and discrete optimization course banner" /></a></p>
<p>It all started with this banner; a course recommendation on Coursera:</p>
<p>How can you say “no” to that? (<em>LEGO bricks!</em>)</p>
<p>This course turned out to be <em>brutal</em>. It advertised 5–10 hours/week of work
and might have <strong>under</strong>stated the case. (In my experience, online courses usually
<strong>over</strong>state the amount of work involved.)</p>
<p>Even though it was a hard class, I learned a lot. One of the
reviews says:</p>
<blockquote>
<p>Difficult but rewarding course.</p>
</blockquote>
<p>and that’s also how I would describe it.</p>
<p>This version of the course is a few years old, see <a href="#how-to-get-started">below</a> for a newer version.</p>
<h2 id="minizinc">MiniZinc</h2>
<p>You can’t teach programming without a programming language. Similarly, you
can’t teach optimization modeling abstractly, you’ll need a modeling language.</p>
<p>MiniZinc is the language you’ll use to describe models. These models get
compiled and run on
<a href="https://en.wikipedia.org/wiki/List_of_optimization_software">solvers</a>. The solvers
will search through the problem space and generate possible solutions.</p>
<p>Let’s go through an example.</p>
<h2 id="an-example-the-change-making-problem">An example: The Change-making problem</h2>
<p>From the <a href="https://en.wikipedia.org/wiki/Change-making_problem">wikipedia page</a>:</p>
<blockquote>
<p>how can a given amount of money be made with the least number of coins<br />
of given denominations?</p>
</blockquote>
<p>Specifically, in the US, a cashier would give back $0.78 as 3 quarters and 3 pennies.</p>
<p><strong><em>BEFORE</em></strong> you grab that marker and head over to that whiteboard, let me stop you.<br />
I <strong><em>know</em></strong> that you know how to solve this: that’s why I picked this problem.
And I’m going to explain why it’s a bad idea for <strong><em>you</em></strong> to solve this problem.</p>
<h2 id="a-solution">A solution</h2>
<p>Using MiniZinc, I would try this:</p>
<p><img src="/assets/minizinc-and-declarative-programming/mzn-solution.png" alt="modeling and discrete optimization course banner" /></p>
<ul>
<li>line 2: the amount, in cents, hardcoded to $0.78</li>
<li>line 3: US denominations, in cents</li>
<li>line 5: an array of counts – how many coins to use for each denomination</li>
<li>line 9: <strong>constraint</strong>: the sum of the counts times the denominations <em>must</em> add up to the amount</li>
<li>line 11: the sum of the counts is the number of coins</li>
<li>line 12: minimize the number of coins</li>
<li>line 14-18: how to format the output</li>
</ul>
<p>Run it:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>minizinc change-making.mzn
coins <span class="o">=</span> 6<span class="p">;</span>
denoms <span class="o">=</span> <span class="o">[</span>25, 10, 5, 1]<span class="p">;</span>
counts <span class="o">=</span> <span class="o">[</span>3, 0, 0, 3]<span class="p">;</span>
<span class="nt">----------</span>
<span class="o">==========</span></code></pre></figure>
<blockquote>
<p>NOTE: This program was simplified for the purpose of discussion – I wouldn’t normally hardcode everything. There are ways to
separate your model from your data, and/or to pass arguments from the command-line.</p>
</blockquote>
<h2 id="so-what">So what?</h2>
<p>MiniZinc code is declarative – you’re not telling it <em>how</em> to solve the problem, you’re just telling it what the problem is:</p>
<ul>
<li>what is constant (amount, denominations)</li>
<li>what is variable (counts)</li>
<li>what constraints we have</li>
<li>what to optimize</li>
</ul>
<p>If you’re still not convinced, let me <a href="https://www.urbandictionary.com/define.php?term=hold+my+beer">hold your beer</a>
while you think of <em>your own</em> solution and think how you would integrate the
following constraints without having to rewrite – or worse: completely re-think –
the whole problem:<br />
(<em>let alone doing it in 1 or 2 lines of code</em>)</p>
<p>Is there a solution with 8 coins?</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">constraint coins = 8;
# coins = 8;
# denoms = [25, 10, 5, 1];
# counts = [2, 2, 1, 3];</code></pre></figure>
<p>What’s the best solution without quarters?</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">constraint counts[1] = 0;
# coins = 11;
# denoms = [25, 10, 5, 1];
# counts = [0, 7, 1, 3];</code></pre></figure>
<p>What’s the best solution using exactly 3 nickels?</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">constraint counts[3] = 3;
# coins = 9;
# denoms = [25, 10, 5, 1];
# counts = [2, 1, 3, 3];</code></pre></figure>
<p>Is there a solution where we give a different number for each coin? (but at least one)</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">constraint alldifferent(counts);
constraint forall(i in 1..4) ( counts[i] >= 1 );
# coins = 10;
# denoms = [25, 10, 5, 1];
# counts = [1, 4, 2, 3];</code></pre></figure>
<p>What if we don’t use pennies?</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">constraint counts[4] = 0;
# =====UNSATISFIABLE=====</code></pre></figure>
<p>Right, because sometimes there are no solutions satisfying all the constraints.</p>
<h2 id="stand-on-the-shoulders-of-giants">Stand on the shoulders of giants</h2>
<p>What you write in MiniZinc gets compiled down to a format that solvers can run.
This happens for the same reasons you don’t usually write software in assembly:</p>
<ul>
<li>you can keep your solution at a higher-level</li>
<li>you can keep your solution more portable – you can run it against different solvers that expect different inputs</li>
<li>certain optimizations can be done by the “compiler”</li>
</ul>
<p>Solvers are magnificent beasts. They combine the insights of many researchers
who have dedicated their lives to finding better ways to run certain algorithms
(and other time-saving shortcuts). For example: if you can point out to the
solver that your problem is a variation of the <a href="https://en.wikipedia.org/wiki/Knapsack_problem">Knapsack problem</a>,
you’ll often get answers orders of magnitude faster than you would otherwise.</p>
<p>Don’t reinvent the wheel: determine that you need a wheel, and use one that’s
been designed by a pro.</p>
<h2 id="how-to-get-started">How to get started?</h2>
<p>MiniZinc has its own
<a href="https://www.minizinc.org/">website</a>,
<a href="https://www.minizinc.org/tutorial/minizinc-tute.pdf">tutorial</a> and
<a href="https://www.minizinc.org/doc-lib/doc.html">documentation</a>.
But I can’t say that I find them to be great resources – not until you’re
already familiar with the content.</p>
<p>I recommend the updated courses on Coursera:</p>
<p><a href="https://www.coursera.org/learn/basic-modeling"><img style="float: left" src="/assets/minizinc-and-declarative-programming/basic-modeling.png" alt="basic modeling for discrete optimization course banner" /></a></p>
<p><a href="https://www.coursera.org/learn/advanced-modeling"><img style="float: left" src="/assets/minizinc-and-declarative-programming/advanced-modeling.png" alt="advanced modeling for discrete optimization course banner" /></a></p>
<p><br style="clear: both" /></p>
<p>Write less code and solve harder problems.</p>
<p>If you think that solving sudoku takes more than a few lines of code, or a few minutes, you’re working too hard.</p>
Get Your Last Downloaded File2017-02-15T00:00:00+00:00https://blog.jpalardy.com/posts/get-your-last-downloaded-file<p>Mac software is setup to receive files in <code class="language-plaintext highlighter-rouge">~/Downloads</code>. Since I live on the
command-line, the duality of downloading files and interacting with them is
jarring.</p>
<p>I wanted a way to say:</p>
<blockquote>
<p>…get the file I just downloaded…</p>
</blockquote>
<p>It’s possible to tab-complete on <code class="language-plaintext highlighter-rouge">~/Downloads/</code> but that directory is usually
full of crap (possibly, that’s the real problem).</p>
<p>I’ve been experimenting with the simplest way to automate this:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">ldf<span class="o">()</span> <span class="o">{</span> <span class="c"># stands for "last downloaded file"</span>
<span class="nb">local </span><span class="nv">file</span><span class="o">=</span>~/Downloads/<span class="si">$(</span><span class="nb">ls</span> <span class="nt">-1t</span> ~/Downloads/ | <span class="nb">head</span> <span class="nt">-n1</span><span class="si">)</span>
<span class="nb">read</span> <span class="nt">-p</span> <span class="s2">"confirm: </span><span class="nv">$file</span><span class="s2"> "</span>
<span class="nb">mv</span> <span class="s2">"</span><span class="nv">$file</span><span class="s2">"</span> <span class="nb">.</span>
<span class="o">}</span></code></pre></figure>
<p>Breakdown:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">ls -1t ... | head</code><br />
Find the last downloaded file, based on modification time – a reasonable assumption.</li>
<li><code class="language-plaintext highlighter-rouge">read -p</code><br />
Show me the file, wait for keyboard. I can confirm or bail out with <code class="language-plaintext highlighter-rouge">ctrl-c</code>.</li>
<li><code class="language-plaintext highlighter-rouge">mv</code><br />
Move the file to the current directory.</li>
</ul>
<p>This hack is what I need most of the time.</p>
<p><strong>Added bonus</strong>: you can keep calling the function if you need multiple files
… it will “pop” them out in reverse, as if from a stack.</p>
Untangling Your Homebrew Dependencies2016-12-29T00:00:00+00:00https://blog.jpalardy.com/posts/untangling-your-homebrew-dependencies<p>Does this look familiar?</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>brew <span class="nb">ls</span> | <span class="nb">wc</span> <span class="nt">-l</span>
217</code></pre></figure>
<p>Wait, what? When did this happen?!</p>
<p>You can try looking at the <code class="language-plaintext highlighter-rouge">brew ls</code> output and try to remember what you
installed when. Your next reflex is probably to search “homebrew dependencies”
because there must be an easy solution…</p>
<h2 id="the-basics">The Basics</h2>
<p>What are vim’s dependencies?</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>brew deps vim
perl
python
ruby</code></pre></figure>
<p>Hmmm… what depends on perl?</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>brew uses perl <span class="nt">--installed</span> <span class="c"># --installed is important, otherwise ALL packages are considered</span>
vim</code></pre></figure>
<p>How can I see <em>all</em> dependencies?</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>brew deps <span class="nt">--installed</span>
apg:
bash-completion:
cairo: fontconfig freetype glib libpng pixman
coreutils:
dirmngr: libassuan libgcrypt libgpg-error libksba pth
dnsmasq:
docker:
dos2unix:
ffmpeg: lame x264 xvid
findutils:
<span class="c">## truncated ##</span></code></pre></figure>
<h2 id="the-real-question-what-can-i-uninstall">The Real Question: What Can I Uninstall?</h2>
<p>You can uninstall packages no other packages depend on. Thankfully, there’s a
command for that:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>brew leaves
apg
bash-completion
coreutils
dnsmasq
docker
dos2unix
ffmpeg
findutils
fswatch
gawk
<span class="c">## truncated ##</span></code></pre></figure>
<p>but here’s some bad news… <code class="language-plaintext highlighter-rouge">brew leaves</code> is broken!</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>brew leaves | <span class="nb">grep </span>perl
perl</code></pre></figure>
<p>but, if you remember from above:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>brew uses perl <span class="nt">--installed</span>
vim
<span class="c"># or</span>
<span class="nv">$ </span>brew deps <span class="nt">--installed</span> | <span class="nb">grep </span>perl
perl:
vim: perl python ruby</code></pre></figure>
<p><code class="language-plaintext highlighter-rouge">brew leaves</code> tells you nobody uses perl, but <code class="language-plaintext highlighter-rouge">brew uses</code> confirms that vim uses perl…</p>
<h2 id="detour-dependencies-and-requirements">Detour: Dependencies and Requirements</h2>
<p>Let’s look at vim info:</p>
<p><img src="/assets/untangling-homebrew-dependencies/brew-info-vim.png" alt="brew info vim" /></p>
<p>There are two sections: “dependencies” and “requirements” … so, what’s the difference?</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dependency: a "real" package you depend on
requirement: an "alias" for one of multiple substitute packages
</code></pre></div></div>
<p>For example: vim needs perl, but it’s not picky about <em>which</em> perl is installed.<br />
If you <code class="language-plaintext highlighter-rouge">brew search perl</code>, you’ll find you have many options; all of which
satisfy vim’s “requirement”.</p>
<h2 id="brew-leaves-is-broken">Brew Leaves is Broken</h2>
<p>To make a long story short, <code class="language-plaintext highlighter-rouge">brew leaves</code> only lists dependencies (but not
requirements).</p>
<p>There are a bunch of <a href="https://www.google.com/search?q=homebrew+dependencies">homemade solutions</a>
out there. I went through homebrew’s <a href="https://github.com/Homebrew/brew/tree/master/Library/Homebrew/cmd">code</a> and <code class="language-plaintext highlighter-rouge">brew deps --installed</code> is the
complete source of truth.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>brew deps <span class="nt">--installed</span>
apg:
bash-completion:
cairo: fontconfig freetype glib libpng pixman
coreutils:
dirmngr: libassuan libgcrypt libgpg-error libksba pth
dnsmasq:
docker:
dos2unix:
ffmpeg: lame x264 xvid
findutils:
<span class="c">## truncated ##</span></code></pre></figure>
<p>A leaf is a package that never shows up on the right side of the colon.</p>
<h2 id="brew-graph">brew-graph</h2>
<p>If you feel that the output of <code class="language-plaintext highlighter-rouge">brew deps --installed</code> is not friendly, you’re not alone.</p>
<p><a href="https://github.com/martido/brew-graph">brew-graph</a> is a ruby script that uses the output of <code class="language-plaintext highlighter-rouge">brew deps --installed</code> and generates a graph, in GraphViz or GraphML format. After you install brew-graph, you can visualize your dependencies:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>brew <span class="nb">install </span>graphviz
<span class="nv">$ </span>brew graph <span class="nt">--installed</span> | dot <span class="nt">-T</span> png <span class="nt">-o</span> graph.png
<span class="nv">$ </span>open graph.png</code></pre></figure>
<p>I think you can get better results by using <code class="language-plaintext highlighter-rouge">fdp</code>, instead of <code class="language-plaintext highlighter-rouge">dot</code>, to
generate the image. <code class="language-plaintext highlighter-rouge">fdp</code> is already installed if <code class="language-plaintext highlighter-rouge">dot</code> is installed; they are
both part of GraphViz. The <a href="https://linux.die.net/man/1/fdp">man page</a> says:</p>
<blockquote>
<p>… fdp draws undirected graphs using a ‘‘spring’’ model. It relies on a force-directed approach in the spirit of Fruchterman and Reingold …</p>
</blockquote>
<p>I would also recommend the new <code class="language-plaintext highlighter-rouge">--highlight-leaves</code> option to color (in gray) packages that can be uninstalled:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>brew graph <span class="nt">--installed</span> <span class="nt">--highlight-leaves</span> | fdp <span class="nt">-T</span> png <span class="nt">-o</span> graph.png
<span class="nv">$ </span>open graph.png</code></pre></figure>
<p><img src="/assets/untangling-homebrew-dependencies/brew-deps-hl.png" alt="brew deps as graph w/ highlights" /></p>
The Best Books I Read in 20162016-12-26T00:00:00+00:00https://blog.jpalardy.com/posts/best-books-i-read-2016<p>I have read many books in 2016, but it wasn’t all good. Some books, however,
deserve to be acknowledged. I broke down my recommendations by categories:
<a href="#technical">technical</a>, <a href="#non-fiction">non-fiction</a> and <a href="#fiction">fiction</a>.
There is no particular order.</p>
<p>I have a similar post for <a href="https://blog.jpalardy.com/posts/best-books-i-read-2015/">2015</a>.</p>
<hr />
<h2 id="technical">Technical</h2>
<h3 id="bitcoin-for-the-befuddled">Bitcoin for the Befuddled</h3>
<p><a href="https://www.amazon.com/dp/1593275730/"><img class="book-cover" src="/assets/best-books-2016/1593275730.jpg" alt="Bitcoin for the Befuddled" /></a></p>
<p>Is this a <em>technical</em> book? It deals with technical topics
(virtual money, blockchains, cryptography, mining…), but it has cute cartoons
to help explain some of the concepts. It reminds me of my review of
<a href="https://blog.jpalardy.com/posts/best-books-i-read-2015/#economix">Economix</a>
– edutainment: learning while being entertained, it’s the best.</p>
<p>Does Bitcoin have to be this complicated? Can it not be explained simply? I
think this book managed to explain everything in the simplest way it <em>could</em> be
done. It also tried to stay away from the temporary aspects of Bitcoin (specific websites or software) and
focus instead on the ideas that won’t change as time passes.</p>
<ul>
<li><a href="https://www.amazon.com/dp/1593275730">amazon</a></li>
</ul>
<hr />
<h2 id="non-fiction">Non-Fiction</h2>
<h3 id="the-life-changing-magic-of-tidying-up">The Life-Changing Magic of Tidying Up</h3>
<p><a href="https://www.amazon.com/dp/1607747308/"><img class="book-cover" src="/assets/best-books-2016/1607747308.jpg" alt="The Life-Changing Magic of Tidying Up: The Japanese Art of Decluttering and Organizing" /></a></p>
<p>I saw it in a bookstore. Then, I stumbled on an
<a href="https://www.nytimes.com/2014/10/23/garden/home-organization-advice-from-marie-kondo.html">article</a>
in the New York Times. Then, suddenly, <em>everybody</em> was talking about it. It’s a
small inexpensive book, so I decided to buy it and see what all the fuss was about.</p>
<p>Did it change my life? Maybe.</p>
<p>I’ve never been much of a hoarder, but I have rooms, closets, drawers, and boxes
that aren’t exactly tidy. When you think about why you’re holding on to
physical things in your life, you probably don’t have good reasons.
<a href="https://www.youtube.com/watch?v=tPPvkhGZT7Y">The things you own end up owning you</a>.</p>
<p>The book proposes a test: take something you own and ask yourself: does it
bring you joy? No? Throw it out. Then, keep doing that with everything you own.</p>
<p>It made me throw out A LOT of things. And it made me feel good about the process.
I wouldn’t elevate this to a religion, like some people have … but it’s worth a read.</p>
<ul>
<li><a href="https://www.amazon.com/dp/1607747308/">amazon</a></li>
</ul>
<h3 id="think-better">Think Better</h3>
<p><a href="https://www.amazon.com/dp/0071494936/"><img class="book-cover" src="/assets/best-books-2016/0071494936.jpg" alt="Think Better: An Innovator's Guide to Productive Thinking" /></a></p>
<p>A whole book on brainstorming. This book was mentioned in an <a href="https://nickbentleygames.wordpress.com/2014/05/12/the-100-10-1-method-for-game-design/">interesting article</a> from Nick Bentley.</p>
<p>Everybody <em>knows</em> how to brainstorm, right? You just blurt out your ideas
and … you’re not supposed to judge and … magic happens!</p>
<p>That’s mostly it – but it’s good to understand the what and the how. In practice,
there are many ways to brainstorm badly and few ways to do it well. I’ve
read other books that tried to explain how to do it – and other books that
explained why brainstorming doesn’t work.</p>
<p>Main takeaway: you can brainstorm by yourself (some people argue that’s the
only way it’s <em>really</em> going to work) and you need to KEEP PUSHING, especially
when you think you’re out of ideas. You need to get to the <a href="https://www.google.com/search?q=the+third+third+brainstorming">3rd third</a>.</p>
<p>Whether or not your job requires acts of creation, it inspired me to think
outside the box and to come up with more ideas than I usually would.</p>
<ul>
<li><a href="https://www.amazon.com/dp/0071494936/">amazon</a></li>
</ul>
<h3 id="nonviolent-communication">Nonviolent Communication</h3>
<p><a href="https://www.amazon.com/dp/189200528X/"><img class="book-cover" src="/assets/best-books-2016/189200528X.jpg" alt="Nonviolent Communication: A Language of Life" /></a></p>
<p>Many people and many blog posts have independently recommended this book. And
yet, I can’t say that the idea of the book, nor its title really excited me. I
found it, by accident, in a bookstore and remembered it was supposed to be
<em>really</em> good. I bought it on a whim.</p>
<p>Wow… this book really made me think. It made me question how I communicate
with others. And I think, now more than ever, it’s a very important book to
discuss <em>how</em> we communicate with each other as a society.</p>
<p>I never realized how <em>violent</em> we are to each other. But if you think back
about our origin as a species, about tribes and dominance, you’ll find that
most of the things we say to each other carry cultural baggage.</p>
<p>If you’re still not sure, you can watch <a href="https://www.youtube.com/results?search_query=nvc+marshall+rosenberg">videos</a>
of the author discussing NVC (nonviolent communication).</p>
<ul>
<li><a href="https://www.amazon.com/dp/189200528X/">amazon</a></li>
</ul>
<h3 id="sustainable-energy-without-the-hot-air">Sustainable Energy Without the Hot Air</h3>
<p><a href="https://www.amazon.com/dp/0954452933/"><img class="book-cover" src="/assets/best-books-2016/0954452933.jpg" alt="Sustainable Energy Without the Hot Air" /></a></p>
<p>Climate change is real, but maybe a bit abstract. Even if you don’t believe in
climate change, you might wonder what will happen when we run out of oil.</p>
<ul>
<li>how do we use energy?</li>
<li>where is this energy coming from?</li>
<li>what can we use instead of oil and coal?</li>
<li>how do we deal with the unpredictability of wind/solar?</li>
</ul>
<p>This book skips the politics and deals quantitatively with the problem. How
much energy is there in a gallon of oil? How much energy do you need to move a
car? The physics is going to be explained to you, and you’re going to make these
calculations yourself.</p>
<p>This book paints a sobering picture of the current situation but offers
alternatives. Sooner or later, we’re going to have to make some changes –
whether we like it or not.</p>
<p>I found this book on Bret Victor’s <a href="http://worrydream.com/ClimateChange/">Climate Change</a> page.</p>
<ul>
<li><a href="https://www.amazon.com/dp/0954452933/">amazon</a></li>
</ul>
<h3 id="whole-earth-discipline">Whole Earth Discipline</h3>
<p><a href="https://www.amazon.com/dp/0143118285/"><img class="book-cover" src="/assets/best-books-2016/0143118285.jpg" alt="Whole Earth Discipline" /></a></p>
<p>It turns out I was wrong about many things:</p>
<ul>
<li>nuclear energy</li>
<li>transgenic crops</li>
<li>farming</li>
<li>poverty</li>
<li>cities</li>
<li>and much more…</li>
</ul>
<p>Most ideas about what’s “green” or what’s good for the planet are <em>not</em> based on
science. I don’t want to oversimplify because I think this book can explain it
better than I can.</p>
<p>I found this book on Bret Victor’s <a href="http://worrydream.com/ClimateChange/">Climate Change</a> page.</p>
<ul>
<li><a href="https://www.amazon.com/dp/0143118285/">amazon</a></li>
</ul>
<hr />
<h2 id="fiction">Fiction</h2>
<h3 id="rainbows-end">Rainbows End</h3>
<p><a href="https://www.amazon.com/dp/B0011T1S4Q/"><img class="book-cover" src="/assets/best-books-2016/B0011T1S4Q.jpg" alt="Rainbows End" /></a></p>
<p>I kept hearing about Vernor Vinge. I started with his
<a href="https://en.wikipedia.org/wiki/Vernor_Vinge#Zones_of_Thought_series">Zones of Thought series</a>,
which I enjoyed but can’t recommend without serious caveats.</p>
<p>Rainbows End, meanwhile, is a self-standing book about an unspecified near
future, augmented reality, ubiquitous computing, AI, mind-control … a vision
about the impact of technology on society.</p>
<p>If you want good science fiction that goes beyond “aliens-and-spaceships” with a
story that makes you think, I recommend you give it a try.</p>
<ul>
<li><a href="https://www.amazon.com/dp/B0011T1S4Q/">amazon</a></li>
<li><a href="https://en.wikipedia.org/wiki/Rainbows_End">wikipedia</a></li>
</ul>
<h3 id="deaths-end">Death’s End</h3>
<p><a href="https://www.amazon.com/dp/B01LW7NVP0/"><img class="book-cover" src="/assets/best-books-2016/B01LW7NVP0.jpg" alt="Death's End" /></a></p>
<p>This is the last book in the “three-body” series. I read the first two books
last year and <a href="https://blog.jpalardy.com/posts/best-books-i-read-2015/#the-three-body-problem">recommended</a>
them. At that time, I hadn’t finished the series because this book hadn’t been
translated yet.</p>
<p>I bought the book the day it came out in September, and I wasn’t disappointed :-)</p>
<ul>
<li><a href="https://www.amazon.com/dp/B01LW7NVP0/">amazon</a></li>
<li><a href="https://en.wikipedia.org/wiki/Death%27s_End">wikipedia</a></li>
</ul>
<hr />
<h2 id="closing-thoughts">Closing Thoughts</h2>
<p>If you have liked this blog post, please write your own. I want to read your book
reviews and recommendations.</p>
10 Pages a Day2016-10-13T00:00:00+00:00https://blog.jpalardy.com/posts/10-pages-a-day<p><a href="http://oreol.info/10-pages-a-day.html">Russian Translation</a>, thanks to Angelina.<br />
<a href="http://sciencevobe.com/2019/10/25/10-pages-a-day/">Macedonian Translation</a>, thanks to Katerina.<br />
<a href="https://www.besserehaltung.de/10-seiten-pro-tag.html">German Translation</a>, thanks to Maximilian.<br />
<a href="http://lpacode.com/10-pages-a-day/">Georgian Translation</a>, thanks to Ana.<br />
<a href="https://guideslib.com/publications/10-pages-a-day/">Bulgarian Translation</a>, thanks to Zlatan.<br />
<a href="https://nobullshitseeds.com/translate/10-paginas-per-dag/">Dutch Translation</a>, thanks to Justin.<br />
<a href="https://prodocs24.com/articles/10-pages-a-day/">Azerbaijanian Translation</a>, thanks to Amir.<br />
<a href="https://medicinskanyheter.com/eric-karlsson/10-aidor-om-dagen.html">Swedish Translation</a>, thanks to Eric.</p>
<h2 id="reading-more">Reading more</h2>
<p>I don’t read enough.</p>
<p>That’s how I have felt for years. I love hearing about books, visiting
bookstores, and buying books. It’s not exactly
<a href="https://en.wikipedia.org/wiki/Tsundoku">tsundoku</a> because I always end up
reading what I buy. But no matter how much I read, it never seemed enough.
There were other people, out there, reading more…</p>
<p>Who are these people who have enough time to read?</p>
<p>If you want to read more … you need to … read more. You need to sit down,
open that book, and read. Turn off Netflix, put down your phone, skip that
tennis game and read instead.</p>
<p>Some books have haunted me for weeks and months; books I didn’t like
but wanted to finish. Many times, I forced myself to sit down and read … and I
was surprised at how much I could read if that’s all I did. It’s
<a href="https://en.wikipedia.org/wiki/Grinding_(video_gaming)">grinding</a> applied to
reading books.</p>
<h2 id="feeling-like-it">Feeling like it</h2>
<p>I spent most of my adult life reading books when I “feel like it”. It took me
years to realize that was getting me nowhere.</p>
<p>You start a book you like, read it a few days in a row … then life gets in
the way. The next thing you know, it has been <em>weeks</em> since you’ve picked up
that book. You can argue that other things were important (true), or that it’s the
book’s fault (for not being more interesting), or that your schedule is opening
up (lol) … really, you need to make time.</p>
<p>You can find time to do what’s important to you.</p>
<h2 id="10-pages-a-day">10 pages a day</h2>
<p>The idea just popped into my head: <em>just</em> read 10 pages a day.</p>
<p>Reading 10 pages doesn’t sound like a lot, and that’s why it works. It’s
a “reasonable” commitment. “Of course I <em>can</em> read 10 pages a day” you think.
And you are right.</p>
<p>But 10 pages a day means:</p>
<ul>
<li>70 pages per week</li>
<li>≈300 pages per month</li>
<li>≈3600 pages per year</li>
</ul>
<p>and that’s probably more than what you usually read.</p>
<h2 id="in-practice">In practice</h2>
<p>Every morning, I get up, eat breakfast, and then I sit down to read 10 pages.
It’s my quiet time. I’ve been doing it for about 6 months now.</p>
<p>Sometimes I keep reading for a bit, depending on the book, but most of the time
I don’t. I might read more later in the day, but that falls back into “if I
feel like it” and depends on whatever else is happening in my life.</p>
<h2 id="be-reasonable">Be reasonable</h2>
<p>If “10 pages a day” is good, is “20 pages a day” better?</p>
<p>No.</p>
<p>I recommend you start with 10 and consider yourself “done” for the day. If you
keep reading, those aren’t “credits” against tomorrow. Also, it doesn’t work if
you skip reading for a week and sit down for 70 pages of “punishment” on the
weekend.</p>
<p>What you are aiming for is <em>sustainability</em>. If you can commit to “20 pages a day”, that’s
great – only if you can keep it up.</p>
My Best Awk Tricks2016-06-29T00:00:00+00:00https://blog.jpalardy.com/posts/my-best-awk-tricks<p>This is a wrap-up of my AWK tutorial series.<br />
You can start with <a href="https://blog.jpalardy.com/posts/why-learn-awk/">why learn awk</a>.<br />
Or you can jump straight to <a href="https://blog.jpalardy.com/posts/awk-tutorial-part-1/">part 1</a> of the tutorial.</p>
<p>If you’ve read the tutorial, the amount of <em>magic</em> should be down to a minimum.</p>
<h2 id="disclaimers">Disclaimers</h2>
<p>Before you ask: I have a cheatsheet, and that’s where I keep the recipes that
follow. I’m just human – I copy and paste what I need.</p>
<p>I’m not the author of these recipes; I only collected them over time.</p>
<p>Allow me to skip the <code class="language-plaintext highlighter-rouge">cat FILE |</code> or <code class="language-plaintext highlighter-rouge">awk 'YOUR SCRIPT' FILE</code> parts. By now,
I trust you to figure that out.</p>
<p>Which specific columns make sense for your specific needs will depend on you.
I might use <code class="language-plaintext highlighter-rouge">$0</code> or <code class="language-plaintext highlighter-rouge">$1</code>, but you’ll have to fix those. That’s what I do after I paste.</p>
<h2 id="uniq-without-sort">uniq without sort</h2>
<p>I posted about this <a href="https://blog.jpalardy.com/posts/unsorted-uniq/">before</a>, but it’s still my favorite:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">awk</span> <span class="s1">'!seen[$0]++'</span></code></pre></figure>
<p>The action is <code class="language-plaintext highlighter-rouge">print</code>, of course. The condition is <em>true</em> the first time a
string is put into the array. It follows that subsequent appearances won’t be
printed.</p>
<p>Related to the above, print duplicates (without sort):</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">awk</span> <span class="s1">'++seen[$0] == 2'</span></code></pre></figure>
<p>Print the 2nd time you see a string.</p>
<h2 id="group-counts-or-sums">Group counts or sums</h2>
<p>This was covered in the tutorial, but it’s damn useful:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">awk</span> <span class="s1">'{ groups[$0]++ } END { for (k in groups) print groups[k], k }'</span> <span class="c"># count</span>
<span class="nv">$ </span><span class="nb">awk</span> <span class="s1">'{ groups[$1] += $2 } END { for (k in groups) print groups[k], k }'</span> <span class="c"># sum</span></code></pre></figure>
<p>Accumulate in an array, report at the <code class="language-plaintext highlighter-rouge">END</code>. In both cases, pay attention to the columns you use.</p>
<h2 id="set-operations-union-intersection-difference">Set operations: union, intersection, difference</h2>
<p>If you have multiple files, and you consider their content as sets, you can generate
a bunch of interesting subsets.</p>
<h3 id="union">Union</h3>
<p><code class="language-plaintext highlighter-rouge">cat</code> all the files and use the “uniq without sort” recipe from above :-)</p>
<h3 id="intersection">Intersection</h3>
<p>For both intersection and difference, you need to accumulate from one file and
process the other file.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">awk</span> <span class="s1">'NR == FNR {lut[$0] = 1; next} $0 in lut {print}'</span> FILE1 FILE2</code></pre></figure>
<p>The main trick is to realize that <code class="language-plaintext highlighter-rouge">NR</code> and <code class="language-plaintext highlighter-rouge">FNR</code> will, by definition, only be equal
during the processing of the first file. The <code class="language-plaintext highlighter-rouge">next</code> statement ensures the rest
of the one-liner is skipped. We load the lut (LookUp Table) array with the relevant
parts from the first file.</p>
<p>Why use <code class="language-plaintext highlighter-rouge">$0 in lut</code> instead of <code class="language-plaintext highlighter-rouge">lut[$0]</code> for the condition? That’s an
optimization I learned the hard way: even a <em>miss</em> lookup in <code class="language-plaintext highlighter-rouge">lut[$0]</code> will
instantiate the array location to an empty string – and over the processing of
HUGE files, you’ll eventually consume a lot of memory.</p>
<p><em>It takes a LOT for this problem to <strong>be</strong> a problem with the amount of memory
that computers have nowadays … that’s why I didn’t cover the <code class="language-plaintext highlighter-rouge">in</code> operator in
the tutorial.</em></p>
<h3 id="difference">Difference</h3>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">awk</span> <span class="s1">'NR == FNR {lut[$0] = 1; next} !($0 in lut) {print}'</span> FILE1 FILE2</code></pre></figure>
<p>This operation isn’t symmetrical: you’re removing the entries from FILE1 from
FILE2. Switch the files around to get the other set difference.</p>
<h2 id="easy-performance">Easy performance</h2>
<p>If your AWK script isn’t fast enough, it might be time to consider whether AWK
is the right tool for the job. How many GB of data are you piping through it?!</p>
<p>That being said, I know 2 tricks to speed up AWK:</p>
<h3 id="drop-unicode-support">Drop unicode support</h3>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ LC_ALL</span><span class="o">=</span>C <span class="nb">awk</span> <span class="s1">'YOUR SCRIPT'</span></code></pre></figure>
<p>The <code class="language-plaintext highlighter-rouge">LC_ALL</code> variable forced to C will drop unicode support and, sometimes, greatly
speed up processing.</p>
<h3 id="use-mawk">Use mawk</h3>
<p>There are many variants of AWK, and the one you’re using is <em>probably</em> GNU AWK.<br />
There are others: <a href="https://invisible-island.net/mawk/">mawk</a> is one of the
FAST one.</p>
<ul>
<li>is mawk already installed?</li>
<li>how much of a pain will it be to install?</li>
<li>will my AWK script work without modifications?</li>
</ul>
<p>These are all good questions. In all likelihood:</p>
<ul>
<li>mawk won’t be installed…</li>
<li>it will be easy to install (<code class="language-plaintext highlighter-rouge">brew install mawk</code>, for example)</li>
<li>your <em>unmodified</em> AWK script will just run FASTER</li>
</ul>
<p>If you’re hitting the performance wall, giving mawk a chance might be worth it.</p>
Awk Tutorial, part 32016-06-16T00:00:00+00:00https://blog.jpalardy.com/posts/awk-tutorial-part-3<p>I <a href="https://blog.jpalardy.com/posts/why-learn-awk/">already mentioned</a> why you should learn AWK.<br />
In <a href="https://blog.jpalardy.com/posts/awk-tutorial-part-1/">part 1</a>, we laid a <em>solid</em> foundation.<br />
In <a href="https://blog.jpalardy.com/posts/awk-tutorial-part-2/">part 2</a>, we covered <em>most</em> of what you would ever need.<br />
Let’s cover what’s left.</p>
<h2 id="input-separators">Input Separators</h2>
<p>How does AWK decide what a “column” is and isn’t?</p>
<p>AWK “trims” the line and separates on adjacent whitespace: <code class="language-plaintext highlighter-rouge">\s+</code> as a regex.
That’s <em>usually</em> what you want, but you can specify what you need with the <code class="language-plaintext highlighter-rouge">-F</code>
option:</p>
<figure class="highlight"><pre><code class="language-txt" data-lang="txt">$ cat netflix.csv | awk -F, '{print $5}' # split columns on commas
Close
98.360001
99.589996
99.839996
101.059998
101.120003
# snip</code></pre></figure>
<p>(source file: <a href="/assets/awk-tutorials/netflix.csv">netflix.csv</a>)</p>
<p>In reality, the <code class="language-plaintext highlighter-rouge">-F</code> option takes a regular expression:</p>
<figure class="highlight"><pre><code class="language-txt" data-lang="txt">$ cat netflix.csv | awk -F'[,-]' '{print $3, "--", $0}'
High -- Date,Open,High,Low,Close,Volume,Adj Close
24 -- 2016-03-24,98.639999,98.849998,97.07,98.360001,10646900,98.360001
23 -- 2016-03-23,99.75,100.389999,98.809998,99.589996,8292300,99.589996
22 -- 2016-03-22,100.480003,101.519997,99.199997,99.839996,9039500,99.839996
21 -- 2016-03-21,101.150002,102.099998,99.50,101.059998,9562900,101.059998
18 -- 2016-03-18,100.50,102.410004,100.010002,101.120003,15437300,101.120003
# snip</code></pre></figure>
<p>It separated on commas or hyphens, and picked the 3rd column (the day). This
can be a useful approach to extract subfields.</p>
<h2 id="output-separators">Output Separators</h2>
<p>When you use a comma (<code class="language-plaintext highlighter-rouge">,</code>) in a print statement, that means “space”, right?<br />
By default, it does. But that’s configurable:</p>
<figure class="highlight"><pre><code class="language-txt" data-lang="txt">$ cat netflix.tsv | awk '{print $1, $6}' # default: space
Date Volume
2016-03-24 10646900
2016-03-23 8292300
2016-03-22 9039500
2016-03-21 9562900
2016-03-18 15437300
# snip
$ cat netflix.tsv | awk 'BEGIN {OFS=","} {print $1, $6}' # custom: comma
Date,Volume
2016-03-24,10646900
2016-03-23,8292300
2016-03-22,9039500
2016-03-21,9562900
2016-03-18,15437300</code></pre></figure>
<p>The OFS (Output Field Separator) variable controls what goes between each
field. I don’t use it very much; I usually format the output
explicitly. But it’s <em>sometimes</em> useful, and it’s good to know.</p>
<h2 id="passing-in-variables">Passing in Variables</h2>
<p>As in the previous example, you could decide to initialize variables in the
<code class="language-plaintext highlighter-rouge">BEGIN</code> block. But that’s not always possible – the <code class="language-plaintext highlighter-rouge">BEGIN</code> block lives inside
the single-quotes which severely limits what you can do.</p>
<p>What if you want to pass in variables, maybe from a shell script?</p>
<figure class="highlight"><pre><code class="language-txt" data-lang="txt">$ cat netflix.tsv | awk -v col=6 '{print $col}' # instead of hardcoding $6
Volume
10646900
8292300
9039500
9562900
# snip</code></pre></figure>
<p>The <code class="language-plaintext highlighter-rouge">-v</code> (mnemonic “var”) option allows you to set variable from outside the
script, a convenient place where shell variables and substitutions are
available. We revisit the <code class="language-plaintext highlighter-rouge">OFS</code> variable from above, in the way I would prefer
to set it:</p>
<figure class="highlight"><pre><code class="language-txt" data-lang="txt">$ cat netflix.tsv | awk -v OFS=, '{print $1, $6}'
Date,Volume
2016-03-24,10646900
2016-03-23,8292300
2016-03-22,9039500
2016-03-21,9562900
2016-03-18,15437300
# snip</code></pre></figure>
<h2 id="arrays">Arrays</h2>
<p>Imagine a programming language without arrays or dictionaries. That’s how we’ve
been using AWK up until now. But everything truly clever you can do with AWK (or any programming language)
probably requires arrays.</p>
<p>AWK arrays are very similar to JavaScript arrays: they can serve both as
“regular” arrays (with number keys) or as dictionaries (with string keys).</p>
<p>How would we <code class="language-plaintext highlighter-rouge">SUM(volume) GROUP BY year</code>?</p>
<figure class="highlight"><pre><code class="language-txt" data-lang="txt"># warning -- scroll right...
$ cat netflix.csv | awk -F'[,-]' '{volume[$1] += $8} END { for(year in volume) print year, volume[year]}'
2009 2904000400
Date 0
2010 7126840000
2002 782530000
2011 11185849500
2003 4256021000
# snip</code></pre></figure>
<ul>
<li>split columns on commas or hyphens</li>
<li>accumulate volume ($8) in a dictionary, using the year ($1) as key</li>
<li>volume is a variable that gets created as a dictionary, because we use it as a dictionary – this was discussed in <a href="https://blog.jpalardy.com/posts/awk-tutorial-part-2/">part 2</a></li>
<li>at the END, print each year and volume sum</li>
</ul>
<p>When working with arrays, this pattern of “accumulation” and “reporting” at the
END is commonplace. But there are some problems with the output, and they highlight
interesting points:</p>
<ul>
<li>the output isn’t sorted, the for-loop makes no guarantee over the order of the keys
<em>(that can be fixed with a trailing <code class="language-plaintext highlighter-rouge">sort -n</code>)</em></li>
<li>the header was considered as a key, “Date”, and accumulated 0<br />
<em>(that can be fixed by removing the first line before AWK (<code class="language-plaintext highlighter-rouge">sed 1d</code>) or by using</em><br />
<em><code class="language-plaintext highlighter-rouge">NR > 1</code> before the accumulator block)</em></li>
</ul>
<h2 id="longer-scripts">Longer Scripts</h2>
<p>I’ve been selling AWK as a language optimized for one-liners, but it’s possible
to reach unpleasant extremes. The last example wasn’t too complicated, but it was
long. It could be made more readable.</p>
<p>It’s possible to package multiple lines of AWK in a bash script:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c">#!/bin/bash</span>
<span class="nb">cat</span> <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span> | <span class="nb">awk</span> <span class="nt">-F</span><span class="s1">'[,-]'</span> <span class="s1">'
{volume[$1] += $8}
END {
for(year in volume) {
print year, volume[year]
}
}
'</span></code></pre></figure>
<ul>
<li>there’s a shebang line, it’s a bash script on the outside</li>
<li><code class="language-plaintext highlighter-rouge">cat "$@"</code> is a passthrough: if you feed a filename to the script, it will be used. If you don’t, an empty <code class="language-plaintext highlighter-rouge">cat</code> won’t hurt.<br />
<em>(it allows you to call <code class="language-plaintext highlighter-rouge">SCRIPTNAME file</code> or <code class="language-plaintext highlighter-rouge">cat file | SCRIPTNAME</code>)</em></li>
<li>notice the trailing single-quote on the AWK line, that’s where your script begins</li>
<li>you can use multiple lines within the single-quotes</li>
<li>use your own good taste to format the code</li>
<li>notice the single-quote at the end, that’s a good place to put a pipe (if you need it)</li>
</ul>
<p>Now you can combine the best of bash scripting, with the power of AWK – all in
a very portable package.</p>
<h2 id="taking-inventory-what-can-you-do">Taking inventory: what can you do?</h2>
<p>Everything I can.</p>
<p>Specifically, you have:</p>
<ul>
<li>more flexible ways to parse inputs with <code class="language-plaintext highlighter-rouge">-F</code></li>
<li>more convenient ways to print outputs with <code class="language-plaintext highlighter-rouge">OFS</code></li>
<li>ultimate programming power: arrays!</li>
<li>a nice way to package your logic in a bash script</li>
</ul>
<h2 id="exercises">Exercises</h2>
<p>Try to:</p>
<ul>
<li>calculate the average closing price, grouped per year</li>
<li>calculate the max closing price, grouped per month</li>
<li>calculate the median volume, in 2015 – you might need <a href="https://www.gnu.org/software/gawk/manual/html_node/Array-Sorting-Functions.html#Array-Sorting-Functions">this</a></li>
</ul>
<p>Answers are <a href="/assets/awk-tutorials/answers-part3.txt">here</a>.</p>
<h2 id="whats-next">What’s next?</h2>
<p>As a conclusion: <a href="https://blog.jpalardy.com/posts/my-best-awk-tricks/">my best AWK tricks</a>.</p>
<p>At this point, I hope they won’t be a list of opaque incantations.<br />
You will be able to see <em>what</em> and <em>how</em> it’s done.</p>
Awk Tutorial, part 22016-05-04T00:00:00+00:00https://blog.jpalardy.com/posts/awk-tutorial-part-2<p>I <a href="https://blog.jpalardy.com/posts/why-learn-awk/">already mentioned</a> why you should learn AWK.<br />
In <a href="https://blog.jpalardy.com/posts/awk-tutorial-part-1/">part 1</a>, we laid a <em>solid</em> foundation.<br />
Let’s build on top of that.</p>
<p><em>NOTE: certain command outputs have been pretty-printed. Pipe through <code class="language-plaintext highlighter-rouge">column -t</code> to obtain similar results.</em></p>
<h2 id="matching-with-regular-expressions">Matching with Regular Expressions</h2>
<p>So far, I’ve shown you ways to match lines based on column values. In practice,
you usually want to match lines with regular expressions. For example, you can
extract data from 2015:</p>
<figure class="highlight"><pre><code class="language-txt" data-lang="txt">$ cat netflix.tsv | awk '/^2015-/'
2015-12-31 116.209999 117.459999 114.279999 114.379997 9245000 114.379997
2015-12-30 118.949997 119.019997 116.43 116.709999 8116200 116.709999
2015-12-29 118.190002 119.599998 116.919998 119.120003 8159200 119.120003
2015-12-28 117.260002 117.349998 113.849998 117.110001 8406300 117.110001
2015-12-24 118.220001 118.800003 117.300003 117.330002 3531300 117.330002
# snip</code></pre></figure>
<p>(<em>achievement unlocked: you have re-created grep…</em>)</p>
<p>A regular expression, by itself, is a shorthand for the condition: <code class="language-plaintext highlighter-rouge">$0 ~ /regex/</code></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">awk</span> <span class="s1">'/^2015-/'</span>
<span class="c"># means:</span>
<span class="nb">awk</span> <span class="s1">'$0 ~ /^2015-/'</span></code></pre></figure>
<p>This means you can match regular expressions on <em>specific</em> columns. You can
extract the data for the 1st of every month:</p>
<figure class="highlight"><pre><code class="language-txt" data-lang="txt">$ cat netflix.tsv | awk '$1 ~ /-01$/'
2016-03-01 94.580002 99.160004 93.610001 98.300003 16997700 98.300003
2016-02-01 91.790001 97.18 91.300003 94.089996 19618000 94.089996
2015-12-01 124.470001 125.57 122.419998 125.370003 12528800 125.370003
2015-10-01 102.910004 106.110001 101.120003 105.980003 17426900 105.980003
2015-09-01 109.349998 111.239998 103.82 105.790001 35977100 105.790001
# snip</code></pre></figure>
<p>That’s already way better than grep.</p>
<h2 id="comparisons-and-logic">Comparisons and Logic</h2>
<p>I glossed over that in <a href="https://blog.jpalardy.com/posts/awk-tutorial-part-1/">part 1</a>,
but AWK has all the usual comparison operators:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$2</span> <span class="o">==</span> 124.47 <span class="c"># equality</span>
<span class="nv">$2</span> <span class="o">!=</span> 124.47 <span class="c"># inequality</span>
<span class="nv">$2</span> <span class="o">></span> 124.47 <span class="c"># greater than</span>
<span class="nv">$2</span> <span class="o">>=</span> 124.47 <span class="c"># greater than or equal</span>
<span class="nv">$2</span> < 124.47 <span class="c"># smaller than</span>
<span class="nv">$2</span> <<span class="o">=</span> 124.47 <span class="c"># smaller than or equal</span>
<span class="nv">$2</span> ~ /^10.<span class="nv">$/</span> <span class="c"># regex match</span>
<span class="nv">$2</span> <span class="o">!</span>~ /^10.<span class="nv">$/</span> <span class="c"># regex negated match -- this one might be new</span></code></pre></figure>
<p>and logical operators:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$1</span> ~ /^2015/ <span class="o">&&</span> <span class="nv">$6</span> <span class="o">></span> 20000000 <span class="c"># and -- high volume in 2015</span>
<span class="nv">$6</span> < 1000000 <span class="o">||</span> <span class="nv">$6</span> <span class="o">></span> 20000000 <span class="c"># or -- low or high volume</span>
<span class="o">!</span> /^2015/ <span class="c"># not -- not in 2015</span></code></pre></figure>
<p>You can <em>almost</em> create arbitrarily complex conditions. You are missing variables…</p>
<h2 id="built-in-variables">Built-in Variables</h2>
<p>Some variables just “exist”; they already contain values and are automatically
updated. These variables are easy to recognize because they are named
in CAPITAL letters. Exception: column variables (starting with a $)
are also built-in variables.</p>
<p>There are a <a href="https://www.math.utah.edu/docs/info/gawk_11.html">bunch</a>
of built-in variables, but you’ll mostly use 2:</p>
<ul>
<li>NR : the number of records (lines) processed since AWK started</li>
<li>NF : the number of fields (columns) on the current line</li>
</ul>
<p>And 2 more if you’re dealing with multiple files:</p>
<ul>
<li>FNR : like NR, but resets to 1 when it begins processing a new file</li>
<li>FILENAME: the name of the file being currently processed</li>
</ul>
<h2 id="user-defined-variables">User-Defined Variables</h2>
<p>There is no need to “declare” the variable or initialize it. A variable “comes
to life” when you use it. You can count (and print) how many lines happened in December
2015:</p>
<figure class="highlight"><pre><code class="language-txt" data-lang="txt">$ cat netflix.tsv | awk '/^2015-12/ {count++; print count, $0}'
1 2015-12-31 116.209999 117.459999 114.279999 114.379997 9245000 114.379997
2 2015-12-30 118.949997 119.019997 116.43 116.709999 8116200 116.709999
3 2015-12-29 118.190002 119.599998 116.919998 119.120003 8159200 119.120003
4 2015-12-28 117.260002 117.349998 113.849998 117.110001 8406300 117.110001
5 2015-12-24 118.220001 118.800003 117.300003 117.330002 3531300 117.330002
# snip
22 2015-12-01 124.470001 125.57 122.419998 125.370003 12528800 125.370003</code></pre></figure>
<p>Not having to declare variables is <em>convenient</em>, but it’s also error-prone. If
you misspell a variable, there won’t be any warning, and it might take you a
while to discover your mistake. You’ve been warned. <strong>Remember:</strong> this is a
language that optimizes for one-liners.</p>
<p>What are variables initialized to?</p>
<figure class="highlight"><pre><code class="language-txt" data-lang="txt">$ awk 'BEGIN {print x + 2}' # => 2
$ awk 'BEGIN {x = x + 2; print x}' # => 2
$ awk 'BEGIN {print x}' # => <blank> -- empty string, really
# BEGIN will be discussed next...</code></pre></figure>
<p>An undefined <code class="language-plaintext highlighter-rouge">x</code> contains the empty string. The first time you access
it, that’s what you get. Strings are converted to numbers for numerical
operations:</p>
<figure class="highlight"><pre><code class="language-txt" data-lang="txt">x + 2
# expands:
"" + 2
# expands:
0 + 2
# expands:
2</code></pre></figure>
<h2 id="special-patterns-begin-and-end">Special Patterns: BEGIN and END</h2>
<p><code class="language-plaintext highlighter-rouge">BEGIN</code> and <code class="language-plaintext highlighter-rouge">END</code> are special conditions that only get triggered once per run.</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">BEGIN</code> gets triggered <em>before</em> processing any line</li>
<li><code class="language-plaintext highlighter-rouge">END</code> gets triggered <em>after</em> all lines have been processed</li>
</ul>
<p>These conditions get triggered <em>even if</em> there are no input lines.</p>
<p><code class="language-plaintext highlighter-rouge">BEGIN</code> is usually used to initialize variables – though now you know that’s not
necessary for zeroes or empty strings. It can also be used to print a header.</p>
<p><code class="language-plaintext highlighter-rouge">END</code> is usually used to crunch a result and print a summary or report:</p>
<figure class="highlight"><pre><code class="language-txt" data-lang="txt">$ cat netflix.tsv | awk 'END {print NR}'</code></pre></figure>
<p>(<em>achievement unlocked: you have re-created wc -l…</em>)</p>
<h2 id="blocks-and-control">Blocks and Control</h2>
<p>You can have multiple condition-block pairs. Each line in the input files gets
presented to each block you write:</p>
<figure class="highlight"><pre><code class="language-txt" data-lang="txt">$ cat netflix.tsv | awk '/^2016-03-24/ {print} $4 == 96.43 {print}'
2016-03-24 98.639999 98.849998 97.07 98.360001 10646900 98.360001
2016-03-15 97.870003 98.510002 96.43 97.860001 9678000 97.860001
# could be written as:
#
# $ cat netflix.tsv | awk '/^2016-03-24/; $4 == 96.43'
#
# because we both know what a missing block means...
# but for this example, it's a bit opaque.</code></pre></figure>
<p>That works great until you have a line that matches both conditions:</p>
<figure class="highlight"><pre><code class="language-txt" data-lang="txt">$ cat netflix.tsv | awk '/^2016-03-24/ {print} $4 == 97.07 {print}'
2016-03-24 98.639999 98.849998 97.07 98.360001 10646900 98.360001
2016-03-24 98.639999 98.849998 97.07 98.360001 10646900 98.360001</code></pre></figure>
<p>The same line was printed twice! There are two solutions to this problem:</p>
<ul>
<li>making your conditions mutually exclusive<br />
(which <em>could</em> be easy, but is often tedious and redundant)</li>
<li>using the <code class="language-plaintext highlighter-rouge">next</code> statement:</li>
</ul>
<figure class="highlight"><pre><code class="language-txt" data-lang="txt">$ cat netflix.tsv | awk '/^2016-03-24/ {print; next} $4 == 97.07 {print}'
2016-03-24 98.639999 98.849998 97.07 98.360001 10646900 98.360001</code></pre></figure>
<p>If you hit a <code class="language-plaintext highlighter-rouge">next</code>, your script will stop matching blocks and go to the
<em>next</em> line from the input file. Using <code class="language-plaintext highlighter-rouge">next</code> means you have to think about the
order of your blocks.<br />
That’s not necessarily a bad thing.</p>
<p>There’s also an <code class="language-plaintext highlighter-rouge">exit</code> statement to stop processing any more input and <em>exit</em> your script.
The <code class="language-plaintext highlighter-rouge">END</code> block will still be executed, if you have one.</p>
<h2 id="taking-inventory-what-can-you-do">Taking inventory: what can you do?</h2>
<p>At this point, a better question would be: what <em>can’t</em> you do?</p>
<p>In review, you can:</p>
<ul>
<li>match a line with regular expressions</li>
<li>match a line with any operator</li>
<li>use built-in variables, both in conditions or in blocks</li>
<li>use your own variables, for all other needs</li>
<li>control what happens at the beginning and the end of your script</li>
<li>skip lines or exit early</li>
</ul>
<h2 id="exercises">Exercises</h2>
<p>Try to:</p>
<ul>
<li>only print lines between February 29, 2016 and March 4, 2016</li>
<li>sum the volumes for all days of January 2016</li>
<li>average the closing price over all days of March 2015</li>
<li>check that <em>all</em> lines have 7 columns</li>
<li>only print every other line (say, even lines)</li>
<li>remove empty lines in a file</li>
</ul>
<p>Answers are <a href="/assets/awk-tutorials/answers-part2.txt">here</a>.</p>
<h2 id="whats-next">What’s next?</h2>
<p><a href="https://blog.jpalardy.com/posts/awk-tutorial-part-3/">Part 3</a></p>
Awk Tutorial, part 12016-04-05T00:00:00+00:00https://blog.jpalardy.com/posts/awk-tutorial-part-1<p>I <a href="https://blog.jpalardy.com/posts/why-learn-awk/">already mentioned</a> why you
should learn AWK.<br />
Let me show you how you can start using it today.</p>
<h2 id="example-data">Example Data</h2>
<p>I think it’s hard to learn AWK in a vacuum. I looked for open data on the web
and picked Netflix historical stock prices. The CSV data is available to download from
<a href="https://finance.yahoo.com/q/hp?s=NFLX+Historical+Prices">Yahoo finance</a> or
<a href="https://www.google.com/finance/historical?q=NASDAQ%3ANFLX">Google finance</a>.
It’s possible to parse this CSV data in AWK, but I replaced commas with TAB
characters to make examples easier. Here is the data we’re going to use:</p>
<figure class="highlight"><pre><code class="language-txt" data-lang="txt">Date Open High Low Close Volume Adj Close
2016-03-24 98.639999 98.849998 97.07 98.360001 10646900 98.360001
2016-03-23 99.75 100.389999 98.809998 99.589996 8292300 99.589996
2016-03-22 100.480003 101.519997 99.199997 99.839996 9039500 99.839996
2016-03-21 101.150002 102.099998 99.50 101.059998 9562900 101.059998
2016-03-18 100.50 102.410004 100.010002 101.120003 15437300 101.120003
...</code></pre></figure>
<p>Download <a href="/assets/awk-tutorials/netflix.tsv">the data</a> if you want to try examples yourself.</p>
<h2 id="printing-columns">Printing Columns</h2>
<p>Printing columns is probably the most useful things you can do in AWK:</p>
<figure class="highlight"><pre><code class="language-txt" data-lang="txt">$ cat netflix.tsv | awk '{print $2}'
Open
98.639999
99.75
100.480003
101.150002
100.50
# snip</code></pre></figure>
<p>Let’s take it one step at a time:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">cat netflix.tsv | awk</code> to send netflix.tsv to the STDIN of AWK</li>
</ul>
<p>Alternatively, <code class="language-plaintext highlighter-rouge">awk '{print $2}' netflix.tsv</code> would have given us the same
result. For this tutorial, I use <code class="language-plaintext highlighter-rouge">cat</code> to visually separate the input data from
the AWK program itself. Using <code class="language-plaintext highlighter-rouge">cat</code> also emphasizes that AWK can treat any input and
not just existing files.</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">{print $2}</code> to print the 2nd column</li>
</ul>
<p>Yes, you need the curly brackets – I’ll come to that shortly. You already
guessed it: column 1 is $1, column 2 is $2, column 7 is $7, etc…</p>
<ul>
<li><code class="language-plaintext highlighter-rouge"># snip</code> to indicate omitted output</li>
</ul>
<p>There are 3485 lines in the data file. For most examples, I’ll truncate the
output because more isn’t always better.</p>
<h2 id="always-use-single-quotes-with-awk">Always Use Single-Quotes with AWK</h2>
<p>Let’s get this out of the way: always use single-quotes with AWK.</p>
<p>As you’ve seen above, column names have dollar signs in them ($1, $2, $7…)
which would usually be substituted by BASH. Single-quotes are how you tell
BASH to keep the content of your strings untouched. Double-quotes won’t work,
and backslash escapes <em>might</em> work but are not worth fighting for.</p>
<p>Let’s keep things <strong>simple</strong> with single-quotes.</p>
<p>If you need to inject some values into your script, I’ll show you how in a
follow-up tutorial.</p>
<h2 id="whats-with-those-curly-brackets--">What’s With Those Curly Brackets? <code class="language-plaintext highlighter-rouge">{ }</code></h2>
<p>What’s the difference between:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">awk</span> <span class="s1">'{print $2}'</span>
<span class="nb">awk</span> <span class="s1">'print $2'</span></code></pre></figure>
<p>Answer: one works and the other doesn’t! (<a href="https://instantrimshot.com/">rimshot</a>)
We’ll need to take a step back to explain the difference. In AWK, a program is
composed of <em>rules</em> which look like:</p>
<figure class="highlight"><pre><code class="language-awk" data-lang="awk"><span class="nx">some</span><span class="o">-</span><span class="nx">condition</span> <span class="p">{</span> <span class="nx">one</span> <span class="nx">or</span> <span class="nx">many</span> <span class="nx">statements</span> <span class="p">}</span></code></pre></figure>
<p>If it were C code:</p>
<figure class="highlight"><pre><code class="language-c" data-lang="c"><span class="k">if</span> <span class="p">(</span><span class="n">some</span><span class="o">-</span><span class="n">condition</span><span class="p">)</span> <span class="p">{</span> <span class="n">one</span> <span class="n">or</span> <span class="n">many</span> <span class="n">statements</span><span class="p">;</span> <span class="p">}</span></code></pre></figure>
<p>In short, the curly brackets (<code class="language-plaintext highlighter-rouge">{ }</code>) tell AWK to <em>do</em> something. AWK allows
either the condition or the action to be missing.</p>
<h2 id="what-does-it-mean-when-the-condition-is-missing">What does it mean when the condition is missing?</h2>
<p>A missing condition defaults to “always run”:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">awk</span> <span class="s1">'{print $2}'</span>
<span class="c"># means:</span>
<span class="nb">awk</span> <span class="s1">'1 {print $2}'</span> <span class="c"># 0 is false, any other value is true</span></code></pre></figure>
<p>if true, print the 2nd column.</p>
<h2 id="what-does-it-mean-when-the-action-is-missing">What does it mean when the action is missing?</h2>
<p>A missing action defaults to “print”:</p>
<figure class="highlight"><pre><code class="language-txt" data-lang="txt">$ cat netflix.tsv | awk '$2 > 100'
Date Open High Low Close Volume Adj Close
2016-03-22 100.480003 101.519997 99.199997 99.839996 9039500 99.839996
2016-03-21 101.150002 102.099998 99.50 101.059998 9562900 101.059998
2016-03-18 100.50 102.410004 100.010002 101.120003 15437300 101.120003
2016-03-07 101.00 101.790001 95.25 95.489998 23855200 95.489998
2016-01-22 104.720001 104.989998 99.220001 100.720001 26772700 100.720001
# snip -- output has been reformated to align</code></pre></figure>
<p>A missing block prints the whole matching line.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">awk</span> <span class="s1">'$2 > 100'</span>
<span class="c"># means:</span>
<span class="nb">awk</span> <span class="s1">'$2 > 100 { print }'</span>
<span class="c"># means:</span>
<span class="nb">awk</span> <span class="s1">'$2 > 100 { print $0 }'</span></code></pre></figure>
<p><code class="language-plaintext highlighter-rouge">$0</code> is a special variable that contains the current line, before it was
separated into fields. <code class="language-plaintext highlighter-rouge">print $0</code> means “print the current line”. <code class="language-plaintext highlighter-rouge">print</code>, by
itself, also prints the current line.</p>
<h2 id="more-printing">More Printing</h2>
<p>You know how to print one column, but what if you need to print many?</p>
<figure class="highlight"><pre><code class="language-txt" data-lang="txt">$ cat netflix.tsv | awk '{print $1, $6, $5}'
Date Volume Close
2016-03-24 10646900 98.360001
2016-03-23 8292300 99.589996
2016-03-22 9039500 99.839996
2016-03-21 9562900 101.059998
2016-03-18 15437300 101.120003
# snip -- output has been reformated to align</code></pre></figure>
<p>A comma between print values will insert a space in the output. AWK also has
<code class="language-plaintext highlighter-rouge">printf</code> which unleashes infinite <a href="https://linux.die.net/man/3/printf">formatting power</a>:</p>
<figure class="highlight"><pre><code class="language-txt" data-lang="txt">$ cat netflix.tsv | awk '{printf "%s %15s %.1f\n", $1, $6, $5}' | sed 1d
2016-03-24 10646900 98.4
2016-03-23 8292300 99.6
2016-03-22 9039500 99.8
2016-03-21 9562900 101.1
2016-03-18 15437300 101.1
# snip</code></pre></figure>
<p>I removed the header line, which had been mangled in the printf.</p>
<p>AWK does string concatenation without an operator: just put 2 values next to
each other. This is useful when you don’t want to reach for printf but still want
some formatting flexibility:</p>
<figure class="highlight"><pre><code class="language-txt" data-lang="txt">$ cat netflix.tsv | awk '{print $1 "," $6}'
Date,Volume
2016-03-24,10646900
2016-03-23,8292300
2016-03-22,9039500
2016-03-21,9562900
2016-03-18,15437300
# snip</code></pre></figure>
<p>Ooooh, we’re back to CSV.</p>
<h2 id="taking-inventory-what-can-you-do">Taking inventory: what can you do?</h2>
<p>This is just the beginning, and there’s more to cover. But you now have a solid
foundation: you know about conditions and actions, columns and printing. You can:</p>
<ul>
<li>print only the columns you want</li>
<li>print them in the order you want</li>
<li>format with all the power of printf</li>
<li>use conditions to print only lines you want</li>
</ul>
<h2 id="exercises">Exercises</h2>
<p>Try to:</p>
<ul>
<li>only print the ‘Date’, ‘Volume’, ‘Open’, ‘Close’ columns, in that order</li>
<li>only print lines where the stock price increased (‘Close’ > ‘Open’)</li>
<li>print the ‘Date’ column and the stock price difference (‘Close’ - ‘Open’)</li>
<li>print an empty line between each line – double-space the file</li>
</ul>
<p>Answers are <a href="/assets/awk-tutorials/answers-part1.txt">here</a>.</p>
<h2 id="whats-next">What’s next?</h2>
<p><a href="https://blog.jpalardy.com/posts/awk-tutorial-part-2/">Part 2</a></p>
Why Learn AWK?2016-03-16T00:00:00+00:00https://blog.jpalardy.com/posts/why-learn-awk<p>Because of the arcane syntax?<br />
Because other languages can’t do the job?</p>
<p>No.</p>
<p>I resisted AWK for a long time. Couldn’t I already do everything I needed with
sed and grep? I felt that anything more complex should be done with a “real”
language. AWK seemed like yet-another-thing to learn, with marginal benefits.</p>
<h2 id="why-learn-awk">Why Learn AWK?</h2>
<p>Let me count the ways.</p>
<h3 id="you-are-working-too-hard">You are working TOO HARD</h3>
<p>Too many times, I’ve seen people working <em>way</em> too hard at the command-line trying to
solve simple tasks.</p>
<p>Imagine programming without regular expressions.</p>
<p>Can you even <em>imagine</em> the alternative? Would it entail building <a href="https://en.wikipedia.org/wiki/Finite-state_machine">FSM</a>s
from scratch? Would it be easy to program? Would it be fun? Would it work the way you want?</p>
<p>That’s life without AWK.</p>
<p>For simple tasks (“only print column 3” or “sum the numbers from column 2”)
<em>almost</em> falling in the “grep-and-sed” category, but where you feel you might
need to open a man page, AWK is usually the solution.</p>
<p>And if you think that creating a new script file (<code class="language-plaintext highlighter-rouge">column_3.py</code> or <code class="language-plaintext highlighter-rouge">sum_col_2.rb</code>),
putting it somewhere on disk, and invoking it on your data isn’t bad – I’m
telling you that you’re working too hard.</p>
<h3 id="available-everywhere">Available EVERYWHERE</h3>
<p>On Linux, BSD, or Mac OS, AWK is already available. It is <a href="https://en.wikipedia.org/wiki/POSIX#Overview">required</a> by any
POSIX-compliant OS.</p>
<p>More importantly, it will be the AWK you know. It has been around for a long
time, and the way it works is stable. Any upgrade would not (could not) break
your scripts – it’s the closest thing to “it just works”.</p>
<p>Contrast with BASH or Python … do you have the right version? Does it have
<em>all</em> the language features you need? Is it backward and forward compatible?</p>
<p>When I write a script in AWK, I know 2 things:</p>
<ul>
<li>AWK is going to be anywhere I deploy</li>
<li>it’s going to work</li>
</ul>
<h3 id="scope">Scope</h3>
<p>You shouldn’t write anything complicated in AWK. That’s a <strong>feature</strong> – it
limits what you’re going to attempt with the language. You are not going to
write a web server in AWK, and you know it wouldn’t be a good idea.</p>
<p>There’s something refreshing about knowing that you’re not going to import a library
(let alone a framework), and worry about dependencies.</p>
<p>You’re writing an AWK script, and you’re going to focus on what AWK is good at.</p>
<h3 id="language-features">Language Features</h3>
<p>Do you want the following? (especially compared to BASH)</p>
<ul>
<li>hashes</li>
<li>floating-point numbers</li>
<li>modern (i.e. Perl) regular expressions</li>
</ul>
<p>It’s all there, ready to go. Don’t worry about the version number, the
bolted-on syntax, or the dependence on other tools.</p>
<h3 id="convenience-minimized-bureaucracy">Convenience: minimized bureaucracy</h3>
<p>In a script sandwich, your logic is the “meat”, and the surrounding bureaucracy
is the “bread”. In practice, bureaucracy means:</p>
<ul>
<li>opening and closing files</li>
<li>iterating over each line of each file</li>
<li>parsing or breaking a line into fields</li>
</ul>
<p>These things are <em>needed</em> but they aren’t what your script is about. AWK takes
care of all that, your code is implicitly surrounded by a loop that’s going to
iterate over every input line.</p>
<p>DISCLAIMER: This isn’t AWK, it’s JavaScript. It might as well be pseudocode.
All code simplified and for illustrative purposes only.</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="c1">// open each file, assign content to "lines"</span>
<span class="nx">lines</span><span class="p">.</span><span class="nx">forEach</span><span class="p">(</span><span class="kd">function</span> <span class="p">(</span><span class="nx">line</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// the code you write goes here</span>
<span class="p">});</span>
<span class="c1">// close all the files</span></code></pre></figure>
<p>AWK is going to break each line into “fields” or “columns” – for many people,
that feature is the main reason to use AWK. By default, AWK breaks a line into
fields based on whitespace (i.e. <code class="language-plaintext highlighter-rouge">/\s+/</code>) and ignores leading or trailing
whitespace.</p>
<p>Also, AWK is automatically going to set a bunch of useful variables for you:</p>
<ul>
<li>NF – how many fields in the current line</li>
<li>NR – what the current line number is</li>
<li>$1, $2, $3 .. $9 .. – the value of each field on the current line</li>
<li>$0 – the content of the current line</li>
<li>and <a href="https://www.math.utah.edu/docs/info/gawk_11.html#SEC108">more</a></li>
</ul>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="c1">// open each file, assign content to "lines"</span>
<span class="kd">var</span> <span class="nx">NR</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="nx">lines</span><span class="p">.</span><span class="nx">forEach</span><span class="p">(</span><span class="kd">function</span> <span class="p">(</span><span class="nx">line</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">NR</span> <span class="o">=</span> <span class="nx">NR</span> <span class="o">+</span> <span class="mi">1</span><span class="p">;</span>
<span class="kd">var</span> <span class="nx">fields</span> <span class="o">=</span> <span class="nx">line</span><span class="p">.</span><span class="nx">trim</span><span class="p">().</span><span class="nx">split</span><span class="p">(</span><span class="sr">/</span><span class="se">\s</span><span class="sr">+/</span><span class="p">);</span>
<span class="kd">var</span> <span class="nx">NF</span> <span class="o">=</span> <span class="nx">fields</span><span class="p">.</span><span class="nx">length</span><span class="p">;</span>
<span class="c1">// the code you write goes here</span>
<span class="p">});</span>
<span class="c1">// close all the files</span></code></pre></figure>
<h3 id="convenience-automatic-conversions">Convenience: automatic conversions</h3>
<p>AWK does automatic string-to-number conversions. That’s something terrible in
“real” programming languages, but very convenient within the scope of the things
you should attempt with AWK.</p>
<h3 id="convenience-automatic-variables">Convenience: automatic variables</h3>
<p>Variables are automatically created when first used; you don’t need to declare variables.</p>
<figure class="highlight"><pre><code class="language-awk" data-lang="awk"><span class="nx">a</span><span class="o">++</span></code></pre></figure>
<p>Let’s unpack it:</p>
<ul>
<li>the variable <code class="language-plaintext highlighter-rouge">a</code> is created</li>
<li>using <code class="language-plaintext highlighter-rouge">++</code> treats it as a number – <code class="language-plaintext highlighter-rouge">a</code> is initialized to 0</li>
<li>the <code class="language-plaintext highlighter-rouge">++</code> operator increments it</li>
</ul>
<p>It’s even more useful with hashes:</p>
<figure class="highlight"><pre><code class="language-awk" data-lang="awk"><span class="nx">things</span><span class="p">[</span><span class="nv">$1</span><span class="p">]</span><span class="o">++</span></code></pre></figure>
<ul>
<li><code class="language-plaintext highlighter-rouge">things</code> is created, as a hash</li>
<li>using dynamic key <code class="language-plaintext highlighter-rouge">$1</code>, a value is initialized to 0 (implicit in <code class="language-plaintext highlighter-rouge">++</code> use)</li>
<li>the <code class="language-plaintext highlighter-rouge">++</code> operator increments it</li>
</ul>
<h3 id="convenience-built-in-functions">Convenience: built-in functions</h3>
<p>AWK has a <a href="https://www.math.utah.edu/docs/info/gawk_13.html#SEC123">bunch</a> of numeric and
string functions at your disposal.</p>
<h2 id="awk-is-perfect">AWK is PERFECT*</h2>
<p>AWK is <em>PERFECT</em> when you use it for what it’s meant to do:</p>
<ul>
<li>very powerful one-liners</li>
<li>(or) short AND portable scripts</li>
</ul>
<h2 id="now-what">Now What?</h2>
<p>Maybe I’ve convinced you to reconsider AWK: good.</p>
<p>How do you learn AWK?</p>
<p>There are many possibilities:</p>
<ul>
<li>reading a bunch of <a href="https://www.google.ca/search?q=awk+examples">examples</a></li>
<li>finding a <a href="https://www.google.ca/search?q=awk+tutorial">tutorial</a></li>
<li>reading the <a href="https://www.gnu.org/software/gawk/manual/gawk.html">manual</a></li>
<li>reading the <a href="https://www.amazon.com/dp/0596000707/">book</a></li>
</ul>
<p>In my <a href="/posts/awk-tutorial-part-1/">next post</a>, I’ll explain everything you need to get you started with AWK.</p>
The Partition Problem2016-01-22T00:00:00+00:00https://blog.jpalardy.com/posts/the-partition-problem<h3 id="update-november-12-2021">Update (November 12, 2021):</h3>
<p>Apparently, I’m not the only person who has ever stumbled into this problem and
wondered if there was a better way.</p>
<p>Check out <a href="https://www.werkema.com/2021/11/01/an-efficient-solution-to-linear-partitioning/">An Efficient Solution to Linear Partitioning</a>
for another take.</p>
<hr />
<p>It started out with a seemingly simple real-life problem:</p>
<blockquote>
<p>Given a playlist of sequential tracks, how do you divide it<br />
in groups of similar durations?</p>
</blockquote>
<p>I wanted to make 30-ish-minute groups; what I need for my daily Spanish practice.</p>
<p><img src="/assets/partition-problem/spanish-lessons.png" alt="spanish lessons" /></p>
<p>It felt like a “solved problem”, I just needed to find some online “app” to do this…</p>
<h2 id="research">Research</h2>
<p>At that point, the last thing I wanted was to code this myself. Instead, I was
willing to put in the time and do my homework.</p>
<p>I spent HOURS on this. To make a <strong>long</strong> story short:</p>
<ul>
<li>it’s a <a href="https://en.wikipedia.org/wiki/Partition_of_a_set">set partition</a> problem</li>
<li>it’s linear, the order matters<br />
<em>no good: listening to lesson 17 before lesson 2</em></li>
<li>it’s discrete, the items cannot be subdivided<br />
<em>no good: stopping in the middle of a lesson</em></li>
<li>it’s similar to the <a href="https://en.wikipedia.org/wiki/Partition_problem">partition problem</a></li>
<li>it’s similar to the <a href="https://en.wikipedia.org/wiki/Subset_sum_problem">subset sum problem</a></li>
<li>there were MANY almost-but-not-quite problems and algorithms</li>
</ul>
<p>The closest thing I found was the <a href="https://www8.cs.umu.se/kurser/TDBAfl/VT06/algorithms/BOOK/BOOK2/NODE45.HTM">linear partition</a> from <a href="https://www.amazon.com/dp/1848000693">Skiena’s book</a> though I found the explanation a bit opaque.
He describes the problem as:</p>
<blockquote>
<p>input: a given arrangement S of non-negative numbers and an integer k</p>
<p>output: partition S into k ranges, so as to minimize the maximum sum over all the ranges</p>
</blockquote>
<h2 id="coding-it-after-all">Coding it, after all</h2>
<p>I never found a piece of software to which I could provide a list of numbers
and how many parts I want. I decided to code it and try a few approaches.
Here’s the <a href="https://github.com/jpalardy/partition-problem/blob/master/tracks-58.delim">data</a>
I’m working with and here’s what it looks like:</p>
<p><img src="/assets/partition-problem/track-durations-barchart.png" alt="track duration barchart" /></p>
<p>where the boxplot of the track durations looks like:</p>
<p><img src="/assets/partition-problem/track-durations-boxplot.png" alt="track duration barchart" /></p>
<p>more details:</p>
<ul>
<li>58 tracks</li>
<li>range: 124-520s (2:04-8:40)</li>
<li>mean: 290s (4:50)</li>
<li>median: 281s (4:41)</li>
<li>total duration: 16805s (4:40:05)</li>
<li>no outliers</li>
</ul>
<h2 id="methodology">Methodology</h2>
<p>The objective is to divide the tracks into groups that are “fair”. There
shouldn’t be a “big group” or a “small group”, they should all be roughly
equal. We want to minimize the spread of the group durations. The <strong>standard
deviation</strong> is a good measure of spread, it’s also familiar and easy to
calculate.</p>
<p>I <a href="https://stats.stackexchange.com/questions/50277/minimize-the-standard-deviation-of-the-total-values-of-groups-of-items-optimiza">came across the idea</a> that minimizing the standard deviation really meant
minimizing the sum of squares. Although correct and easier to
calculate, I found that it was almost more abstract and in units I found hard
to relate to. Consequently, the standard deviation is what I’m going to use to
describe the “goodness” of my solutions.</p>
<p>For the sake of comparison, I’ll try to split the tracks into 10 groups – 9
groups made the average group duration over 30m and I’m still aiming for around
30m. I’ll come back to this point when I break into different number of groups
later.</p>
<h2 id="simplest-approach-same-number-of-tracks-in-each-group">Simplest approach: same number of tracks in each group</h2>
<p>Relevant code: <a href="https://github.com/jpalardy/partition-problem/blob/master/count.R">count.R</a></p>
<p>This is not <em>completely</em> unreasonable, for tracks with similar durations, you
might get good results by making groups with the same number of tracks. On
the other hand, it doesn’t even look at the track durations, so it’s not using
all the information available.</p>
<p><img src="/assets/partition-problem/count.png" alt="track breakdown by groups" /></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> sd: 304.5 -- range: 1040-2035s (17:20-33:55)
</code></pre></div></div>
<p>Most groups have 6 tracks, some have 5 – that’s what you’d expect from
breaking 58 tracks into 10. A lot of long tracks at the beginning and small
tracks at the end give us a lot of variation between groups.</p>
<p>Let’s call this our baseline – clearly, we can do better.</p>
<h2 id="greedy-approach-accumulate-into-a-group-until-a-maximum">Greedy approach: accumulate into a group until a maximum</h2>
<p>Relevant code: <a href="https://github.com/jpalardy/partition-problem/blob/master/cutoff.R">cutoff.R</a></p>
<p>When I started thinking about the problem, it was the first solution that
came to mind. Accumulate tracks in a group until <em>around</em> 30 minutes, then
start a new group.</p>
<p>Using a cutoff of 1680.5 (total duration divided into 10 groups):</p>
<p><img src="/assets/partition-problem/cutoff.png" alt="track breakdown by groups" /></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> sd: 406.0 -- range: 148-1672s (2:28-27:52)
</code></pre></div></div>
<p>We can see many problems:</p>
<ul>
<li>we ended up with 12 groups – but maybe that’s not a deal-breaker<br />
<em>the choice of 10 groups was arbitrary anyway</em></li>
<li>last group is extreme, stuck with one leftover track</li>
<li>on the plus side, the other groups at relatively even</li>
</ul>
<p>This approach doesn’t rely on the number of groups, it has a specific cutoff to
determine group membership. It’s possible to change the cutoff until you get
a certain number of groups, or a lower standard deviation:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> cutoff buckets sd
1500 13 288.9883
1550 12 131.9025
1600 12 131.9025
1650 12 266.5167
1700 12 406.5718
1750 11 265.3115
1800 11 270.0163
1850 11 475.3502
1900 10 245.7068
1950 10 399.2165
2000 9 145.1454
2050 9 145.1454
2100 9 145.1454
2150 9 336.371
2200 9 428.1255
2250 8 149.2667
2300 8 149.2667
2350 8 434.7446
2400 8 535.3405
2450 8 535.3405
2500 8 695.6937
</code></pre></div></div>
<p>The “best” attempt falls at cutoff is 2000:</p>
<p><img src="/assets/partition-problem/cutoff-best.png" alt="track breakdown by groups" /></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> sd: 145.1 -- range: 1513-2000s (25:13-33:20)
</code></pre></div></div>
<p>It’s clear from that data above that the exact value of the cutoff has a big
impact on the end result. If you’re willing to try multiple values looking for
improvements, this algorithm might be enough for you.</p>
<p>But keep in mind that this algorithm makes no effort in looking at neighbor
solutions which might be big improvements. For example, you could “even out”
the last group by stealing some tracks from its neighbor. Then, you can repeat
until … all groups level out.</p>
<h2 id="minizinc-letting-the-solver-do-its-thing">Minizinc: letting the solver do its thing</h2>
<p>Relevant code: <a href="https://github.com/jpalardy/partition-problem/blob/master/minizinc/partition.mzn">partition.mzn</a></p>
<p>Although I enjoyed the <a href="https://www.coursera.org/course/modelingoptimization">Modeling Discrete Optimization</a> class,
I wasn’t sure that I would find a practical use for what I had learned. But <a href="https://www.minizinc.org/">Minizinc</a>
is designed to solve problems like this one – it is a language to describe constraints. You
don’t have to write the search logic, that’s handled by a solver.</p>
<p>Above, I mentioned that Skiena’s algorithm was looking to “minimize the maximum
sum over all the ranges”. I ran that and found:</p>
<p><img src="/assets/partition-problem/minizinc-max.png" alt="track breakdown by groups" /></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> sd: 245.2 -- range: 1040-1892s (17:20-31:32)
</code></pre></div></div>
<p>It’s pushing down on top values until they level out. It’s obvious looking at group 10 that the solver didn’t care
about the group values below the lowest max. We can immediately do better if we try to minimize <code class="language-plaintext highlighter-rouge">max - min</code> which
will push down the max AND pull up the min:</p>
<p><img src="/assets/partition-problem/minizinc-diff.png" alt="track breakdown by groups" /></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> sd: 132.5 -- range: 1488-1892s (24:48-31:32)
</code></pre></div></div>
<p>The solver converges quickly. Even better, we didn’t have to write any
complicated logic – the solver did all the work. But we know that the solver
didn’t investigate the groups between the min and max. After all, all the
values between min and max don’t change min and max. Depending on your data,
you might land on the best answer – or tweaking the mid-groups might improve
the standard deviation.</p>
<p>It would have been best to try to minimize the “sum of squares” (or the
standard deviation) directly, but the non-linear nature of that problem
prevented the solver from doing a good job. The solver got lost, and I couldn’t
find a way to guide it with my current Minizinc skills.</p>
<h2 id="the-water-flow-way-inspired-by-nature">The water flow way: inspired by nature</h2>
<p>Relevant code: <a href="https://github.com/jpalardy/partition-problem/blob/master/water-flow.R">water-flow.R</a></p>
<p>If you imagine all the groups as containers filled with water and connected by
pipes, you would only have to open the valves for all the containers to find
their level. In the discrete case, it means that a whole track flows from one
container to the next.</p>
<p>I decided to experiment with a simple algorithm:</p>
<ol>
<li>start from some initial state</li>
<li>pick the biggest group</li>
<li>send a track to one of its neighbor group</li>
<li>goto 2</li>
</ol>
<p>It’s not exactly how water flows, but it’s inspired by it. Always picking the
biggest group bypasses the case where I could pick a group smaller than both
its neighbors. I can code this function, call <code class="language-plaintext highlighter-rouge">optim</code> in R in <a href="https://en.wikipedia.org/wiki/Simulated_annealing">simulated annealing</a> (“sann”)
mode to keep repeating the process.</p>
<p><img src="/assets/partition-problem/water-flow.png" alt="track breakdown by groups" /></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> sd: 120.9 -- range: 1488-1892s (24:48-31:32)
</code></pre></div></div>
<p>The range didn’t change compared to the Minizinc version because the min and
max didn’t change. However, the algorithm did a better job at smoothing out
group 8, 9 and 10.</p>
<p>I was surprised how well this simple algorithm worked in practice. I had
expected it to get bogged down based on initial conditions, but that didn’t
happen even with extreme initial conditions. This might be dependent on the
exact data – your mileage may vary. Maybe it’s best to start the algorithm
with something “almost right”, something like the output of one of the other
method above.</p>
<h2 id="durations-and-number-of-groups">Durations and Number of Groups</h2>
<p>Now that we have a workable algorithm to reduce the spread of the groups, we
can come back to question of how many groups would be ideal.</p>
<p><img src="/assets/partition-problem/scatter-per-k.png" alt="durations per number of groups" /></p>
<p>To stay under 30m (1800s), I would divide the tracks into 11 groups.</p>
<p><img src="/assets/partition-problem/water-flow-11.png" alt="track breakdown by groups" /></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> sd: 109.7 -- range: 1348-1740s (22:28-29:00)
</code></pre></div></div>
<p>This is using the water-flow code. Meanwhile, if you’re curious, the other
methods for 11 groups:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> count: sd = 304.5
cutoff: sd = 265.3
minizinc: sd = 137.1
</code></pre></div></div>
<h2 id="conclusion">Conclusion</h2>
<p>Simple problems sometimes have complicated solutions :-)</p>
<p>I don’t intend this to be the final word on the problem. In fact, part of me
hopes that I missed something.</p>
<p>Let me know on twitter or by email.</p>
The Best Books I Read in 20152015-12-14T00:00:00+00:00https://blog.jpalardy.com/posts/best-books-i-read-2015<p>I have read many books in 2015, but it wasn’t all good. Some books, however,
deserve to be acknowledged. I broke down my recommendations by categories:
<a href="#technical">technical</a>, <a href="#non-fiction">non-fiction</a> and <a href="#fiction">fiction</a>.
There is no particular order.</p>
<hr />
<h2 id="technical">Technical</h2>
<h3 id="an-introduction-to-statistical-learning">An Introduction to Statistical Learning</h3>
<p><a href="https://www.amazon.com/dp/1461471370/"><img class="book-cover" src="/assets/best-books-2015/1461471370.jpg" alt="An Introduction to Statistical Learning" /></a></p>
<p>This book wasn’t planned. Last January, I heard about the <a href="https://lagunita.stanford.edu/courses/HumanitiesandScience/StatLearning/Winter2015/about">MOOC</a>,
and saw that people were excited about it. I started the class, bought the
book, and I learned <em>many</em> things I didn’t know … but I was an ML beginner.</p>
<p>Eventually, I took the <a href="https://www.coursera.org/learn/machine-learning">Machine Learning MOOC on Coursera</a>, and it reinforced everything I had learned in Statistical Learning.</p>
<p>I felt that I had read the book too fast the first time around. When I got back
to this book, in October, I re-read it cover-to-cover and found that it was now
a “review” and “obvious” – a stark departure from where I had been at the
beginning of the year.</p>
<p>The <a href="https://lagunita.stanford.edu/courses/HumanitiesSciences/StatLearning/Winter2016/about">Statistical Learning MOOC</a> is being given again in January 2016. I strongly recommend it.</p>
<ul>
<li>no wikipedia link…</li>
<li><a href="https://www.amazon.com/dp/1461471370">amazon</a></li>
</ul>
<hr />
<h2 id="non-fiction">Non-Fiction</h2>
<h3 id="the-case-for-mars">The Case For Mars</h3>
<p><a href="https://www.amazon.com/dp/145160811X/"><img class="book-cover" src="/assets/best-books-2015/145160811X.jpg" alt="The Case For Mars" /></a></p>
<p>Many other books on Mars, both fiction and non-fiction, refer to this book
first published in 1996. This book will sell you why we must go to Mars, how
we can do it, and how to keep the costs down – reasonable even.</p>
<p>If you liked <a href="https://en.wikipedia.org/wiki/The_Martian_(Weir_novel\)">The Martian</a> and would like to:</p>
<ul>
<li>see how much it has been influenced by this book</li>
<li>get even deeper into the details</li>
</ul>
<p>you will be pleased.</p>
<p>Does our generation need a <a href="https://www.youtube.com/watch?v=g25G1M4EXrQ&feature=youtu.be&t=103">BIG project</a>? In that case, let’s fix global warming. And let’s go to Mars – let’s do both.</p>
<ul>
<li><a href="https://en.wikipedia.org/wiki/The_Case_for_Mars">wikipedia</a></li>
<li><a href="https://www.amazon.com/dp/145160811X">amazon</a></li>
</ul>
<h3 id="the-checklist-manifesto">The Checklist Manifesto</h3>
<p><a href="https://www.amazon.com/dp/0312430000/"><img class="book-cover" src="/assets/best-books-2015/0312430000.jpg" alt="The Checklist Manifesto" /></a></p>
<p>From airline pilots to surgeons, checklists save lives. It’s a short book that
could have been shorter, but still an enjoyable read.</p>
<p>Anywhere where many steps are necessary and skipping ahead can have
consequences, checklists can help. The book covers many examples, through
stories, and tries to convince that “we don’t need a checklist” is a
(demonstratively) false and reckless idea.</p>
<ul>
<li><a href="https://en.wikipedia.org/wiki/The_Checklist_Manifesto">wikipedia</a></li>
<li><a href="https://www.amazon.com/dp/0312430000">amazon</a></li>
</ul>
<h3 id="superintelligence">Superintelligence</h3>
<p><a href="https://www.amazon.com/dp/0199678111/"><img class="book-cover" src="/assets/best-books-2015/0199678111.jpg" alt="Superintelligence" /></a></p>
<p>This was a recommendation from <a href="https://www.hellointernet.fm/podcast/52">CGP Grey</a>, and I started out as a
skeptic. It sounds like Skynet, and I’ve already watched that movie. (and others worse)</p>
<p>However, most people in the field believe that we will be able to design a
machine with human-level intelligence – the real question is one of timeline:
in 10 years, 50 years, 100 years? What’s going to happen after that? Probably
nothing good.</p>
<p>This book says that superintelligence <em>might</em> happen, and covers:</p>
<ul>
<li>the different types of possible superintelligence</li>
<li>what a superintelligence might be capable of</li>
<li>different outcomes and scenarios</li>
<li>how to control a superintelligence, if possible</li>
</ul>
<p>The newyorker has an <a href="https://www.newyorker.com/magazine/2015/11/23/doomsday-invention-artificial-intelligence-nick-bostrom">introductory article</a> about it.</p>
<p>I went through the whole book thinking that there was a good idea for a
science-fiction novel every 2-3 pages. But it also made me realize that
I had always assumed that it was impossible, or not something to worry about.
I recommend the book, <em>especially</em> if you disagree with the premise.</p>
<ul>
<li><a href="https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dangers,_Strategies">wikipedia</a></li>
<li><a href="https://www.amazon.com/dp/0199678111">amazon</a></li>
</ul>
<h3 id="economix">Economix</h3>
<p><a href="https://www.amazon.com/dp/0810988399/"><img class="book-cover" src="/assets/best-books-2015/0810988399.jpg" alt="Economix" /></a></p>
<p>Economic ideas often hide behind jargon and formulas – it doesn’t have to be
like that. Here is a comic book that covers the history of economics and
presents many concepts in a <em>very</em> approachable way.</p>
<ul>
<li>What works and what doesn’t?</li>
<li>What are the myths?</li>
<li>Austerity or government spending?</li>
</ul>
<p>The author has done his homework and brings information from an extensive
bibliography. I wish more subjects could be covered in this format. This
is <a href="https://en.wikipedia.org/wiki/Educational_entertainment">edutainment</a> at
its best.</p>
<ul>
<li>no wikipedia link…</li>
<li><a href="https://www.amazon.com/dp/0810988399">amazon</a></li>
</ul>
<hr />
<h2 id="fiction">Fiction</h2>
<h3 id="revelation-space">Revelation Space</h3>
<p><a href="https://www.amazon.com/dp/0441009425/"><img class="book-cover" src="/assets/best-books-2015/0441009425.jpg" alt="Revelation Space" /></a></p>
<p>Humans have spread to nearby stars, all <em>without</em> <a href="https://en.wikipedia.org/wiki/Faster-than-light">FTL</a> drives, and there are
many mysteries to explore in this dark <a href="https://en.wikipedia.org/wiki/Space_opera">space opera</a>.</p>
<p>I liked the first book because it touched on many interesting ideas – one of
my favorite being the <a href="https://waitbutwhy.com/2014/05/fermi-paradox.html">Fermi paradox</a>.
I kept reading to explore different aspects of this universe, and to see how
things were going to work out.</p>
<ul>
<li><a href="https://en.wikipedia.org/wiki/Revelation_Space">wikipedia</a></li>
<li><a href="https://www.amazon.com/dp/0441009425">amazon</a></li>
</ul>
<h3 id="the-three-body-problem">The Three-Body Problem</h3>
<p><a href="https://www.amazon.com/dp/0765377063/"><img class="book-cover" src="/assets/best-books-2015/0765377063.jpg" alt="The Three-Body Problem" /></a></p>
<p>I kept hearing about Liu Cixin, <a href="https://www.newyorker.com/books/page-turner/chinas-arthur-c-clarke">China’s Arthur C. Clarke</a>. It doesn’t hurt that the book won the 2015 Hugo award. Science-fiction is often inspired by American culture and descriptions of the book hinted at “something different”.</p>
<p>It’s hard to talk about the book without revealing spoilers. It’s part of a
trilogy, and I read the first two books. Let’s just say it’s well done,
unusual, and that I can’t wait for the third book to be translated into English.</p>
<ul>
<li><a href="https://en.wikipedia.org/wiki/The_Three-Body_Problem">wikipedia</a></li>
<li><a href="https://www.amazon.com/dp/0765377063">amazon</a></li>
</ul>
<h3 id="aurora">Aurora</h3>
<p><a href="https://www.amazon.com/dp/0316098108/"><img class="book-cover" src="/assets/best-books-2015/0316098108.jpg" alt="Aurora" /></a></p>
<p>This book comes from Kim Stanley Robinson, the author of the excellent
<a href="https://en.wikipedia.org/wiki/Mars_trilogy">Mars Trilogy</a>. It also falls under
the <a href="https://en.wikipedia.org/wiki/Hard_science_fiction">hard science fiction</a> category.
The book follows a <a href="https://en.wikipedia.org/wiki/Generation_ship">generation ship</a> on its
way to colonize another star system.</p>
<p>Like any good science fiction, it raises more questions than it answers:</p>
<ul>
<li>Is it ethical to colonize with a generation ship?</li>
<li>From a technical point of view, what are the challenges?</li>
<li>Is it even possible for humans to colonize new worlds?</li>
</ul>
<p>As an introduction, here’s a <a href="https://www.youtube.com/watch?v=3T1-lE5i98M">short video interview</a> with the author.</p>
<p>Mr.Robinson’s style is meandering; there is a lot of reflections, descriptions,
and dialogues. To me, that’s an essential part of what makes his stories rich, deep
and … mournful. Nostalgia is why I keep coming back.</p>
<ul>
<li><a href="https://en.wikipedia.org/wiki/Aurora_(novel\)">wikipedia</a></li>
<li><a href="https://www.amazon.com/dp/0316098108">amazon</a></li>
</ul>
<h2 id="closing-thoughts">Closing Thoughts</h2>
<p>If you have liked this blog post, please write your own. I want to read your book
reviews and recommendations.</p>
Sending HTML to Mailgun with cURL2015-10-25T00:00:00+00:00https://blog.jpalardy.com/posts/sending-html-to-mailgun-with-curl<p>I used to run Postfix and have it send my emails through
<a href="https://www.mailgun.com/">Mailgun</a>, mostly for the reasons explained in
<a href="https://liminality.xyz/the-hostile-email-landscape/">The Hostile Email Landscape</a>.</p>
<p>But Mailgun accepts emails from a POST request, so I thought it would be a good
opportunity to get rid of Postfix on my virtual private server.</p>
<h2 id="mailgun-and-curl">Mailgun and cURL</h2>
<p>On their <a href="https://www.mailgun.com/">main page</a>, Mailgun shows you how to send an email with curl:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">curl <span class="nt">-s</span> <span class="nt">--user</span> <span class="s2">"api:</span><span class="nv">$API</span><span class="s2">-KEY"</span> <span class="se">\</span>
https://api.mailgun.net/v3/samples.mailgun.org/messages <span class="se">\</span>
<span class="nt">-F</span> <span class="nv">from</span><span class="o">=</span><span class="s1">'me@example.com'</span> <span class="se">\</span>
<span class="nt">-F</span> <span class="nv">to</span><span class="o">=</span><span class="s1">'you@example.com'</span> <span class="se">\</span>
<span class="nt">-F</span> <span class="nv">subject</span><span class="o">=</span><span class="s1">'Hello'</span> <span class="se">\</span>
<span class="nt">-F</span> <span class="nv">text</span><span class="o">=</span><span class="s1">'how are you?'</span></code></pre></figure>
<p>That works fine, but it’s <strong>overly simplified</strong>. Here’s what I need to know:</p>
<ul>
<li>how to send HTML</li>
<li>how to send multiple lines</li>
<li>how to pipe to curl (have curl read from STDIN)</li>
</ul>
<p>Some answers:</p>
<ul>
<li>use <code class="language-plaintext highlighter-rouge">html=...</code> to send HTML, they have examples in the <a href="https://documentation.mailgun.com/">Mailgun documentation</a>.</li>
<li>in curl, you can replace a value with <code class="language-plaintext highlighter-rouge">@filename</code> to read from file</li>
<li><code class="language-plaintext highlighter-rouge">@-</code> reads from STDIN</li>
</ul>
<p>Based on their <a href="https://documentation.mailgun.com/user_manual.html#sending-via-api">documentation</a>
and putting it all together, I tried this:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">cat </span>some.html | curl <span class="nt">-s</span> <span class="nt">--user</span> <span class="s2">"api:</span><span class="nv">$API</span><span class="s2">-KEY"</span> <span class="se">\</span>
https://api.mailgun.net/v3/samples.mailgun.org/messages <span class="se">\</span>
<span class="nt">-F</span> <span class="nv">from</span><span class="o">=</span><span class="s1">'me@example.com'</span> <span class="se">\</span>
<span class="nt">-F</span> <span class="nv">to</span><span class="o">=</span><span class="s1">'you@example.com'</span> <span class="se">\</span>
<span class="nt">-F</span> <span class="nv">subject</span><span class="o">=</span><span class="s1">'Hello'</span> <span class="se">\</span>
<span class="nt">-F</span> <span class="nv">html</span><span class="o">=</span><span class="s2">"@-"</span></code></pre></figure>
<p>but kept getting this error:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
"message": "'html' parameter is not a string"
}
</code></pre></div></div>
<p>It could have been easy: Mailgun <em>could</em> cover more (realistic) cases in their
documentation. As it stands, even when they send HTML, it just a one-liner.</p>
<h2 id="the-solution">The Solution</h2>
<p>I had never seen an example of this, but you can change <code class="language-plaintext highlighter-rouge">@-</code> to <code class="language-plaintext highlighter-rouge"><-</code>:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">cat </span>some.html | curl <span class="nt">-s</span> <span class="nt">--user</span> <span class="s2">"api:</span><span class="nv">$API</span><span class="s2">-KEY"</span> <span class="se">\</span>
https://api.mailgun.net/v3/samples.mailgun.org/messages <span class="se">\</span>
<span class="nt">-F</span> <span class="nv">from</span><span class="o">=</span><span class="s1">'me@example.com'</span> <span class="se">\</span>
<span class="nt">-F</span> <span class="nv">to</span><span class="o">=</span><span class="s1">'you@example.com'</span> <span class="se">\</span>
<span class="nt">-F</span> <span class="nv">subject</span><span class="o">=</span><span class="s1">'Hello'</span> <span class="se">\</span>
<span class="nt">-F</span> <span class="nv">html</span><span class="o">=</span><span class="s2">"<-"</span></code></pre></figure>
<p>The <code class="language-plaintext highlighter-rouge"><</code> means send the file ‘as text’, not as a file. The curl man page itself:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-F, --form <name=content>
(HTTP) This lets curl emulate a filled-in form in which a user
has pressed the submit button. This causes curl to POST data
using the Content-Type multipart/form-data according to RFC
2388. This enables uploading of binary files etc. To force the
'content' part to be a file, prefix the file name with an @
sign. To just get the content part from a file, prefix the file
name with the symbol <. The difference between @ and < is then
that @ makes a file get attached in the post as a file upload,
while the < makes a text field and just get the contents for
that text field from a file.
</code></pre></div></div>
<p>You can see the difference in a netcat capture of both commands:</p>
<p><img src="/assets/mailgun-and-curl/at_lt.gif" alt="at versus lt" /></p>
<p>Finally, I packaged this curl command in a script to replace both the
postfix and the mailx-bsd packages.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">cat</span> /usr/local/bin/send-with-mailgun <span class="c"># replace variables with your values</span>
<span class="c">#!/bin/bash</span>
curl <span class="nt">--user</span> <span class="s2">"api:</span><span class="nv">$API</span><span class="s2">-KEY"</span> <span class="se">\</span>
https://api.mailgun.net/v3/<span class="k">${</span><span class="nv">MAILBOX</span><span class="k">}</span>/messages <span class="se">\</span>
<span class="nt">-F</span> <span class="nv">html</span><span class="o">=</span><span class="s2">"<-"</span> <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
<span class="o">></span> <span class="nb">cat </span>some.html | send-with-mailgun <span class="nt">-F</span> <span class="nv">subject</span><span class="o">=</span><span class="s2">"Hello"</span> <span class="nt">-F</span> <span class="nv">to</span><span class="o">=</span><span class="s2">"you@example.com"</span> <span class="nt">-F</span> <span class="nv">from</span><span class="o">=</span><span class="s2">"me@example.com"</span></code></pre></figure>
How To Shuffle and Sample on the Command-Line2015-10-16T00:00:00+00:00https://blog.jpalardy.com/posts/how-to-shuffle-and-sample-on-the-command-line<h2 id="shuffling">Shuffling</h2>
<p>How do you shuffle on the command-line? With <code class="language-plaintext highlighter-rouge">shuf</code>:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">seq </span>5 <span class="c"># sequence from 1..5</span>
1
2
3
4
5
<span class="o">></span> <span class="nb">seq </span>5 | <span class="nb">shuf</span> <span class="c"># same sequence, shuffled</span>
4
3
2
1
5</code></pre></figure>
<p>On Linux, you already have <code class="language-plaintext highlighter-rouge">shuf</code>.</p>
<p>On Mac OS X, <code class="language-plaintext highlighter-rouge">brew install coreutils</code> installs shuf as <code class="language-plaintext highlighter-rouge">gshuf</code> (g for GNU), but
I usually alias gshuf to shuf to fix that.</p>
<p>You <em>could</em> use <code class="language-plaintext highlighter-rouge">sort -R</code> / <code class="language-plaintext highlighter-rouge">sort --random-sort</code> as a poor-man shuf. For larger
files, that’s a terrible idea because sort will sort the whole file <em>before</em>
shuffling.</p>
<h2 id="sampling">Sampling</h2>
<p>Sampling is <em>the selection of a subset of individuals from within a statistical population</em> – <a href="https://en.wikipedia.org/wiki/Sampling_(statistics\)">wikipedia</a>.</p>
<p>You have to pass the <code class="language-plaintext highlighter-rouge">-n</code> flag to <code class="language-plaintext highlighter-rouge">shuf</code>:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">seq </span>100 | <span class="nb">shuf</span> <span class="nt">-n</span> 5 <span class="c"># pick 5 from 1..100</span>
78
71
74
4
52</code></pre></figure>
<p>You can allow repeated picks of the same value with the <code class="language-plaintext highlighter-rouge">-r</code> flag:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">seq </span>5 | <span class="nb">shuf</span> <span class="nt">-r</span> <span class="nt">-n</span> 10 <span class="c"># pick 10 from 1..5, with repeats</span>
2
2
1
1
2
5
4
3
3
2
<span class="o">></span> <span class="nb">seq </span>5 | <span class="nb">shuf</span> <span class="nt">-n</span> 10 <span class="c"># pick 10 from 1..5, WITHOUT repeats</span>
2
1
3
5
4</code></pre></figure>
<p>Picking one thing is simply:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">seq </span>100 | <span class="nb">shuf</span> <span class="nt">-n</span> 1 <span class="c"># pick 1 from 1..100</span>
37</code></pre></figure>
<h2 id="why-shuffle-why-sample">Why Shuffle? Why Sample?</h2>
<p>Every time I’m faced with too many things to look at, I don’t trust myself for
picking a representative sample. It’s too easy to say “I’ll pick the first 10”
and to miss a problem that only happened later.</p>
<p>For example, you might have a system that generates files in a directory. There
might be hundreds or thousands of files and you only want to get a feel for
their content.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> find <span class="nb">.</span> <span class="nt">-type</span> f | <span class="nb">shuf</span> <span class="nt">-n</span> 10 <span class="c"># pick 10 files...</span></code></pre></figure>
<p>Or you might want to get a feel for what’s happening in a log file:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">cat</span> /var/log/nginx/access.log | <span class="nb">shuf</span> <span class="nt">-n</span> 100</code></pre></figure>
<p>Depending on your specific situation, it might bring the interesting question
of <a href="https://blog.jpalardy.com/posts/statistics-how-many-would-you-check/">how many things to look at</a>.</p>
Wrapping Command-Line Tools2015-09-29T00:00:00+00:00https://blog.jpalardy.com/posts/wrapping-command-line-tools<p>Let’s say you want to improve <code class="language-plaintext highlighter-rouge">cd</code>:</p>
<ul>
<li>after cd, print the current directory</li>
</ul>
<p><img src="/assets/wrapping-cl-tools/cd1.png" alt="cd with pwd print" /></p>
<ul>
<li>given a filename, cd to the directory containing the file</li>
</ul>
<p><img src="/assets/wrapping-cl-tools/cd2.png" alt="cd to a file" /></p>
<p>Here’s something that won’t work:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">cd</span><span class="o">()</span> <span class="o">{</span>
<span class="c"># check if path is a file, etc</span>
<span class="nb">cd</span> <span class="s2">"</span><span class="nv">$dest</span><span class="s2">"</span>
<span class="c"># print pwd</span>
<span class="o">}</span></code></pre></figure>
<p>The problem is that <code class="language-plaintext highlighter-rouge">cd</code> calls itself recursively. Maybe there’s a way around
this with clever aliases. Maybe you could call your version <code class="language-plaintext highlighter-rouge">mycd</code>… but you’ve
been typing <code class="language-plaintext highlighter-rouge">cd</code> for years, and you’re working against your muscle memory.</p>
<p>Thankfully, there’s a workaround for this specific problem:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">cd</span><span class="o">()</span> <span class="o">{</span>
<span class="c"># check if path is a file, etc</span>
<span class="nb">builtin cd</span> <span class="s2">"</span><span class="nv">$dest</span><span class="s2">"</span>
<span class="c"># print pwd</span>
<span class="o">}</span></code></pre></figure>
<p>(my full <code class="language-plaintext highlighter-rouge">cd</code> code is available <a href="https://github.com/jpalardy/dotfiles/blob/3a12a2ae69336bdf6cb95df1c1338e1c91f48032/bash/commands/cd.bash#L3-L15">in my dotfiles</a>.)</p>
<p>The key part is the <code class="language-plaintext highlighter-rouge">builtin cd</code> invocation which prevents a recursive loop.
<code class="language-plaintext highlighter-rouge">builint cd</code> calls the <em>real</em> <code class="language-plaintext highlighter-rouge">cd</code> from the shell. The same trick works for
commands that aren’t builtin, for example, <code class="language-plaintext highlighter-rouge">command ls</code> would bypass aliases and
functions named <code class="language-plaintext highlighter-rouge">ls</code> and look for something in the PATH.</p>
<p>From the bash man page:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>builtin shell-builtin [arguments]
Execute the specified shell builtin, passing it arguments, and
return its exit status. This is useful when defining a function
whose name is the same as a shell builtin, retaining the func-
tionality of the builtin within the function. The cd builtin is
commonly redefined this way. The return status is false if
shell-builtin is not a shell builtin command.
command [-pVv] command [arg ...]
Run command with args suppressing the normal shell function
lookup. Only builtin commands or commands found in the PATH are
executed. If the -p option is given, the search for command is
performed using a default value for PATH that is guaranteed to
find all of the standard utilities. If either the -V or -v
option is supplied, a description of command is printed. The -v
option causes a single word indicating the command or file name
used to invoke command to be displayed; the -V option produces a
more verbose description. If the -V or -v option is supplied,
the exit status is 0 if command was found, and 1 if not. If
neither option is supplied and an error occurred or command can-
not be found, the exit status is 127. Otherwise, the exit sta-
tus of the command builtin is the exit status of command.
</code></pre></div></div>
<p>I was playing with docker recently, and the documentation said to call
<code class="language-plaintext highlighter-rouge">boot2docker shellinit</code> to set some environment variables <em>before</em> calling the
<code class="language-plaintext highlighter-rouge">docker</code> command. Life is too short for that …</p>
<ul>
<li>I don’t want to remember to call a command</li>
<li>I don’t want to remember what command to call</li>
<li>I don’t want to type that command either</li>
</ul>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">docker<span class="o">()</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">[</span> <span class="nt">-z</span> <span class="s2">"</span><span class="nv">$DOCKER_HOST</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
</span><span class="nb">eval</span> <span class="s2">"</span><span class="si">$(</span>boot2docker shellinit<span class="si">)</span><span class="s2">"</span>
<span class="k">fi
</span><span class="nb">command </span>docker <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
<span class="o">}</span></code></pre></figure>
<p>If <code class="language-plaintext highlighter-rouge">DOCKER_HOST</code> isn’t set, call <code class="language-plaintext highlighter-rouge">boot2docker shellinit</code>, then call <code class="language-plaintext highlighter-rouge">docker</code>
like I asked.</p>
Statistics: How Many Would You Check?2015-04-09T00:00:00+00:00https://blog.jpalardy.com/posts/statistics-how-many-would-you-check<p><a href="http://softdroid.net/statistika-mnogo-li-vy-proveryaete">Russian Translation</a>, thanks to Vlad.<br />
<a href="http://www.opensourceinitiative.net/edu/jpalardy/">Ukrainian Translation</a>, thanks to Sandi.<br />
<a href="https://www.autonvaraosatpro.fi/blogi/2018/04/23/statistika-kui-palju-peaks-siis-kontrollima/">Estonian Translation</a>, thanks to Johanne.<br />
<a href="http://expereb.com/statistics-how-many-would-you-check/">Spanish Translation</a>, thanks to Laura.<br />
<a href="http://jandcconsultants.com/statistiques-combien-voudriez-vous-verifier/">French Translation</a>, thanks to Jean-Etienne.<br />
<a href="https://mobilemall.pk/blog/statistics-how-many-would-you-check-urdu/">Urdu Translation</a>, thanks to Samuel.<br />
<a href="https://www.coupontoaster.co.uk/blog/statistics-how-many-would-you-check-sindhi/">Sindhi Translation</a>, thanks to Samuel.</p>
<p>Imagine this situation:</p>
<blockquote>
<p>You just performed a batch update on millions of users in your database.
There were no error messages and you are confident that everything went well.
But it wouldn’t hurt to check…</p>
<p>How many users would you have to check to feel confident that
everything worked for at <em>least</em> 95% of users?</p>
</blockquote>
<p>Here are some thoughts:</p>
<ul>
<li>if you don’t check, you don’t know: confidence is 0%. After all, maybe your batch update didn’t work at all, but there were no error messages.</li>
<li>if you check ALL, you know the answer: confidence is 100%. But that might be a lot of work…</li>
<li>if you check some users, maybe 10, and the update worked … you can start to feel good. How confident can you be?</li>
</ul>
<p>I don’t think the answer is obvious. I had to take some time to think about it.</p>
<h2 id="a-detour-average-rating">A detour: average rating</h2>
<p>I remembered reading
<a href="https://www.evanmiller.org/how-not-to-sort-by-average-rating.html">How Not To Sort By Average Rating</a>,
and I thought I could apply the same logic to this problem.</p>
<p>If you only have one review, and it’s positive, is that 100%? Intuitively, we
know that it’s not: it’s just one person’s opinion. As more and more people
give positive reviews, we can start feeling better about the accuracy of the
score.</p>
<p>The quote from the article is:</p>
<blockquote>
<p>Given the ratings I have, there is a 95% chance that the “real” fraction of positive ratings is at least what?</p>
</blockquote>
<p>We can use the lower bound of the Wilson confidence interval.</p>
<h2 id="in-practice-with-r">In practice, with R</h2>
<p>The <code class="language-plaintext highlighter-rouge">binom.wilson</code> function, from the <code class="language-plaintext highlighter-rouge">binom</code> package, can be used like this:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="o">></span><span class="w"> </span><span class="n">binom.wilson</span><span class="p">(</span><span class="m">18</span><span class="p">,</span><span class="w"> </span><span class="m">20</span><span class="p">)</span><span class="w">
</span><span class="n">method</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">mean</span><span class="w"> </span><span class="n">lower</span><span class="w"> </span><span class="n">upper</span><span class="w">
</span><span class="m">1</span><span class="w"> </span><span class="n">wilson</span><span class="w"> </span><span class="m">18</span><span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">0.9</span><span class="w"> </span><span class="m">0.6989664</span><span class="w"> </span><span class="m">0.9721335</span></code></pre></figure>
<p>In other words, if we sampled 18 positives and 2 negatives (18/20), the “real”
fraction probably falls between 0.699 and 0.972 (mean: 0.9).</p>
<p>For our example, we could invoke it with 100% success:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="o">></span><span class="w"> </span><span class="n">binom.wilson</span><span class="p">(</span><span class="m">10</span><span class="p">,</span><span class="w"> </span><span class="m">10</span><span class="p">)</span><span class="w">
</span><span class="n">method</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">mean</span><span class="w"> </span><span class="n">lower</span><span class="w"> </span><span class="n">upper</span><span class="w">
</span><span class="m">1</span><span class="w"> </span><span class="n">wilson</span><span class="w"> </span><span class="m">10</span><span class="w"> </span><span class="m">10</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">0.7224672</span><span class="w"> </span><span class="m">1</span></code></pre></figure>
<p>The upper bound isn’t interesting: we’re not interested in the best case
scenario. But if you check 10 and they are all successful, you can feel
confident that it worked for (lower bound) 72.2% of users.</p>
<p>If we keep checking, and we keep finding successes, we can feel more and more
certain about “true” success:</p>
<p><img src="/assets/lowconf_vs_checks/lowconf_vs_checks.png" alt="lower bound success over the number of checks" /></p>
<p>It takes 73 checks to reach a lower bound of 95% of “true” success (the red line).</p>
<h2 id="analysis">Analysis</h2>
<p>Here is the analysis as a <a href="/assets/lowconf_vs_checks/lowconf.Rmd">RMarkdown document</a> and
the resulting output <a href="/assets/lowconf_vs_checks/lowconf.html">html document</a>.</p>
Inverse Globbing2015-03-28T00:00:00+00:00https://blog.jpalardy.com/posts/inverse-globbing<p>It felt like a command-line blind spot: is it possible to invert a glob?</p>
<p>What’s the opposite of <code class="language-plaintext highlighter-rouge">rm *.mp3</code>? How would you delete everything <em>except</em> the
MP3 files?</p>
<p>It turns out that there’s a syntax for that:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">rm</span> <span class="o">!(</span><span class="k">*</span>.mp3<span class="o">)</span></code></pre></figure>
<p>if you put <code class="language-plaintext highlighter-rouge">shopt -s extglob</code> in your .bashrc. There are <a href="https://stackoverflow.com/questions/17191622/why-would-i-not-leave-extglob-enabled-in-bash">no downsides</a>
to using that config. Extended globbing allows you to do <a href="https://mywiki.wooledge.org/glob">other things</a> too.</p>
<p>Without extended globbing, you could try:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">rm</span> <span class="si">$(</span><span class="nb">ls</span> | <span class="nb">grep</span> <span class="nt">-v</span> mp3<span class="nv">$)</span></code></pre></figure>
<p>but that won’t account for filenames with spaces and other characters that
would need to be escaped.</p>
Watch YouTube Faster2015-01-02T00:00:00+00:00https://blog.jpalardy.com/posts/watch-youtube-faster<p><a href="https://medicinskanyheter.com/eric-karlsson/titta-pa-youtube-snabbare.html">Swedish Translation</a>, thanks to Eric.</p>
<p>It seems that YouTube supports faster (and slower) playback since mid-2014, on their
<a href="https://www.youtube.com/html5">new HTML5 player</a>.</p>
<p>Just click the gear on the bottom-right, go to <code class="language-plaintext highlighter-rouge">Speed</code> and experiment:</p>
<p><img src="/assets/youtube-faster/speed.jpg" alt="what/where to click" /></p>
<p>Why is this a good idea? Because I find that watching at 1.5x doesn’t affect my
understanding of the content. Of course, it depends on the video: how
technical or complicated it is, how well edited it is, and the signal-to-noise
ratio.</p>
<p>How much time does it save?</p>
<ul>
<li>1.25x = 0.80 duration = 20% savings</li>
<li>1.5x = 0.67 duration = 33% savings</li>
<li>2x = 0.50 duration = 50% savings</li>
</ul>
<p>For a typical 1-hour conference presentation, at 1.5x, that’s 20 minutes of my
life that I get back.</p>
<p>If you don’t want to click, like me, there’s a keyboard shortcut to
decrease/increase the speed: <code class="language-plaintext highlighter-rouge"><</code> and <code class="language-plaintext highlighter-rouge">></code>, respectively.</p>
How to Have the Last Word (on the command-line)2014-12-12T00:00:00+00:00https://blog.jpalardy.com/posts/how-to-have-the-last-word<p>Have you ever done this?</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">% <span class="nb">mkdir </span>some_dir
% <span class="nb">cd </span>some_dir</code></pre></figure>
<p>Did you type <code class="language-plaintext highlighter-rouge">some_dir</code> twice? Here’s the same task, done in 3 different ways.</p>
<h2 id="worst-doing-it-manually">Worst: doing it manually</h2>
<p><img src="/assets/last-word/manually.gif" alt="doing it manually" /></p>
<p>This is the looooong way. I’m typing everything and pretending I’m on a
typewriter.</p>
<h2 id="better-using-the-history">Better: using the history</h2>
<p><img src="/assets/last-word/history.gif" alt="using the history" /></p>
<p>I’m using <code class="language-plaintext highlighter-rouge">!$</code> (the last word from the previous command) instead of typing it
out. Better still, I’m using ‘magic-space’ to do automatic history expansion
and check what I’m going to get.</p>
<p>More on “magic-space” on <a href="https://www.ukuug.org/events/linux2003/papers/bash_tips/">this page</a>,
scroll to slide 15.</p>
<h2 id="best-using-esc-">Best: using ESC-.</h2>
<p><img src="/assets/last-word/esc-dot.gif" alt="using esc-dot" /></p>
<p>How did the directory’s name just appear? There <em>is</em> a keyboard shortcut for that
and it’s <em>already</em> configured on your terminal: type <code class="language-plaintext highlighter-rouge">ESC</code> followed by the
period (<code class="language-plaintext highlighter-rouge">ESC-.</code>).</p>
<p>(It also works as <code class="language-plaintext highlighter-rouge">META-.</code> or <code class="language-plaintext highlighter-rouge">ALT-.</code>)</p>
<p>If you keep pressing <code class="language-plaintext highlighter-rouge">ESC-.</code>, it will traverse the history backward and insert
the last word from previous commands.</p>
<h2 id="discussion">Discussion</h2>
<p>There are a lot of commands that take a “target” as their last argument:</p>
<ul>
<li>cd</li>
<li>cp / mv / rm</li>
<li>rmdir / mkdir</li>
<li>grep</li>
<li>cat / touch</li>
<li>(any text editor)</li>
</ul>
<p>It’s not uncommon to chain multiple of these commands on the same “target”. You
just created directory, you might very well want to step into it.</p>
<p>Let the command-line help.</p>
type -a2014-10-23T00:00:00+00:00https://blog.jpalardy.com/posts/type-a<p>I <a href="https://blog.jpalardy.com/posts/which-a-versus-whereis/">mentioned previously</a>
that <code class="language-plaintext highlighter-rouge">which -a</code> is what you want to use to find what command you’re about to run.</p>
<p>I was wrong.</p>
<p>I found accidentally, probably while reading something on stackoverflow, that
<code class="language-plaintext highlighter-rouge">type -a</code> is a better-in-every-way replacement for <code class="language-plaintext highlighter-rouge">which -a</code>. Compare:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">% which <span class="nt">-a</span> <span class="nb">ls</span>
/Users/jpalardy/local/bin/ls
/bin/ls
% <span class="nb">type</span> <span class="nt">-a</span> <span class="nb">ls
ls </span>is aliased to <span class="s1">'ls --color=auto'</span>
<span class="nb">ls </span>is /Users/jpalardy/local/bin/ls
<span class="nb">ls </span>is /bin/ls</code></pre></figure>
<p>Not only does <code class="language-plaintext highlighter-rouge">type -a</code> give me all the same information, but it also tells me
about aliases and functions. Consider what happens for functions which do
not have corresponding executables in the PATH:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">% which <span class="nt">-a</span> gril
<span class="c"># -- no output --</span>
% <span class="nb">type</span> <span class="nt">-a</span> gril
gril is a <span class="k">function
</span>gril <span class="o">()</span>
<span class="o">{</span>
<span class="nb">grep</span> <span class="nt">-il</span> <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
<span class="o">}</span></code></pre></figure>
<p>It tells me it’s a function <em>and</em> it shows me its definition. <code class="language-plaintext highlighter-rouge">type -a</code> has decreased,
substantially, the amount of grepping my dotfiles trying to remember how certain
functions work.</p>
One-liner Bash Functions2014-09-16T00:00:00+00:00https://blog.jpalardy.com/posts/one-liner-bash-functions<p>Here’s a problem:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">grep </span>something file</code></pre></figure>
<p>If you’re going to search for different strings, repeatedly, what you really
want is for “something” to be the last argument. This is true even if you
master the
<a href="https://www.howtogeek.com/howto/ubuntu/keyboard-shortcuts-for-bash-command-shell-for-ubuntu-debian-suse-redhat-linux-etc/">keyboard</a>
<a href="https://www.bigsmoke.us/readline/shortcuts">shortcuts</a> to minimize the pain.</p>
<p>While bash doesn’t support anonymous functions, it’s casual about aliases and
named functions. It’s easy to create a function in the current session, and it
will disappear when you exit the shell. Here’s a bash function to solve our
argument problem:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">grep_file<span class="o">()</span> <span class="o">{</span>
<span class="nb">grep</span> <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span> file
<span class="o">}</span></code></pre></figure>
<p>But it’s not convenient to define a bash function like this: as you press enter
between the lines, you can’t go back and edit the top of the function.
You <em>could</em> type the function in a file and source it, but it breaks
the flow of what you’re doing.</p>
<p>If you’ve ever tried to type the function on one line, you’ve seen this situation:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>grep_file<span class="o">()</span> <span class="o">{</span> <span class="nb">grep</span> <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span> file <span class="o">}</span>
<span class="o">></span></code></pre></figure>
<p>You’re stuck on the next line, and bash is waiting for input … as if you had
forgotten to close a string or a parenthesis.</p>
<p>The solution is explained on <a href="https://tldp.org/LDP/abs/html/functions.html">this page</a>:</p>
<p><a href="https://tldp.org/LDP/abs/html/functions.html"><img src="/assets/one-liner-bash-functions/semicolon.png" alt="one line bash function" /></a></p>
<p>It’s all about that semicolon:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>grep_file<span class="o">()</span> <span class="o">{</span> <span class="nb">grep</span> <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span> file<span class="p">;</span> <span class="o">}</span></code></pre></figure>
Warp2014-07-10T00:00:00+00:00https://blog.jpalardy.com/posts/warp<p>I write software for myself. I don’t try to think too hard what’s useful to
other people; I write a lot of things and see what I like. Some
experiments are useless, they seemed like a good idea at the time, but I don’t
end up using them.</p>
<p>That’s fine: <a href="https://www.phrases.org.uk/meanings/226950.html">let a thousand flowers bloom</a>.</p>
<p>One of those experiments was <a href="https://github.com/jpalardy/vim-slime">vim slime</a>.
At the time, it was a quick hack, something neat … I almost didn’t bother; it
seemed “obvious”. It ended up being the most popular <a href="https://technotales.wordpress.com/2007/10/03/like-slime-for-vim/">blog post</a>
I ever wrote, and probably the piece of software I wrote that’s used by the most people.</p>
<p>When I wrote <a href="https://github.com/jpalardy/warp">warp</a>, I felt the same way that
I did about vim slime: quick hack, fun, but no big deal. I just had too many
hosts I could SSH into and no easy way to manage them; I wanted to pick from a
list, something that I could put in git and edit by hand.</p>
<p>It seemed so “obvious”, nothing worth showing off, and I didn’t advertise it.
People just saw me using it and starting to ask “what is that?” and “can I
install that?”.</p>
<p><img src="/assets/warp/warp.gif" alt="warp demo" /></p>
<p>Warp is just one of the many “vim pickers” I made; I have others, of
varying usefulness. They are <a href="https://github.com/jpalardy/dotfiles/blob/bde83a23172d86a9c6ddd73636529a244c5b5ee7/bash/commands/listers.bash">part</a>
of vim dotfiles and involve buying into a lot of my customizations. So, I made an effort
and picked the warp “logic” apart, cleaned it up a bit, and I am now sharing it with the world.</p>
<p>The details are in the <a href="https://github.com/jpalardy/warp">repo</a>.</p>
Unsorted uniq2014-05-30T00:00:00+00:00https://blog.jpalardy.com/posts/unsorted-uniq<p>Everybody gets caught the first time: <code class="language-plaintext highlighter-rouge">uniq</code> filters repeated lines – but only
if they follow each other. This assumption greatly reduces the memory footprint
of <code class="language-plaintext highlighter-rouge">uniq</code> and … its usefulness.</p>
<p>I <a href="https://blog.jpalardy.com/posts/alternative-to-sort-uniq-c/">explained previously</a> how awk
could be used to replace the classic <code class="language-plaintext highlighter-rouge">sort | uniq -c</code> incantation. In short, by
skipping the <code class="language-plaintext highlighter-rouge">sort</code>, you can scale the solution to much bigger files.</p>
<p>If, however, all you need is an unsorted <code class="language-plaintext highlighter-rouge">uniq</code>, there’s an even shorter awk command you can use:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cat animals.txt
cats
cats
cats
dogs
birds
cats
dogs
dogs
birds
dogs
dogs
birds
$ cat animals.txt | awk '!cnts[$0]++'
cats
dogs
birds
</code></pre></div></div>
<p>Broken down as:</p>
<ul>
<li>$0 – the whole current line</li>
<li>cnts[$0] – hash (automatically created) lookup of the current line</li>
<li>!cnts[$0] – if not in the hash (first occurence)</li>
<li>!cnts[$0]++ – if not in the hash, also, increment by one (flagging it as “seen”)</li>
<li>{ print $0 } – implicit action of the conditional above, print the current line</li>
</ul>
<p>Or, in English, print the current line if you’ve never seen it, and mark it seen.</p>
<p>As a bonus, this command doesn’t have to process the whole file; it will
print the new unique lines as they present themselves.</p>
Bookpiles is Kanban for Books2014-04-01T00:00:00+00:00https://blog.jpalardy.com/posts/bookpiles-is-kanban-for-books<p>The <a href="https://www.amazon.com/dp/1453802266/">Personal Kanban</a> book
came with a feeling of déjà vu: I had already been doing all that with my
books! When I wrote <a href="https://bookpiles.ca/">bookpiles.ca</a>, a little web
application to track the books I read, I had no idea I was doing Kanban.</p>
<h2 id="rule-1-visualize-your-work">Rule 1: Visualize Your Work</h2>
<p>Visualization is central to bookpiles, the book covers <em>are</em> the books, and they
move between “piles”. I wanted things I could manipulate, making an
analogy with the physical world.</p>
<p><a href="https://www.amazon.com/dp/1453802266/"><img src="/assets/bookpiles_kanban/pk_cover.png" alt="personal kanban cover" /></a></p>
<p><em>Ceci n’est pas un livre?</em></p>
<p>It would be nice to see the books in different piles at the same time, but
while a Kanban board contains a handful of tasks, <a href="https://bookpiles.ca/jonathan/books">my bookpiles</a>
contains hundreds of books. The books also spend longer in each column, and
forever in “done”.</p>
<p>I was surprised to find that the name and purpose of piles in bookpiles were
very close to the recommended columns in Kanban.</p>
<p>Kanban columns versus piles:</p>
<ul>
<li>backlog: maybe and buy</li>
<li>ready: ready</li>
<li>doing: reading</li>
<li>done: done</li>
<li>pen: stalled</li>
</ul>
<p><img src="/assets/bookpiles_kanban/the_piles.png" alt="the book piles" /></p>
<p>You hear about books and hold them in “maybe” or even “buy”. This is similar to
the concept of the “wishlist” on Amazon, except to express the level of
commitment that you feel.</p>
<p>You select books (presumably buy books) and they become “ready”. At this point,
you have already committed yourself.</p>
<p>You read books, and they hold their position in “reading”. There’s a maximum
number of books that makes sense here, see the next section.</p>
<p>You might get derailed and banish books to “stalled”, the pen.</p>
<p>You, hopefully, retire books to “done”.</p>
<h2 id="rule-2-limit-your-work-in-progress">Rule 2: Limit Your Work-in-Progress</h2>
<p>This is a lesson I learned the hard way, but early on. With my newfound
clarity, I started reading too many books at the same time. The results were
disappointing: slow or uneven progress, losing track of content – I was
juggling too many balls.</p>
<p>Since then, I have tried to limit myself to 1 or 2 concurrent books. I try to do
a “fun” book and an “improvement” book.</p>
<p>I found that audiobooks allow me to add one more book to the WIP. Audiobooks
don’t steal time from regular books; they are a parallel track: I listen to
audiobooks when I commute, wash dishes or do other chores.</p>
<h2 id="results">Results</h2>
<p>Bookpiles has certainly kept me focused. I know what I’m supposed to be
reading, what books are coming up, and it holds a reference to all the books
I’ve heard about that I might want to read someday.</p>
<p>I also <em>love</em> moving the books from one pile to another–especially to “done”.
Then, I can get excited about what I’m going to read next.</p>
Splitting A File Based On Its Content2014-03-11T00:00:00+00:00https://blog.jpalardy.com/posts/splitting-a-file-based-on-its-content<p>My <a href="https://blog.jpalardy.com/posts/alternative-to-sort-uniq-c/">previous post</a>
reminded me of a similar problem: how do you split a file based on its content?
Counting might be one of the things you want to do with each subset of the
file.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cat sample.data
cats
cats
cats
dogs
birds
cats
dogs
dogs
birds
dogs
dogs
birds
$ cat sample.data | awk '{ print > $1 ".data" }'
$ ls
birds.data cats.data dogs.data sample.data
$ wc -l *.data
3 birds.data
4 cats.data
5 dogs.data
12 sample.data
24 total
</code></pre></div></div>
<p>For those not comfortable with <code class="language-plaintext highlighter-rouge">awk</code>:</p>
<ul>
<li>print the whole line (<code class="language-plaintext highlighter-rouge">print $0</code> and <code class="language-plaintext highlighter-rouge">print</code>, by itself, are the same)</li>
<li>redirect (>) to a filename formed by the concatenation of $1 (the first field) and “.data”</li>
</ul>
<p>IMPORTANT: redirection is slightly different in awk than on the shell. Like in
<code class="language-plaintext highlighter-rouge">bash</code>, > means overwrite and » means append, but in <code class="language-plaintext highlighter-rouge">awk</code> the file is only
opened once per session (and closed automatically at the end). That’s why the
files will contain all the matching lines and not only the last matched line. (<a href="https://www.gnu.org/software/gawk/manual/html_node/Redirection.html">details</a>)</p>
<p>Without <code class="language-plaintext highlighter-rouge">awk</code>, I managed to make this work by extracting each unique line (k
uniques) and by iterating over the whole file (n lines) to extract each
subset… it was clumsy and slow: O(k × n).</p>
<p>This is a general technique, but I usually use it to split HTTP server log files by:</p>
<ul>
<li>calendar days (2014-03-11)</li>
<li>hours (15)</li>
<li>HTTP statuses (200, 404, 503 … also: 2xx, 3xx, 4xx, 5xx)</li>
<li>URLs</li>
</ul>
<p>to count requests, review in Vim, and, probably, do some calculations on durations.</p>
<p>It’s possible to do the calculations directly in <code class="language-plaintext highlighter-rouge">awk</code> in one pass, but I find
that splitting the file isn’t wasteful if you’re not sure exactly how you’ll
treat the data – each subset file will be smaller and easier to deal with than
the whole. It can be painful to run a long job on a file and find your logic
faulty. It’s better to do multiple passes, each trivial.</p>
Alternative to sort | uniq -c2014-03-04T00:00:00+00:00https://blog.jpalardy.com/posts/alternative-to-sort-uniq-c<p>It’s an old trick, use <code class="language-plaintext highlighter-rouge">sort | uniq -c</code> to count the unique lines from a file:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cat animals.txt
cats
cats
cats
dogs
birds
cats
dogs
dogs
birds
dogs
dogs
birds
$ cat animals.txt | sort | uniq -c
3 birds
4 cats
5 dogs
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">uniq -c</code> does the counting, but it has to come after a <code class="language-plaintext highlighter-rouge">sort</code>. <code class="language-plaintext highlighter-rouge">sort</code> is
usually fast enough for most day to day usage on the command-line. That was
until recently, when I had to count from a file with 72 million lines… I let
<code class="language-plaintext highlighter-rouge">sort</code> run because I was curious, and it took 38 minutes (!) to process the file.</p>
<p>The “problem”, of course, is sorting. While <code class="language-plaintext highlighter-rouge">sort</code> performs surprisingly well with files
having up to millions of lines, you can’t escape its property: it must use an
<a href="https://en.wikipedia.org/wiki/Sorting_algorithm">n log n</a> algorithm.</p>
<p>Here’s the <code class="language-plaintext highlighter-rouge">awk</code> code equivalent to the <code class="language-plaintext highlighter-rouge">sort | uniq -c</code> above:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cat animals.txt | awk '{ cnts[$0] += 1 } END { for (v in cnts) print cnts[v], v }'
5 dogs
3 birds
4 cats
</code></pre></div></div>
<p>For those not comfortable with <code class="language-plaintext highlighter-rouge">awk</code>:</p>
<ul>
<li>use the whole line ($0) as a key</li>
<li>every occurrence of $0 get += 1 in a hash (cnts)</li>
<li>at the END of the file, iterate over every entry in the hash, print the key and its count</li>
</ul>
<p>I don’t bother optimizing my command-line counting unless I know I’m working
with a big file. Even then, I don’t type the above: I paste it from my notes.</p>
<p>For the record: <code class="language-plaintext highlighter-rouge">awk</code> took only 2 minutes to process that file with 72 million lines.</p>
The bash-completion Problem2014-01-21T00:00:00+00:00https://blog.jpalardy.com/posts/the-bash-completion-problem<p>The <a href="https://github.com/scop/bash-completion">bash-completion</a> package adds
a bunch of useful features, but it also <em>limits</em> what happens when you press <code class="language-plaintext highlighter-rouge">tab</code>.
For example, the “text editor” completion excludes files that probably are not text files:</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">complete -f -X '*.@(o|so|so.!(conf)|a|[rs]pm|gif|jp?(e)g|mp3|mp?(e)g|avi|asf|ogg|class)' vi vim gvim rvim view rview rgvim rgview gview emacs xemacs sxemacs kate kwrite</code></pre></figure>
<p>And the sqlite3 completion tries to outsmart you:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">local </span><span class="nv">dbexts</span><span class="o">=</span><span class="s1">'@(sqlite?(3)|?(s?(3))db)'</span></code></pre></figure>
<p>It looks <strong>only</strong> for files with an extension .sqlite, .sqlite3, .s3db, .sdb, .db.</p>
<p>When <code class="language-plaintext highlighter-rouge">tab</code> doesn’t do what you expect, you can try to read the source. If
you aren’t sure what completions you have, try this:</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">% complete | head
complete -F _kill kill
complete -F _renice renice
complete -F _postconf postconf
complete -F _ldapwhoami ldapwhoami
complete -F _ldapaddmodify ldapadd
complete -F _java java
complete -F _filedir_xspec oodraw
complete -F _filedir_xspec elinks
complete -F _filedir_xspec freeamp
complete -o default -F _longopt split</code></pre></figure>
<h2 id="the-temporary-fix-meta-">The temporary fix: meta-/</h2>
<p>But wait … isn’t bash-completion just trying to <em>help</em>?! Isn’t it doing the
right thing most of the time?</p>
<p>Tab completion is awesome when it works, but it can drive you crazy when it doesn’t.</p>
<p>If you press <code class="language-plaintext highlighter-rouge">tab</code> and nothing happens … try pressing: <code class="language-plaintext highlighter-rouge">meta-/</code>
(try: <code class="language-plaintext highlighter-rouge">ESC-/</code> or <code class="language-plaintext highlighter-rouge">alt-/</code>) to force file completion.</p>
<h2 id="the-permanent-fix-complete--r">The permanent fix: complete -r</h2>
<p>You can remove a specific completion manually:</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">% complete -r sqlite3</code></pre></figure>
<p>or you can integrate it after you <a href="https://github.com/jpalardy/dotfiles/blob/00645a668d3fd59bd13b57c0d1c656bf23691710/bash/completion/bash.bash">source bash-completion</a>: (line 4 and 5)</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><table class="rouge-table"><tbody><tr><td class="gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
</pre></td><td class="code"><pre><span class="k">for </span><span class="nb">dir </span><span class="k">in</span> /usr/local/etc /etc<span class="p">;</span> <span class="k">do
if</span> <span class="o">[</span> <span class="nt">-e</span> <span class="k">${</span><span class="nv">dir</span><span class="k">}</span>/bash_completion <span class="o">]</span><span class="p">;</span> <span class="k">then
</span><span class="nb">source</span> <span class="k">${</span><span class="nv">dir</span><span class="k">}</span>/bash_completion
<span class="nb">complete</span> <span class="nt">-r</span> vim
<span class="nb">complete</span> <span class="nt">-r</span> sqlite3
<span class="nb">break
</span><span class="k">fi
done
if</span> <span class="o">[</span> <span class="nt">-z</span> <span class="s2">"</span><span class="nv">$BASH_COMPLETION</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
</span><span class="nb">echo</span> <span class="s2">"no bash-completion"</span>
<span class="k">fi</span>
</pre></td></tr></tbody></table></code></pre></figure>
<ul>
<li>it looks for bash-completion in the usual places</li>
<li>it sources the first one it finds</li>
<li>it removes completions I didn’t want</li>
<li>it warns you if it couldn’t find it – for example, on a new machine, before you install everything you need</li>
</ul>
<p>In the end, whether you fix the problem short-term or long-term depends on how
annoying this is to you.</p>
which -a versus whereis2014-01-12T00:00:00+00:00https://blog.jpalardy.com/posts/which-a-versus-whereis<h2 id="which--a">which -a</h2>
<p>The command <code class="language-plaintext highlighter-rouge">which</code> is not a revelation, most people are aware of it. You type:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">% which find
/usr/bin/find</code></pre></figure>
<p>and you get the full path of that executable. This is useful if you have
multiple copies of an executable in different places in your PATH and you are
wondering which one would be used.</p>
<p>Sometimes, you want to see ALL the matching executables in your PATH. That’s
what the <code class="language-plaintext highlighter-rouge">-a</code> flag is for:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">% which <span class="nt">-a</span> <span class="nb">awk</span>
/Users/jonathan/local/bin/awk
/usr/local/bin/awk
/usr/bin/awk</code></pre></figure>
<h2 id="whereis">whereis</h2>
<p>I had a vague notion that the <code class="language-plaintext highlighter-rouge">whereis</code> command was used to answer such
questions. Unfortunately, it does something relatively useless:</p>
<blockquote>
<p>The whereis utility checks the standard binary directories for the specified
programs, printing out the paths of any it finds.</p>
</blockquote>
<p>The key point being “standard binary directories”, that is “where stuff is supposed to be”,
that is “I’m not going to check your PATH”… Here is <code class="language-plaintext highlighter-rouge">whereis</code> lying to me:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">% whereis <span class="nb">awk</span>
/usr/bin/awk</code></pre></figure>
<p>I was enlightened by <a href="https://superuser.com/a/430004">this answer</a> on superuser.com.</p>
grep -f2013-12-10T00:00:00+00:00https://blog.jpalardy.com/posts/grep-f<p>This is the kind of thing you don’t need until you <em>really</em> do:</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">-f file, --file=file
Read one or more newline separated patterns from file. Empty
pattern lines match every input line. Newlines are not consid-
ered part of a pattern. If file is empty, nothing is matched.</code></pre></figure>
<p>Here’s a scenario that recently came up:</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">You have a file with millions of entries, one-per-line, in a tab-separated
format. One of the fields (and not necessarily the first one) is the "primary key"
you are using to identify the field.
You ran a batch job, and the logs are telling you about some transient failures.
You grep for the failures and accumulate a bunch of "primary keys". You will need
to rerun the job for those entries.</code></pre></figure>
<p>Essentially, you need to “grep” the original file for the keys that failed. The
problem is that you might have thousands of keys and millions of entries.
Depending on the exact size of the data you are dealing with and the amount of
time available, you might be able to “brute force” the solution. It might look
like this <sup>1</sup>:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">% <span class="nb">cat </span>file_with_keys | <span class="k">while </span><span class="nb">read </span>key<span class="p">;</span> <span class="k">do </span><span class="nb">grep</span> <span class="nv">$key</span> large_file.tsv<span class="p">;</span> <span class="k">done</span> <span class="o">></span> subset.tsv</code></pre></figure>
<p>It spawns 1 grep per key – but it’s a one-liner. Compare with the following,
which accomplishes the same thing with 1 process:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">% <span class="nb">grep</span> <span class="nt">-f</span> file_with_keys large_file.tsv <span class="o">></span> subset.tsv</code></pre></figure>
<p>It is much faster.</p>
<p>Did I miss anything? How would you tackle this?</p>
<hr />
<ol>
<li>Your grep might need qualifiers (-w, for example), but this will depend on your data.</li>
</ol>
Comparing Command-Line JSON Pretty Printers2013-11-07T00:00:00+00:00https://blog.jpalardy.com/posts/comparing-command-line-json-pretty-printers<p>Trying to read badly-formatted JSON is just a waste of time. On the
command-line, or in your text editor with
<a href="http://usevim.com/2012/06/29/vim101-pipe/">filtering</a>, there are tools that
will fix your JSON and make everything right.</p>
<p>There are many options and none are perfect, let’s review our choices:</p>
<p><a href="/assets/jsonpp/grid.png"><img src="/assets/jsonpp/grid.png" alt="JSON tool comparison" /></a></p>
<p>(click to enlarge)</p>
<h2 id="input-files">Input Files</h2>
<p>The worst-case is what I used: a mix of strings, numbers, and booleans without
any human-friendly spaces or alignment.</p>
<p>sample.json:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="p">{</span><span class="dl">"</span><span class="s2">first</span><span class="dl">"</span><span class="p">:</span><span class="dl">"</span><span class="s2">Jimmy</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">last</span><span class="dl">"</span><span class="p">:</span><span class="dl">"</span><span class="s2">James</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">age</span><span class="dl">"</span><span class="p">:</span><span class="mi">29</span><span class="p">,</span><span class="dl">"</span><span class="s2">sex</span><span class="dl">"</span><span class="p">:</span><span class="dl">"</span><span class="s2">M</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">salary</span><span class="dl">"</span><span class="p">:</span><span class="mi">63000</span><span class="p">,</span><span class="dl">"</span><span class="s2">registered</span><span class="dl">"</span><span class="p">:</span><span class="kc">false</span><span class="p">}</span></code></pre></figure>
<p>The ideal output is the following:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="p">{</span>
<span class="dl">"</span><span class="s2">first</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Jimmy</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">last</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">James</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">age</span><span class="dl">"</span><span class="p">:</span> <span class="mi">29</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">sex</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">M</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">salary</span><span class="dl">"</span><span class="p">:</span> <span class="mi">63000</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">registered</span><span class="dl">"</span><span class="p">:</span> <span class="kc">false</span>
<span class="p">}</span></code></pre></figure>
<p>I also thought it would be interesting to see how pretty-printers would react to invalid JSON.
I took sample.json and changed a few things.</p>
<p>invalid.json:</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">{"first":Jimmy","last":"James","age":29,"sex":"M","salary":63000"registered":false}</code></pre></figure>
<p>Can you spot (all) the mistakes? That’s my point – you <strong>don’t</strong> have to – that’s
what computers are for.</p>
<h2 id="discussion-python--m-jsontool">Discussion: python -m json.tool</h2>
<p>Use the json.tool module from python which will pretty-print STDIN by default.</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text"># cat sample.json | python -m json.tool
{
"age": 29,
"first": "Jimmy",
"last": "James",
"registered": false,
"salary": 63000,
"sex": "M"
}</code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text"># cat invalid.json | python -m json.tool
No JSON object could be decoded</code></pre></figure>
<h3 id="pros">pros</h3>
<ul>
<li>python is probably already installed on your computer</li>
</ul>
<h3 id="cons">cons</h3>
<ul>
<li>order of the keys is changed – alphabetized, it seems</li>
<li>no help if the JSON is invalid – you’re on your own</li>
</ul>
<h2 id="discussion-yajl">Discussion: YAJL</h2>
<p><a href="https://lloyd.github.io/yajl/">YAJL</a> is written in C and can be installed
with your favorite package manager. The command you want to use is
<code class="language-plaintext highlighter-rouge">json_reformat</code>.</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text"># cat sample.json | json_reformat
{
"first": "Jimmy",
"last": "James",
"age": 29,
"sex": "M",
"salary": 63000,
"registered": false
}</code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text"># cat invalid.json | json_reformat
lexical error: invalid char in json text.
{"first":Jimmy","last":"James"
(right here) ------^</code></pre></figure>
<h3 id="pros-1">pros</h3>
<ul>
<li>order of the keys is preserved – no funny business</li>
<li>tries to help with invalid JSON</li>
</ul>
<h3 id="cons-1">cons</h3>
<ul>
<li>needs to be installed</li>
<li>didn’t find the real problem in the invalid JSON</li>
</ul>
<h2 id="discussion-jq">Discussion: jq</h2>
<p><a href="https://stedolan.github.io/jq/">jq</a> is written in C and can be installed
with your favorite package manager.</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="err">#</span> <span class="nx">cat</span> <span class="nx">sample</span><span class="p">.</span><span class="nx">json</span> <span class="o">|</span> <span class="nx">jq</span> <span class="p">.</span>
<span class="p">{</span>
<span class="dl">"</span><span class="s2">registered</span><span class="dl">"</span><span class="p">:</span> <span class="kc">false</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">salary</span><span class="dl">"</span><span class="p">:</span> <span class="mi">63000</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">sex</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">M</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">age</span><span class="dl">"</span><span class="p">:</span> <span class="mi">29</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">last</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">James</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">first</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Jimmy</span><span class="dl">"</span>
<span class="p">}</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text"># cat invalid.json | jq .
parse error: Invalid numeric literal at line 1, column 15</code></pre></figure>
<h3 id="pros-2">pros</h3>
<ul>
<li>colored output – only on terminal, can be disabled</li>
<li>tries to help with invalid JSON</li>
<li>pretty-printing is only one of the things jq does – it’s a powertool that replaces sed/awk/grep</li>
</ul>
<h3 id="cons-2">cons</h3>
<ul>
<li>order of the keys is changed – hash order?</li>
<li>didn’t find the real problem in the invalid JSON</li>
<li>needs to be installed</li>
</ul>
<h2 id="discussion-jsonlint">Discussion: jsonlint</h2>
<p><a href="https://github.com/zaach/jsonlint">jsonlint</a> is an NPM package: <code class="language-plaintext highlighter-rouge">npm install -g jsonlint</code>.</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text"># cat sample.json | jsonlint
{
"first": "Jimmy",
"last": "James",
"age": 29,
"sex": "M",
"salary": 63000,
"registered": false
}</code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text"># cat invalid.json | jsonlint
[Error: Parse error on line 1:
{"first":Jimmy","last":"James
---------^
Expecting 'STRING', 'NUMBER', 'NULL', 'TRUE', 'FALSE', '{', '[', got 'undefined']</code></pre></figure>
<h3 id="pros-3">pros</h3>
<ul>
<li>optimal output</li>
<li>order of the keys is preserved – no funny business</li>
<li>will tell you exactly what’s wrong with your invalid JSON</li>
</ul>
<h3 id="cons-3">cons</h3>
<ul>
<li>needs to be installed</li>
</ul>
<h2 id="discussion-jsontool">Discussion: jsontool</h2>
<p><a href="https://github.com/trentm/json">jsontool</a> is an NPM package: <code class="language-plaintext highlighter-rouge">npm install -g jsontool</code>. The command you
want to use is <code class="language-plaintext highlighter-rouge">json</code>.</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text"># cat sample.json | json
{
"first": "Jimmy",
"last": "James",
"age": 29,
"sex": "M",
"salary": 63000,
"registered": false
}</code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text"># cat invalid.json | json
json: error: input is not JSON: Unexpected 'J' at line 1, column 10:
{"first":Jimmy","last":"James","age":29,"sex":"M","salary":63000"registered":false}
.........^
{"first":Jimmy","last":"James","age":29,"sex":"M","salary":63000"registered":false}</code></pre></figure>
<h3 id="pros-4">pros</h3>
<ul>
<li>optimal output</li>
<li>order of the keys is preserved – no funny business</li>
<li>will tell you exactly what’s wrong with your invalid JSON</li>
<li>it can group / extract / filter</li>
</ul>
<h3 id="cons-4">cons</h3>
<ul>
<li>needs to be installed</li>
</ul>
<h2 id="conclusion">Conclusion</h2>
<p>I used to install YAJL but now, since I always install node.js and NPM, I skip it.</p>
<p>Here’s what I used to recommend:</p>
<ul>
<li>use python if you have nothing else, or if it does what you need</li>
<li>otherwise use jq, for pretty printing and for <a href="https://stedolan.github.io/jq/manual/">everything</a> else it does – jq is amazing</li>
<li>but use jsontool if you care about key ordering – for example, if you’re reformatting package.json</li>
</ul>
<p>However, now that I have compiled this grid, I would use jsonlint instead of jsontool. I always
install jsonlint anyway, because that’s what <a href="https://blog.jpalardy.com/posts/how-to-configure-syntastic/">syntastic</a>
integrates with – I just had not realized that the output of both jsonlint and jsontool is equivalent.</p>
Grep and Output Buffering2013-11-02T00:00:00+00:00https://blog.jpalardy.com/posts/grep-and-output-buffering<p>Why does this work?</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">tail</span> <span class="nt">-f</span> some.log | <span class="nb">grep </span>something</code></pre></figure>
<p>But this doesn’t?</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">tail</span> <span class="nt">-f</span> some.log | <span class="nb">grep </span>something | <span class="nb">cut</span> <span class="nt">-c</span> -<span class="si">$(</span>tput cols<span class="si">)</span></code></pre></figure>
<p>(It doesn’t matter what’s on the 3rd pipe, but, in this case, it crops on terminal
width – it’s similar to <code class="language-plaintext highlighter-rouge">set nowrap</code> in vim)</p>
<p>I had been bitten many times by something like this, and it seemed like this
command would either:</p>
<ul>
<li>not output anything (at least for a time), then:</li>
<li>output pages in bursts</li>
</ul>
<p>Clearly, it is a buffering problem, but where’s the problem: <code class="language-plaintext highlighter-rouge">grep</code> or <code class="language-plaintext highlighter-rouge">cut</code>?
How could it be <code class="language-plaintext highlighter-rouge">grep</code> if my second example worked?</p>
<h2 id="reproducing-the-problem">Reproducing the problem</h2>
<p>I went with this code:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># generator.bash</span>
<span class="k">for </span>i <span class="k">in</span> <span class="o">{</span>1..1000<span class="o">}</span><span class="p">;</span> <span class="k">do
</span><span class="nb">echo</span> <span class="s2">"this is line </span><span class="nv">$i</span><span class="s2">"</span>
<span class="nb">sleep </span>1
<span class="k">done</span></code></pre></figure>
<p>which is meant to mimic the log output of some running daemon. It outputs lines
slowly because it will cause more buffering issues. Now run this:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">bash generator.bash | <span class="nb">grep </span>line | <span class="nb">cut</span> <span class="nt">-c</span> 9-</code></pre></figure>
<p>Where’s the output? What’s the problem? How do you fix this?</p>
<h2 id="solutions">Solutions</h2>
<p><a href="https://www.perkin.org.uk/posts/how-to-fix-stdio-buffering.html">How to fix stdio buffering</a> gave me most of the answer. <code class="language-plaintext highlighter-rouge">grep</code> has
a flag for this:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">bash generator.bash | <span class="nb">grep</span> <span class="nt">--line-buffered</span> line | <span class="nb">cut</span> <span class="nt">-c</span> 9-</code></pre></figure>
<p>but a more general answer might be to use <code class="language-plaintext highlighter-rouge">stdbuf</code> (on Linux):</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">bash generator.bash | <span class="nb">stdbuf</span> <span class="nt">-o0</span> <span class="nb">grep </span>line | <span class="nb">cut</span> <span class="nt">-c</span> 9-</code></pre></figure>
<p>Bonus round:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">tail -f</code> keeps flushing its STDOUT, so it doesn’t cause a problem.</li>
<li><code class="language-plaintext highlighter-rouge">awk</code> doesn’t buffer either</li>
<li><code class="language-plaintext highlighter-rouge">sed</code> has the <code class="language-plaintext highlighter-rouge">--unbuffered</code> flag</li>
</ul>
<h2 id="why">Why?</h2>
<p>The above article linked to the very interesting <a href="https://www.pixelbeat.org/programming/stdio_buffering/">buffering in standard streams</a>.
Here’s how buffering is usually set up:</p>
<ul>
<li>STDIN is always buffered.</li>
<li>STDERR is never buffered.</li>
<li>if STDOUT is a terminal, line buffering will be automatically selected. Otherwise, block buffering (probably 4096 bytes) will be used.</li>
</ul>
<p>I have used the command-line for years without worrying about this, but these 3
points explain all “weird” behaviors.</p>
Bookmarklet: Toggle Ruby2013-09-30T00:00:00+00:00https://blog.jpalardy.com/posts/bookmarklet-toggle-ruby<p>Ruby characters are characters placed on top of Chinese characters to show
pronunciation. Wikipedia can explain <a href="https://en.wikipedia.org/wiki/Ruby_character">ruby characters</a> better than I can.</p>
<p>In practice, it looks like this: <ruby>日本語<rt>にほんご</rt></ruby>は<ruby>難<rt>むずか</rt></ruby>しいです. The Japanese characters above the text are called <a href="https://en.wikipedia.org/wiki/Furigana">furigana</a>.</p>
<p>What’s interesting is what the HTML markup looks like:</p>
<p><img src="/assets/toggleruby/markup.png" alt="HTML ruby markup" /></p>
<p>The <code class="language-plaintext highlighter-rouge">ruby</code> tag contains a <code class="language-plaintext highlighter-rouge">rt</code> (ruby text) with the pronunciation. The browser
has some flexibility over the rendering, but the ruby usually goes on top, in a
smaller font.</p>
<p>I created a bookmarlet to toggle the ruby text’s visibility: <a href="javascript:(function()%7Bvar%20id%3D%22c8f38bbf013e6f254dfe129984188c9a2646b793%22%2Cd%3Ddocument%2Cstyle%3Dd.getElementById(id)%3Bstyle%3Fd.head.removeChild(style)%3A(style%3Dd.createElement(%22style%22)%2Cstyle.innerHTML%3D%22rt%20%7B%20visibility%3A%20hidden%3B%20%7D%22%2Cstyle.id%3Did%2Cd.head.appendChild(style))%3B%7D)()">toggle ruby</a>. Click it now and watch the ruby in the 2nd paragraph disappear/appear on each click. Drag the link to your bookmark toolbar to install.</p>
<p>Toggling the ruby looks like this on a news site:</p>
<p><img src="/assets/toggleruby/toggleRuby.gif" alt="HTML ruby markup" /></p>
<p>Why is this useful? If you’re practicing your kanji reading, ruby text is the
answer. You might want to hide it — until you try to read it for yourself. Then,
you can show it again to check your answer.</p>
<p>The code for this (<a href="https://github.com/jpalardy/bookmarklets/blob/main/src/toggleRuby.js">toggleRuby.js</a>)
is in my <a href="https://github.com/jpalardy/bookmarklets">bookmarklets</a> repository on github.</p>
Garbage Collected Directories2013-09-03T00:00:00+00:00https://blog.jpalardy.com/posts/garbage-collected-directories<p>“I wish all these temporary directories wouldn’t clutter my home directory.”</p>
<p>We are all familiar with directories cleverly named “tmp”, “test”, “whatever”,
or “DELETE-ME”. These directories are usually created for short-lived
experiments or to contain a “mess”.</p>
<p>For example: you download a package, untar/gunzip it, <code class="language-plaintext highlighter-rouge">./configure; make; make
install</code>. You start to play with what you installed and forget about the
directory you’re leaving behind. That clutter will haunt you later.</p>
<p>I created a simple script to clean up after myself, I call it <code class="language-plaintext highlighter-rouge">tad</code> (__t__hrow-__a__way
__d__irectory). The <a href="https://github.com/jpalardy/dotfiles/blob/00645a668d3fd59bd13b57c0d1c656bf23691710/bash/commands/tad.bash">code</a> is on Github, but
here’s a snapshot:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tad<span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">ts</span><span class="o">=</span><span class="si">$(</span><span class="nb">date</span> +%s<span class="si">)</span>
<span class="nb">local </span><span class="nv">d</span><span class="o">=</span><span class="s2">"</span><span class="nv">$HOME</span><span class="s2">/.throw-away/</span><span class="nv">$ts</span><span class="s2">"</span>
<span class="nb">mkdir</span> <span class="nt">-p</span> <span class="nv">$d</span>
<span class="o">(</span><span class="nb">cd</span> <span class="nv">$d</span><span class="p">;</span> bash<span class="o">)</span>
<span class="nb">rm</span> <span class="nt">-r</span> <span class="nv">$d</span>
<span class="o">}</span></code></pre></figure>
<p>It creates a directory under <code class="language-plaintext highlighter-rouge">$HOME/.throw-away</code>, changes (cd) into it and
launches a shell there. When the shell exits, the directory is deleted with all
its content.</p>
<p>In practice, it means that I type <code class="language-plaintext highlighter-rouge">tad</code> and I’m in a sandbox. And when I exit
the shell, the sandbox is gone, and I’m back in the original directory from
which I issued the command. This script is the formalization of something I
used to do manually.</p>
<p>What does it mean to have garbage collection for directories?</p>
<p>It changes the way you work. I have had this script for quite some time: I use
it all the time and keep finding new ways to use it. Not having to name the
directory and not having to clean up <em>frees</em> me – those things are
<a href="https://en.wikipedia.org/wiki/Accidental_complexity">accidental complexity</a>.</p>
Software I like: ImageOptim2013-08-24T00:00:00+00:00https://blog.jpalardy.com/posts/software-i-like-imageoptim<p><a href="https://imageoptim.com/">ImageOptim</a> is software that makes me happy. It’s a simple
install, does one thing and does it well: it tries to reduce the size of image
files losslessly.</p>
<p>Here’s the 15-second demo:</p>
<p><img src="/assets/imageoptim/imageOptim.gif" alt="ImageOptim demo" /></p>
<p>Drag-and-drop to get ≈70% reduction in file size? Yes.</p>
<p>Of course, the amount of compression you’ll get will depend on a lot of
factors, most of which you and I never tried to understand. And if we
understood them, we could manually feed the right parameters to the compressors
that ImageOptim uses.</p>
<p>If you want to start tweaking, there are preferences for that:</p>
<p><img src="/assets/imageoptim/options.png" alt="ImageOptim preferences" /></p>
<p>When do I use ImageOptim? All the time. But if you’re looking for more specific
use cases:</p>
<ul>
<li>when I take a screenshot, and I’m about to email/share it</li>
<li>when I put an image online</li>
<li>when I integrate an asset into a webapp</li>
</ul>
<p>It never hurts to see if I can squeeze out a little more: ImageOptim won’t
modify the image unless it can compress it.</p>
<p>One warning: ImageOptim modifies files in-place. If you have “master” files, I
would recommend making copies and working off of those (always a good idea).</p>
How to Configure Syntastic2013-08-21T00:00:00+00:00https://blog.jpalardy.com/posts/how-to-configure-syntastic<p><a href="https://github.com/scrooloose/syntastic">Syntastic</a> is great. When it’s configured properly, it will
highlight warnings and errors inline in the current Vim buffer. Here’s the
obligatory Syntastic screenshot:</p>
<p><img src="/assets/syntastic/features.png" alt="Syntastic features" /></p>
<p>My experience with Syntastic was that it worked even though I had done nothing
to configure it. The default configuration had done its job and, when I saved a
buffer, my mistakes would be highlighted.</p>
<p>(I’m going to discuss C, but this applies to any language supported by
Syntastic. I picked C because it was the first language that forced me to
understand how Syntastic worked due to its vast number of checkers.)</p>
<p>Trying to understand how Syntastic works, I dug into the <a href="https://github.com/scrooloose/syntastic/tree/master/syntax_checkers">syntax checkers</a>
directory and opened the <a href="https://github.com/scrooloose/syntastic/tree/master/syntax_checkers/c">c</a> subdirectory. Inside
I found a bunch of files, each named after the executable used to check the
syntax. That made a lot of sense, I could see where this was going.</p>
<p><img src="/assets/syntastic/c_checkers.png" alt="Syntastic C checkers" /></p>
<p>But which one(s) of these would be used when checking my file? You can find out:</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">:SyntasticInfo</code></pre></figure>
<p>and it might look like this on your system:</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">Syntastic info for filetype: c
Available checkers: gcc make splint
Currently active checker(s): gcc</code></pre></figure>
<p>Think of it as a funnel:</p>
<p><img src="/assets/syntastic/funnel.png" alt="Syntastic checkers funnel" /></p>
<p>The first one, “ALL the checkers supported by Syntastic” changes over time; as
people contribute more checkers, the list will grow. You can check the
<a href="https://github.com/scrooloose/syntastic/tree/master/syntax_checkers">syntax checkers</a> subdirectories on Github or the
<a href="https://github.com/scrooloose/syntastic/wiki/Syntax-Checkers">wiki</a>.</p>
<p>The second one, “the checkers installed on your computer”, is self-explanatory :-)</p>
<p>The last one, “the checker(s) Syntastic will use”, is worth discussing. The
logic seems to be the first item from the list from <code class="language-plaintext highlighter-rouge">:SyntasticInfo</code>’s
“Available checkers” filtered through a <a href="https://github.com/scrooloose/syntastic/blob/d82ee05a80c023f1d531569e56595f5f9cb0fde2/plugin/syntastic/registry.vim#L6">whitelist</a>.
That’s not the whole story, however, because you can skip this logic and
manually tell Syntastic what to use.</p>
<p>Here’s what I recommend:</p>
<ol>
<li>start with seeing <a href="https://github.com/scrooloose/syntastic/wiki/Syntax-Checkers">what’s possible</a></li>
<li>install the checkers you want – make sure the executable names match what Syntastic expects</li>
<li>check <code class="language-plaintext highlighter-rouge">:SyntasticInfo</code> – if it matches your needs, you are done.</li>
<li>drop a line in your <code class="language-plaintext highlighter-rouge">.vimrc</code> to specify the checkers</li>
</ol>
<p>Here’s what that config might look like:</p>
<figure class="highlight"><pre><code class="language-vim" data-lang="vim"><span class="k">let</span> <span class="nv">g:syntastic_c_checkers</span><span class="p">=[</span><span class="s1">'make'</span><span class="p">,</span><span class="s1">'splint'</span><span class="p">]</span></code></pre></figure>
<p>This follows the <code class="language-plaintext highlighter-rouge">g:syntastic_<filetype>_checkers</code> format. If you manually
specify that list, you can put multiple checkers. If you do, each checker will
be called, in order, as long as the previous checkers don’t return an error.
In that case, you might want to list checkers from fastest to slowest, or from
looser to stricter.</p>
<p>I generalized on a few points, a lot of this behavior is customizable. The whole truth
can be found in the <a href="https://github.com/scrooloose/syntastic/blob/master/doc/syntastic.txt">documentation</a>.</p>
New blog2013-08-18T00:00:00+00:00https://blog.jpalardy.com/posts/new-blog<p>First it was <a href="https://jpalardy.blogspot.ca/">Blogger</a>, then <a href="https://technotales.wordpress.com/">Wordpress</a>.</p>
<p>This time, I decided to “own” my blog and put it under my domain. This blog is generated by <a href="https://jekyllrb.com/">Jekyll</a> and
themed by <a href="https://zachholman.com/posts/left/">Left</a> (thank you, <a href="https://twitter.com/holman">Zach Holman</a>).</p>
<p>I blog about technical things and, on other platforms, I always had to struggle
with code formatting, code highlighting and previewing. It’s great to be able
to do it all locally, with a text-editor, and using source control. If I only
wrote stories, if it were all text, I’m sure it would matter a lot less.</p>
<p>All these little annoyances add up, and I hope to get back to blogging on a more
frequent basis. I always have something exciting to talk about, but not always
the time or willingness to blog about it.</p>
<p>This blog does not have comments, but I don’t know (yet) if it matters. In any case, you
can reach me through <a href="https://github.com/jpalardy">the</a> <a href="https://twitter.com/jpalardy">usual</a> <a href="mailto:jonathan.palardy@gmail.com">channels</a>.</p>