2014-04-23T23:11:49Z /feed.html/ Debugging strange errors with strace Carl Johan Gustavsson 2014-04-23T20:11:00Z 2014-04-23T20:11:00Z /blog/2014/04/debugging-with-strace.html <p>Yesterday my collegue came to me with a strange error in our development environment. For some reason the message queue consumers failed fetching new messages from Amazon <span class="caps">SQS</span> with a <span class="caps">SSL</span> library error. The error message we got&nbsp;was </p> <pre><code>[Errno 185090050] _ssl.c:340: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib" </code></pre> <p>which could be interpreted as the process could not load the certificate revocation list&nbsp;properly. </p> <p>This error appeared after my collegue deployed a very minor and totally unrelated code change. However, we proceeded to deploy the previous version, just to be sure. The error&nbsp;persisted.</p> <p>After googling a bit I found that the file it was supposed to be looking for was the <code>cacerts.txt</code> inside <code>boto</code>. I went ahead and check if that file existed in our virtualenv and it did. Very&nbsp;strange.</p> <p>That meant I needed to know the absolute path of the file it could not find. For that I needed to see what syscall it failed on, so strace comes to the rescue. Strace is a neat utility to trace all syscalls a process does and the signals it receives. I attached strace to one of our server processes by the following&nbsp;command.</p> <pre><code>strace -v -p &lt;pid&gt; 2&gt;&amp;1 | grep cacert </code></pre> <p>The output I got was similar to the following (I didn&#8217;t save the output at the&nbsp;time)</p> <pre><code>open("/xxxxxx/1398298388/lib/python2.7/site-packages/boto/cacerts/cacerts.txt", O_RDONLY) = -1 </code></pre> <p>So it was looking for the file I checked for. However, there is one thing here, you might recognize <code>1398298388</code> as a unix timestamp. It is the time when our virtual environment was created. We create a new one for each build and then deploy the entire virtual environement to our servers, activate it and restart the processes. In the <code>xxxxxxx</code> directory we have the latest virtual environements we have&nbsp;deployed. </p> <p>This specific virtual environment the process was launched from was now missing from our server. It is due to that we only keep the last few environments on the server. What had happend was that our processes had not been restarted properly after the last deploys. We have supervisor monitoring our server processes and we had recently change how processes are started by introducing a wrapper script. This script did not forward the shutdown signals properly. Restarting the processes properly solved the <span class="caps">SSL</span>&nbsp;error.</p> <p>The conclusion that can be drawn is that strace is very useful for debugging errors that have no obvious reason and you need to look closer at the process to see what it actually tries to&nbsp;do. </p> Asyncio part 1 Carl Johan Gustavsson 2014-04-16T23:45:00Z 2014-04-16T23:45:00Z /blog/2014/03/asyncio-pt-1.html <p>Python 3.4 was released a couple of weeks ago and with it the new asynchronous library <a href="https://docs.python.org/3.4/library/asyncio.html">asyncio</a>. As I have worked quite a lot with Twisted at my day job I am quite thrilled for this new built-in async&nbsp;library.</p> <p>The new asyncio library is also a good reason to start doing Python 3 development. Due to using Twisted at my dayjob I have mostly used&nbsp;2.7. </p> <p>To explore asyncio I&#8217;m going to write a small application using it. Recently I have gotten a bit interested in distributed hash tables (<span class="caps">DHT</span>) and that seems like a good fit for for testing out an async library. Having researched about DHTs I found the <a href="http://www.cs.rice.edu/Conferences/IPTPS02/109.pdf" title="Kademlia: A Peer-to-peer Information System Based on the XOR Metric">paper on Kademlia</a>. Go ahead and read it if you haven&#8217;t already, it is quite easy to read and describes the rationale behind the design decisions. Next resource I found was this <a href="http://xlattice.sourceforge.net/components/protocol/kademlia/specs.html" title="Kademlia: A Design Specification">specification</a>. </p> <p>After reading I started by building a very simple <a href="http://en.wikipedia.org/wiki/JSON-RPC" title="Wikipedia: JSON-RPC"><span class="caps">JSON</span> <span class="caps">RPC</span></a> layer using asyncio primitives communicating over <span class="caps">UDP</span>. The progress can be found at the <a href="https://github.com/cjgu/asyncio-kademlia">asyncio-kademilia</a> project page on github] and more details will come in part&nbsp;2.</p> San Francicso move Carl Johan Gustavsson 2014-02-01T00:00:00Z 2014-02-01T00:00:00Z /blog/2014/02/san-francisco.html <p>I finally got some time over to continue the blog project after my move to San Francisco. Now it is actually published on my domain&nbsp;also.</p> <p>And by the way San Francisco is awesome even in winter, kind of like Swedish summer. I got myself a bike here also, a single-speed this time. I&#8217;ve always been a bit against the whole fixie/single-speed thinge, but so far I like my single-speed. It is not a fixie a the moment but can be converted so I guess I at least have to try it before judging it&nbsp;more.</p> <p>San Francisco is great to bike in, I would say better than in Stockholm, maybe except for the hills. The bike lanes are better and car drivers seems to be nicer and more&nbsp;careful.</p> Hyde, the static web generator Carl Johan Gustavsson 2013-12-27T00:00:00Z 2013-12-27T00:00:00Z /blog/2013/12/hyde.html <p>So, I needed a small Christmas project, and a place to dump some random thoughts, what would be a better place than a simple blog. When I say simple I mean static <span class="caps">HTML</span>, preferably served from Amazon&nbsp;S3.</p> <p>After taking a look at the Python-based options, I found <a href="http://blog.getpelican.com/">Pelican</a>, which seemed really nice, but I quickly noticed that it is <span class="caps">AGPL</span>-licensed. Second best alternative was <a href="https://github.com/hyde/hyde/">Hyde</a> which is <span class="caps">MIT</span>-licensed. As liberal licensing appeal a lot to me, the choice was easy to&nbsp;make.</p> <p>The standard design template certainly needed a bit of work (and it is by no means done) but is now in an acceptable&nbsp;state.</p> <p>For publishing to S3, I didn&#8217;t like the complexity of the S3 addon publishers that were available so I decided to use a s3cmd instead, making it a one line&nbsp;deploy:</p> <pre><code>s3cmd sync deploy/ s3://&lt;bucket-name&gt;/ </code></pre> <p>And thats it, very easy. Also, s3cmd sync uses the etag/md5 of files making sure to only upload files that have changed on subsequent&nbsp;deploys.</p>