<p>As you might have noticed, I&#8217;ve been revamping my website in the last days.</p>

<p>One of the major problems was to deal with the usual stuff developers hate (including me) that <a href="http://sergiosantos.info">S&#233;rgio</a> likes to call <strong>wtf-8</strong>: encoding issues.</p>

<p>We have a <a href="http://www.dreamhost.com">DreamHost</a> account and the <a href="http://www.python.com">python</a> we run there is <span class="caps">ASCII</span>. So all the website output was <span class="caps">ISO</span>-8859&#8211;1 but the content from the feeds ( using <a href="http://feedparser.org">feedparser</a> btw) were utf-8. After a few tries, I finally managed to get it right.</p>

<p>First I tried <a href="http://chardet.feedparser.org/">chardet</a> to detect the encoding and then use encode and decode functions like this:</p>

<div class="code">
import chardet<br />
for line in page.output().split("\n"):<br />
&nbsp;&nbsp;&nbsp;&nbsp;try:<br />
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;it = chardet.detect(line)<br />
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print line.decode(it['encoding']).encode('utf-8')<br />
&nbsp;&nbsp;&nbsp;&nbsp;except:<br />
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print line</div>

<p>But this turned out not to be a good solution, because reading the feed I had to convert it to a useful encode, and I couldn&#8217;t encode into <span class="caps">UTF</span>-8 a <span class="caps">UTF</span>-8 string. So I looked for a better and easy solution and, guess what, I found it. When reading the <span class="caps">UTF</span>-8 feed, I had just to use</p>

<div class="code">
title=str(entry['title'].encode('ascii', 'xmlcharrefreplace'))
</div>

<p>This will convert any character that can&#8217;t be converted to ascii into it&#8217;s <span class="caps">XML</span> Character Referente code. Problem solved.</p>

<p>One thing I also found and I&#8217;m looking forward to add to <strong>Pungi</strong> is a <span class="caps">HTML</span> tidy wrapper for Python. <a href="http://utidylib.berlios.de/">uTidylib</a> or <a href="http://www.lemburg.com/files/python/mxTidy.html">mxTidy</a>. However I couldn&#8217;t make any of them work on DreamHosting hosting yet. If you had, feel free to help me :)</p>