blogroll tags

HTTPS and SSL in practice

To the administrators of the Oxford Student Self Registration website,

When accessing https://www.studentsystem.ox.ac.uk/ for the first time, I was surprised to see Firefox claim that it uses an invalid security certificate. As somebody who does understand the risks involved, I was unwilling to simply blindly accept the certificate, and had a closer look.

It turns out that the signing authority of your certificate, known as "Cybertrust Educational CA", is not included in Mozilla browsers by default, and thus Firefox rightly rejects your certificate. I was able to locate this CA certificate; it is, in turn, signed by a CA known as "GTE Global Root", which is included in Firefox.

Please configure your server to supply the intermediate certificate, "Cybertrust Educational CA", to browsers. This is the correct procedure and the only way in which the site will be accepted by Mozilla browsers, and any other browsers with security policies that are, as they should be, similarly strict.

This newsgroup thread may be of interest: http://groups.google.com/group/mozilla.dev.tech.crypto/browse_thread/thread/dcad2183363667f1?pli=1

Allow me to explain why I feel this is important: by using an apparently invalid security certificate, especially on a website students must use, you are requiring students to simply trust an unknown certificate in good faith, in effect training them not to take these security warnings seriously. While it is true that in this case, it was possible to verify the certificate manually, most students will probably never possess the necessary expertise - why should they? - and instead "learn" to treat computer security with an attitude that leaves the door wide open for fraudulent sites and man-in-the-middle attacks.

Coming from an education institution, this is particularly worrying.

Yours faithfully,

Thomas Jollans

PS: The contact form I'm using uses an unencrypted connection to send the email, even when this website is being accessed over a secure connection. This might also be worth changing.

Addendum: I just sent this letter to Oxford University Computing Services (OUCS) using their so-called suggestions form. Perhaps not the best place, but of those potential addressees I could find this was the one where I felt most confident that my message would end up in the right place. This whole episode reminded me of, and indeed my writing this letter was perhaps inspired by, a blog post by my cousin Leon about a month ago, concerning a similarly problematic attitude at the University of Greenwich.


vi survival guide

I wrote this article, or shall we call it crash course, on vi, back in 2006, and published it Nuxified. (it even made a little noise on digg, which was new and cool in those pre-reddit days) I am republishing it here in the hope that somebody may still find it useful. Appendix 2 is now slightly out of date since VIM added tabs (an excellent feature I use all the time)

1. Introduction
2. Basic crash course
3. Other Modes
  3.1. A guide to NORMAL mode
    3.1.1. Movement
    3.1.2. Modifying, Deleting, Copying, Pasting
    3.1.3. Repetition and how to avoid it
  3.2. Ex mode
4. Searching and replacing
APPENDIX. VIM - Vi IMproved
  APPENDIX.1. VISUAL mode
  APPENDIX.2. Multiple Windows
Final Words

1. Introduction

If you are trying to work at a command line in pretty much any Operating System, no matter if it is a GNU/Linux brand such as Ubuntu or Fedora Core, another UNIX-like system like Mac OS X or FreeBSD or even Microsoft Windows, you will most definately need a text editor sooner or later. If you are using a UNIX-like system, like GNU/Linux variants, there are many text editors that may or may not be installed. Luckily there is one de-facto standard: vi is included in almost every GNU/Linux or UNIX distribution. "vi" is a truncation of the word "visual" and has, contrary to popular belief, nothing to do with the Roman number 6. Many people are not too fond of vi, but are, alas, often forced into using it when it is the only known option. You may prefer easier editors like gedit (Gnome), kate (KDE), emacs or ne (command line), but these may not be present when working at the command line. Therefore, it is best to know at least the basics of vi and to understand it a little better; the basics of survival in vi, which you will learn in this guide. Unlike what you may have heard about the invention of the language C, the vi editor was not the idea of some kind of sick joke. It was for real and it served a real purpose. I am writing this in vi.

vi is very old, 1976 to be exact - that is only 6 years after 1970-01-01, which is the second no. 0, the big bang, the birth of Jesus Christ of UNIX system clocks. In the age of informatin and the internet, 30 years is practically a century. Remembering the age can help you to accept it if you do not particularly like it. Still, however, many people like the fact that they can memorize the keystrokes and be at home with it for quick tasks, and they like its tiny footprint of disk space and RAM. They like it both as the old reliable standard on most distributions of GNU/Linux and as a simple but powerful editing tool.

The vi editor has an interesting history. It was created in a Berkeley University dorm room, Evans Hall, in 1976 by Bill Joy and some of his friends, although he was the one who fleshed most of it out, using the Pascal language. He designed it for a version of Unix he was working on at Berkeley University called BSD Unix, which stands for Berkeley Software Distribution Unix and forms the base of some modern Unices such as the polular FreeBSD. Bill Joy eventually moved on to Sun Microsystems as one of its original founding members and has since moved on to other projects. Before Bill wrote vi, the students at the university were using an editor called ed. The ed tool, still as widespread as vi, unfortunately, is much harder to use. The students discovered that they could use terminal escape codes so that the terminal moved the cursor to the bottom, and they dubbed this "open mode" where short commands could be typed. There were later derivatives of the ed tool called em and en, and students were experimenting with reprogramming each and borrowing each other's features. Eventually the most popular became Bill's project, vi. When the VAX came about, the terminal escape codes standardized as VT100 (Vax Terminal 100) and this carried over into the BSD Unix and is still there today, implemented, for example, in the Linux console and xterm. These codes were utilized in moving the cursor around in the editor, scrolling sections, blocking off certain sections so that they do not scroll, and other cursor movement features. Anyway, what is most important to note about vi's history is that there was very limited memory and disk space, so commands seemed to be more efficient if they were short keystrokes, rather than typing words. Also, menus and even a mouse weren't invented yet at XEROX PARC center, so short keystroke commands seemed to fit the ticket. Dan Bricklin's Visicalc program would later hinge off of that aspect and use keystrokes the brought up menus, and that approach led to the success of Lotus 1-2-3. So short keystrokes it became, and each of us who use Linux should get to know how to survive in using these keystrokes. Like many things in Unix and Linux, when enough people know about a command and depend on it, it becomes one of the defacto items that just has to be rolled in with it, like it or leave it, or enough people will complain.

back to top

2. Basic crash course

Let's start with the most basic vi knowledge - what you will need to use vi when it is the only choice. vi is different from most editors in that it has different so-called modes. When you start vi, you are in NORMAL mode. Here every key has some special function. For example, pressing 'x' deletes the character under the cursor. Beginners will start off preferring the INSERT mode; you enter it by pressing 'i' in NORMAL mode. Here letters are letters, backspace is backspace etc - like in any old editor. This is a kind of "safe" mode for beginners, as, unlike in NORMAL mode, keys don't do unexpected things. When you have made the changes you want, switch back to NORMAL mode by pressing <ESC>. You can always return to insert mode by pressing i. Now you will want to do some special things such as saving. Here are some command you should know:

:w <ENTER>write file
:wq <ENTER>write file and quit
:w FILENAME <ENTER>write file as FILENAME
:e FILENAME <ENTER>edit file FILENAME
:q <ENTER>quit
:q! <ENTER>quit without saving

back to top

3. Other Modes

By now you should be able to survive in vi for a quick config file edit, code hack or simple ASCII art.But vi is a lot more than insert mode...

3.1. A guide to NORMAL mode

Normal mode is, as you may have gathered or guessed the vi mode. You can do almost anything from inside NORMAL mode.

3.1.1. Movement

Even in NORMAL mode, you can use the arrow keys to navigate, but this always means moving your hand from the letters to the arrow keys and back, resulting in some latency. On laptop keyboards, this is especially inconvieneant. Instead, vi offers the following movement keys:

k up
h leftl right
j down

On most keyboards, these keys are under a touch typist's right hand and thus very accessible. By the way, vi is a perfect editor to learn touch typing with ! These keys can be memorized in different ways. h and l are the furthest left and right of the set; the letter j looks slightly like it's pointing downward and the key usually has a little stud (for finding the key when touch typing) at the bottom.
Here are some more movement keys, some of which are especially useful in combination with other commands (see next section):

Inside the line
$moves to the end of the line
0moves to the beginning of the line
^moves to the first non-whitespace character - i.e. the beginning of an indented line.
Movement via words (where a word is a sequence of alphanumeric OR punctuation signs)
wnext word
eend of current word
bprevious word
Movement via words (where a word is a sequence of non-blank characters)
Wnext word
Eend of current word
Bprevious word
Other Jumps
Hjump to top of screen
Ljump to bottom of screen
{jump to previous paragraph
}jump to next paragraph
fxjump to next occurence of character x (where x is any character, of course)
txjump right before next occurence of character x ('till x) (again, x in anything)
Fxjump to previous occurence of character x (again, x in anything)
Txjump right after previous occurence of character x (back 'till x) (again, x in anything)
Gjump to EOF (End of file)
LINE Gjump to line no. LINE
3.1.2. Modifying, Deleting, Copying, Pasting

Obviously, you are in vi to edit text. You could also be in vi to hear beeps when you press some wierd keys, but then you probably wouldn't be reading a guide to vi. You have already learned about INSERT mode, but there is more to it that just 'i' ! To "just" get into INSERT mode, you can use the following commands:

iplaces you in INSERT mode before the current character.
Iplaces you in INSERT mode at the beginning of the line. Same as ^i
aplaces you in INSERT mode after the current character. Same as i<RIGHT-ARROW> (or li when not at the eng of the line)
Aplaces you in INSERT mode at the end of the line. Same as $a
oplaces you in INSERT mode in a new line below the current one. Same as $i<ENTER>
Oplaces you in INSERT mode in a new line above the current one. Same as ko

vi also offers simple command for single-character deleting and editing:

xdelete character under cursor. like <DEL> in INSERT mode.
Xdeletes the previous character. like Backspace in INSERT mode.
rxreplace the character under the cursor with character x (where x is any character)

Now let's go over to some more larger-scale editing in NORMAL mode, as you seriously don't want to use 'x' or 'X' to delete a paragraph of text, for example. This is also the movement commands from section 3.1.1. become really useful.

d{motion}delete all text up to the destination of movement {motion}
c{motion}delete all text up to the destination of movement {motion} and enter INSERT mode (change the text)
y{motion}copy (yank in vi-speak) all text up to the destination of movement {motion}
ppaste (or put) the last deleted, changed or yanked text after the current position
Ppaste (or put) the last deleted, changed or yanked text before the current position

for better comprehention, a few examples are in order.

dwdelete up to the beginning of the next word
{c}change current paragraph
ywPduplicate word

There are also a few special cases, namely:

dddeleted current line
ccchange current line
yyyank current line

when you delete, change or yank a line, p and P also operate line-wise: p places the yanked line after the current line, P places it before the current line.

3.1.3. Repetition and how to avoid it

Repetition is annoying and senseless. vi has some features to make your edit easier in this respect. First of all comes the command . (yes, the period.); it repeats the last command. For instance, if your last command was dd, then pressing . deletes another line.
Counts are another very useful feature that let you repeat a command a specific number of times. The syntax is as follows:

{number}{command}do {command} {number} times.

This also works for motions that are part of a command. A few examples:

10jmove 10 lines down
3yyyank three lines
c3wchange 3 words

To round the section off, I'll show you some commands that every editor, including Microsoft® Notepad, supports:

uundo last command (including INSERT mode sessions)
<Ctrl+r>redo last undone command

Unlike Microsoft® Notepad, vi supports doing this multiple times.

3.2. Ex mode

"ex mode" is a command-line like mode. in it you can type certain longer commands, which are executed by typing <ENTER> There are multiple ways to use it:

:{ex-mode command}execute one ex mode command
gQenter ex mode

Sound familiar ? the saving, editing and quitting commands in section 2 where actually ex mode commands. Here are some important ex mode commands: (without the colon): part in square brackets are optional.

w[rite] [FILENAME]write file to FILENAME (if given)
w[rite]! FILENAMEwrite file to FILENAME, overwriting if it exists.
e[dit] FILENAMEedit FILENAME
e[dit]! FILENAMEedit FILENAME without saving
q[uit]quit
q[uit]!quit without saving
! COMMANDexecute shell command COMMAND
vi[sual]exit ex mode

back to top

4. Searching and replacing

Searching and replacing are two very important features when editing text files; just think of how easy it is to lose a line in a large config file or on an enourmous screen. I'll start with the simpler of the two: searching. The commands are issued in NORMAL mode.

/REGEXsearch for strings matching the regular expression /REGEX/.
?REGEXsearch for strings matching the regular expression /REGEX/ backwards.
ngo to next occurence
Ngo to previous occurence

This may sound dreadfully complicated to you, who may be thinking along the lines of "regular expression... huh?". Well, don't panic. Regular expressions are a standardized way of expressing patterns to search for. The syntax is not easy to understand and is beyond the scope of this document. So you get the idea what it is, I'll give a few examples: '[0-9]{1,3}' matches any one to three-digit number and 'joe' matches the three letters 'joe' (lowercase) and nothing else. If you're not dealing with special characters, you can forget about the regex buisiness.

Now to replacing. The syntax was inherited from [s]ed, so it will be familiar to the gurus ;) These are ex mode commands, and again use regular expressions, this time with bracket substitution, as it is replacing. (again, you can safely ignore that most of the time)

s/foo/bar/replace the first occurence of foo on the current line with bar
s/foo/bar/greplace all occurences of foo on the current line with bar
%s/foo/bar/[g]perform s/foo/bar/[g] on every line in the file.
X,Ys/foo/bar/perform s/foo/bar/[g] on every line between line # X and # Y.

APPENDIX. VIM - Vi IMproved

There are many vi clones and dereviatives out there. In fact, when you open "vi", you will probably not see the original vi, but a clone like nvi or vim. VIM is probably by far the most advanced clone; it has numerous features that are, among others, interesting for programmers. Here I will explain two features that are, in my eyes, the most interesting additions apart from syntax highlighting, which is enabled with the ex command syntax on: Multiple windows and VISUAL mode.

APPENDIX.1. VISUAL mode

Yes, another mode. In this mode, you can select text and perform operations that usually take a movement as argument. For example, selecting text and pressing 'd' will delete the text. As usual, you enter VISUAL mode from NORMAL mode, but unlike the other modes, VISUAL has multiple (three) sub-modes.

vEnter normal VISUAL mode
VEnter VISUAL LINE mode
<Ctrl+v>Enter VISUAL BLOCK mode

normal VISUAL mode

This is pretty much like selecting in most graphical editors: you select character-wise. Let me show you an example:

Some text text text just to show you what VISUAL mode is all about. As I said this is:
    Some text text text just to show you what VISUAL mode is all about. As I said this is:
        Some text text text just to show you what VISUAL mode is all about. As I said this is:
Some text text text just to show you what VISUAL mode is all about. As I said this is:
    Some text text text just to show you what VISUAL mode is all about. As I said this is:
        Some text text text just to show you what VISUAL mode is all about. As I said this is:

VISUAL LINE

This is a little bit niftier. It selects line-wise, not character-wise. Doing 'd' on VISUAL LINE selected text is like doing 'dd' on all the lines.

Some text text text just to show you what VISUAL mode is all about. As I said this is:
    Some text text text just to show you what VISUAL mode is all about. As I said this is:
Some text text text just to show you what VISUAL mode is all about. As I said this is: Some text text text just to show you what VISUAL mode is all about. As I said this is: Some text text text just to show you what VISUAL mode is all about. As I said this is:
Some text text text just to show you what VISUAL mode is all about. As I said this is:

VISUAL BLOCK

This is the niftiest of selection modes. Here you select blocks (duh). I cannot convey this better than with the example below.

Some text text text just to show you what VISUAL mode is all about. As I said this is:
    Some text text text just to show you what VISUAL mode is all about. As I said this is:
        Some text text text just to show you what VISUAL mode is all about. As I said this is:
Some text text text just to show you what VISUAL mode is all about. As I said this is:
    Some text text text just to show you what VISUAL mode is all about. As I said this is:
        Some text text text just to show you what VISUAL mode is all about. As I said this is:

APPENDIX.2. Multiple Windows

Vim has support for splitting the screen into multiple windows. These are not the overlapping windows you know from popular graphical user interfaces, these parts of a split screen as users of GNU Emacs and GNU screen will know. With them you can, for example, edit in one part of the screen and look at a different file or a different part of the current file at the same time. To see what this is, well, grab a copy of vim and try it out ! Here I will only list the most important window-related commands. use the ex command help usr_08 in vim to get to the relevant section in the online documentation.

Window-related ex commands
splithorizontally split the current window and display the current buffer (=file) in the new window
vsplitvertically split the current window and display the current buffer in the new window
newhorizontally split the current window and create a new buffer in the new window.
vnewvertically split the current window and create a new buffer in the new window.
onlydestroy all other windows.
only!destroy all other windows, discarding unsaved changes
qdestroy the current window.
qadestroy all windows (exiting vim)
qa!destroy all windows (exiting vim), discarding unsaved changes
Window-related NORMAL commands
<ctrl+w> hmove one window left
<ctrl+w> jmove one window down
<ctrl+w> kmove one window up
<ctrl+w> lmove one window right

back to top

Final Words

As you have seen, vi is a complicated, unusual editor with many annoyances to the normal user. Surprisingly, the vi editing model has greatly improved productivity of many, and, if you edit much, like a programmer, could increase your efficiency. But if you actually do decide to seriously learn vi or vim, try to use it exclusively, at least for a time. And, above all, stay away from INSERT mode as much as you can. over-using INSERT mode won't help in the slightest if you want to really use it.


The Telephone isn't all bad, after all.

When I was a child, I severely disliked talking on the telephone, especially when I was the caller. I don't really know why this was — maybe I had an issue with talking to people I can't see, or maybe I just didn't like talking to people in general; not talking to somebody is so much easier when they're not there. I certainly used to be at odds with the idea that there's no way of knowing which member of the household you're calling will pick up the phone, but that issue was soon fixed by mobile phones. Anyway, I may not always have been the telephone's biggest fan, but today, I would like to stress what a great thing it really is.

Public Telephones.

Photograph: nouspique. (CC BY-NC-SA)

With the spread of the mobile phone and the internet, quick text-based communication has become more and more widespread. In the business world, the e-mail has replaced not only the fax but also many a quick phone call, while in my age group, everything seams to revolve around Facebook messages. It has been argued that an e-mail much more civilised than a phone call: by calling somebody, you are asking them to stop whatever they're doing to occupy themselves with whatever you have to say, and, after all, who are you to say that you're more important than whatever they were working on? An e-mail affords the recipient the liberty of staying in control of their own precious time.

This inherent asynchrony of e-mail can be an advantage, especially in cases where you want a thought-through response, but it can be a distinct disadvantage in discussions that lend themselves well to certain possibilities that an immediate spoken dialog offers. For example, when communicating by e-mail, you cannot interject, in the middle of an argument, “Yes, I've thought of that and see where you are going, but have you considered …”. When considering a philosophical point, it may not hurt to always write out an argument in full, as did the great thinkers in their letters in times when mail was not transported by glass fibre, but by horse-drawn carriages, but when you're trying to arrange a get-together, the immediate back-and-forth of a phone call can save an astounding amount of time.

Why? That is difficult to say. I think the fact that anything uttered gets feedback immediately changes a lot: it allows us to think together, to, in a way, synchronize our trains of thought to a certain extent. It also allows suggestions to be shot down or reaffirmed immediately, which can allow a consensus to be formed in minutes instead of days. There are also some more subtle psychological arguments that might be made: Firstly, a phone call is generally a complete conversation, while an e-mail can be just a tiny part of one, which makes losing focus quite easy. Secondly, it is extremely easy to just quickly skim an email and decide to think about and answer it later, and then forget, or ignore, the whole thing for a few hours (or days) too many. A phone call may be more intrusive, but it does get the attention it needs

Finally, to round the post off, the inspirational example: yesterday evening, a bunch of phone calls in the space of about ten minutes organized a time (“in half an hour”), place (“I dunno, at my place?”), and four Schafkopf players. On the other hand, there's a thread on my Facebook account that has been trying to go to the cinema for weeks.

As always, feel free to leave your thoughts in the comments.


Taming Thunderbird

After setting up Mozilla Thunderbird 3 on my netbook, one of my favourite e-mail clients (alongside KDE's good old KMail), and finally managing to get it to behave, I thought I'd share some tips. I'm using Thunderbird 3.1.10 as shipped by Ubuntu.

Let my server do the spam filtering, okay?

I have a well-trained spamassassin set-up running on my own IMAP server, which takes care of filtering spam out of my inbox, and learns from a "spam" folder I populate by hand (and with a honeypot). Now Thunderbird has its own junk mail controls, and it is actually possible to teach these to play along.

First of all, I want Thunderbird's prominent "junk" button to train good old spamassassin, not just its own filter, by moving messages to the designated spam folder: in Preferences → Security → Junk, select When I mark […] Move them to the account's "Junk" folder.

The next problem is telling Thunderbird that the "Junk" folder is not actually called "Junk". This is done in the account's settings, on the Junk Settings page. Here, you can not only select the folder junk goes into, but also turn Thunderbird's own filter off, in case you're like me and think that one spam filter should bloody well be enough.

When I say delete…

By default, when you delete a message in Thunderbird, it isn't "expunged" until you close the program. This has the unfortunate effect that, when you (re)move messages on one computer, leave it on, and check your inbox on another computer, or your phone, those messages will still be there. This can be extremely confusing, or, at the very least, somewhat annoying. There is a (well-hidden) solution, though (thanks, Ryan): set mail.imap.expunge_after_delete to true in the config editor (Preferences → Advanced → Config Editor).

My archive is my castle

Changing which folder to use as an archive folder is fairly straightforward: it is set on the same configuration page as the "Sent" folder, in case you need to change that: Copies & Folders in the account's settings.

Thunderbird, by default, splits the archive up by year. I'm fine with this setting myself, but I've done some digging and it appears that there are some hidden configuration options to change this. A few weeks ago, Jim Porter added a graphical configuration screen to the upstream Mozilla source code which I expect to see in a release soon. (bug/commit)

Woah, you want to index everything?!

Thunderbird's global search is, in principle, a great feature, but there are some things I just don't want to see when I search for specific messages. This is, especially, spam, and I also like to keep my mailing list archives separate from the body of mail I may want to search. Luckily, Thunderbird lets you exclude specific folders from the global search in the individual folder's properties.

Encrypt, sign

E-Mail is an inherently insecure medium, even if you don't use Yahoo! mail. The situation, however, can be vastly improved by using the (comparatively) popular GnuPG software. For Thunderbird, you'll want the Enigmail extension. I won't waste any more words on it here as it has an excellent configuration wizard and I'm sure you can find more information without me if you want or need it.

Consider the android

Since I got an Android phone, my contacts have lived in Google's ominous data banks. My phone can synchronise with Google, and, as it turns out, so can Thunderbird! The extension I'm using is called Google Contacts (unimaginative but obvious), and it works like a charm. It creates a new address book and takes care that it stays synchronised with Google, both ways. It even supports multiple Google accounts, for the double-faced among us.

The Lightning calendar add-on can also synchronize with your Google account using the Google calendar provider extension. There's a nice graphical tutorial here on how to configure that.

That's it for today. Feel free to leave feedback and suggestions in the comments.


from hell import interesting_revelations

Somewhat inspired by the philosophical thickets in the depths of one of the more fundamental discussions on python-list aka comp.lang.python, I wrote a little function in C that grossly violates the Python object model's integrity and swaps two object structures in-place. What I wasn't expecting is that this can be used to illustrate some interesting facets of the CPython internals.

The original version looked like this:

static PyObject *
swap(PyObject *self, PyObject *args)
{
    PyObject *obj1, *obj2;
    Py_ssize_t len;
    PyObject *temp;

    if (!PyArg_ParseTuple(args, "OO", &obj1, &obj2)) {
        return NULL;
    }

    len = obj1->ob_type->tp_basicsize;
    if (obj2->ob_type->tp_basicsize != len) {
        PyErr_SetString(PyExc_TypeError, "types have different sizes (incompatible)");
        return NULL;
    }

    temp = PyMem_Malloc(len);
    memcpy(temp, obj1, len);
    memcpy(obj1, obj2, len);
    memcpy(obj2, temp, len);
    obj2->ob_refcnt = obj1->ob_refcnt;
    obj1->ob_refcnt = temp->ob_refcnt;

    Py_INCREF(Py_None);
    return Py_None;
}

Simple: get the object size in memory, and swap using a temporary variable. This sort of works — but not quite.

Python 3.1.2 (release31-maint, Jul  8 2010, 09:18:08) 
[GCC 4.4.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from hell import swap
>>> a = "this is the first string"
>>> b = "this is the second string!"
>>> swap(a,b)
>>> a
'this is the second string!'
>>> b
'this is the first string'
>>> t1 = (1,2,3)
>>> t2 = (a,)
>>> swap(t1, t2)
>>> t1
(1,)
>>> t2
zsh: segmentation fault  python3

As you can see, it swapped the strings without any problems (I'll show you some problems further below), but it behaved strangely with the tuples: the new t1 does have only one element, like the old t2, but that one element is the first element of the old t1! Also, what the flip happens when you try to access t2?

Turns out tuple is a variable-size type. That means it can be created with any number of items, and have an according size in memory depending on how large it has to be. My original code only respected the “basic size” of the type, meaning that, in the case of tuples, it copied the information on how many items there are, but not the actual items. When trying to print t2, Python reads beyond the end of the tuple structure, probably dereferences an invalid pointer, and dies a painful death.

On a side note, Python's list type is, contrary to what you might expect, not a variable-size type — it cannot be, since, in the case of variable-size types, the length must be known when the object is created (and allocated), and can never change. (The reason is that realloc(3)-ing an object might move it, which would invalidate pointers, which is when all hell would break loose). Lists don't keep their items in the actual object structure, they simply keep a pointer.

Armed with the knowledge of variable-size types, we can fix hell.swap to work for tuples:

    len1 = obj1->ob_type->tp_basicsize
           + ((PyVarObject*)obj1)->ob_size * obj1->ob_type->tp_itemsize;

    len2 = obj2->ob_type->tp_basicsize
           + ((PyVarObject*)obj2)->ob_size * obj2->ob_type->tp_itemsize;

    if (len1 != len2) {
        PyErr_SetString(PyExc_TypeError, "objects have different sizes (incompatible)");
        return NULL;
    }

    temp = PyMem_Malloc(len1);
    memcpy(temp, obj1, len1);
    memcpy(obj1, obj2, len1);
    memcpy(obj2, temp, len1);
    obj2->ob_refcnt = obj1->ob_refcnt;
    obj1->ob_refcnt = temp->ob_refcnt;
    PyMem_Free(temp);

Recompile, and we're ready for more apocalyptic idiocy. This time, after checking that tuples actually work as expected, we will be swapping strings in the wrong place to the great detriment of our sanity.

Python 3.1.2 (release31-maint, Jul  8 2010, 09:18:08) 
[GCC 4.4.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from hell import swap
>>> t1, t2, t3 = (1,2,3), (None,), ("a", "b", "erm...")
>>> swap(t1, t2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: objects have different sizes (incompatible)
>>> swap(t1, t3)
>>> t1
('a', 'b', 'erm...')
>>> t3
(1, 2, 3)
>>> s = set(t1)
>>> swap(t1[0], t1[1])
>>> s
{'b', 'erm...', 'a'}
>>> 'b' in s
False
>>> 'a' in s
False
>>> 'erm...' in s
True
>>> 'a' in list(s)
True
>>> 'b' in list(s)
True
>>> 

Okay, erm, what? It looks like it's there, but it's not, but then it is? There is, of course, a simple explanation for this:

sets (like dicts) are, for speed, implemented as a hash table. When you look up something in a set or dict, it first calculates a hash, and then searches for that. However, since it's possible for two objects to have the same hash, it also checks for equality. You will only get a result when there is an object around with both the same hash and is equal.

So, what happens here is: when you execute 'a' in s, the hash of 'a' is calculated, and all the items of the set that are referred to by that hash are checked whether they actually are 'a'. Since swapping, however, the hash of 'a' is associated with 'b' and vice versa — the set is corrupted because it correctly assumes that the hash of an object either never changed or does not, in fact, exist (lists, for example, aren't hashable at all, since they're mutable, and the hash would have to change when the object changes, which would defeat the whole point).

There you have it: that's what you get when you muck around in Python's memory.

I've uploaded the source code to JollyBOX code. Use it wisely.

>> from hell import swap
>>> swap(str, int)
zsh: segmentation fault  python3
% ]]>


Some thoughts on proprietary software

Just now I read Bradley Kuhn's recent blog post entitled Proprietary Software Licensing Produces No New Value In Society. The argument made is, in essence, that by receiving money for a proprietary license, a developer is paid without doing any work. This argument is, in its simplicity, quite pre-industrial in nature and fundamentally flawed. Let me explain:

Bradley, in your post, you compared software development to constructing houses. The problem with this is that houses aren't copied - they're singletons. We need a better analogy.

Think suits

Let's say you want to buy a new suit. You have a couple of fundamentally different options: you can either contact a tailor, and pay them to make one. This is a very simple model: the tailor works on a suit, knowing that they'll be paid. In the end, you pay them for the actual work involved. It makes a lot of sense. This is akin to custom software development, where one is paid by the hour.

However, there is a cheaper alternative: go to a store and buy a ready-to-wear, off-the-shelf product. It's probably good enough, and you'll pay a lot less. You're actually getting value for money, I'm sure you'll agree that it's perfectly reasonable to pay for this. However, the way the money flows is a lot less direct and obvious:

At the beginning, someone designed the suit you're buying, without being paid (or being paid by a company that isn't getting paid yet). Someone set up a production line, without being paid directly, on the mere speculation that someone might buy the suit. And now, you, the customer, are (in addition to the manufacturing and distribution costs, which don't exist in software development) retroactively paying the designer for the work they might have done years ago.

Instead of clothing, I could have used any number of other examples, such as any kind of engineered hardware, or even books. However, nobody buys custom-tailored books.

With software, in addition to financing speculative work done in the past without direct remuneration, you're usually paying for support, for bug-fixes, and for future upgrades: You are, actually, helping to finance continued work. Here, I'm mostly thinking of small software development shops, not so much big corporations like Oracle or Microsoft. For more of an insider's perspective (I myself am a student and have experience only with custom (web) software development and free software projects), I can recommend a nice article by Virgil Dupras of Hardcoded Software, recently linked on the python-dev list.

There is an ongoing micro-discussion on identi.ca that might interest you.

As a small clarification: I support free software, but I think that a strict interpretation of freedoms 2 and 3 can have its problems in a world governed by markets and money.


On the evolution of snakes.

It's been a number of years since I first learned programming in Python with Mark Pilgrim's excellent, but now somewhat outdated, book, Dive Into Python. It has managed to become outdated because the Python language is being developed and improved all the time and new features are being added. One of the best features of Python is, beside the standard libraries, arguably, the documentation, which is good enough to include What's New documents for every release.

I've decided to have a look at the backlog of new features, and consider how I use Python today in ways that simply didn't exist when I originally came across the language.

Read about my findings after the break. (Technical language is used. Knowledge of Python and its features is presumed.)

Read more...


Creating the perfect keyboard layout

After having read this post's title, you might have though “ah, he'll be presenting [insert favourite subculture keyboard layout eg neo]! nice!”. If so, you'd be wrong. If, on the other hand, you're thinking “perfect keyboard layout? There's no such thing!”, then I couldn't agree more. Anyway, …

I have been using the standard US keyboard layout for years, almost always without actually using an American keyboard. The main reason I chose it over the German layout is that characters like []{}\|/`, used in many programming languages, are placed in a civilized manner, meaning I can type them quickly and without breaking my fingers.

The standard US layout has a certain problem, though: when it comes to typing in languages that don't happen to be English, it fails spectacularly. Since I have to write quite a lot of German and, nowadays, French, on my computer(s), this is quite a drawback.

Umn, I fixed it

It's not that hard to create your own keyboard layout, which I have done. I chose to use a standard US layout as base, leaving every single key binding intact, using level-3, i.e. AltGr, bindings to represent missing characters.

The German umlauts and ligature ÄäÖöÜüßẞ (the last character is the capital ß) I decided to map to the most obvious places imaginable: on the A, O, U and S keys, so AltGr+Shift+U produces Ü.

I created support for most romance languages by adding Çç to the C key (as above), Ññ to the N key, and a number of hidden dead keys: AltGr+' e renders é, AltGr+" e renders ë; the keys for `,~ and ^ act equivalently. The characters Ææ,Œœ,Øø and Åå are on the W,I,Q and Z keys, respectively, ensuring full support for French, Danish, Norwegian, Swedish, and probably other languages. The Esperanto alphabet is completed by the dead circumflex ˆ and AltGr+y, rendering ŭ. The Dutch IJ and ij are on the J key. Some other possibly useful characters, beside the quotes „ « » ‹ › “ ” (on the [] and <> keys), are ¿ ¡ € £ ‰ ¹²³⁴⁵⁶⁷⁸⁹⁰ ≠ ± Ππ Μμ Δδ Ττ ° § ¦ – —. If you're really interested in the details of the layout, please, try it out!

Yes, you can have it

For X11 (Linux and other Unices): github (tarball)

For Microsoft Windows: us_tj2c.zip. (older version, missing a number of characters. German, French, and Spanish are supported equally.)


Bringing synergy to the table

Once again, my desk is rather clean (a state that doesn't tend to last for long, let's see how long it holds this time), and I have decided to reintroduce a multi-head setup at my workstation. I've done this twice, and always enjoyed the space, and now I've also seen what it's like to have only a 20in wide flat screen sitting in front of you. In addition to that, I made sure there was a place for my laptop, as the recent acquisition of a PCMCIA network card made it desktop-compatible once again.

my desktop

To spice things up a bit, and, I'll be honest, to spare myself the tedious labour of stretching toward my laptop whenever I want to use it, I've started using one quite magnificent piece of software called Synergy. What Synergy does is, quite simply, allow me to control my laptop with my desktop's keyboard and mouse (over the network). This feels just like having multiple X11 displays: I just move my mouse pointer off the left edge of my big screen and focus is on my laptop. This allows me, for instance, to have an IRC client running on the wee laptop screen without really being distracting :-)

By the way, the video deliberately placed on the right-hand screen in the photograph is one of Julien Doré's extremely arty «Les Limites»-clips. Grep Youtube for it if you will.


Impressum

spam goes here