Sunday, September 14, 2008

2.6 RC 1 is out

If you didn't already know it, 2.6 RC 1 was released two days ago. There's nothing big to report (and there shouldn't be of course!). In fact, the biggest change you may notice is the absence of a 3.0 RC alongside. This is due to the fact that we believe although 2.6 is almost rock solid, 3.0 definitely still needs some more TLC. We are still aiming for a 2.6 final on October 1st, but 3.0 has been pushed back to the 15th. (You can, of course, see the whole release schedule in PEP 361.) Please, please, please take the time to download it and test it with your project; now is when we want to know about grotesque bugs and incompatibilities!

Thursday, September 4, 2008

Fun with 2to3

Recently, I've been working on 2to3 code refactoring tool. It's quite exciting really; how often does automatic editing of source code work?

2to3 is completely written in Python using the stdlib. The main steps in code translation are:

  1. 2to3 is given a Python source file and a list of transformations (in units called fixers) to apply to it.

  2. 2to3 generates a custom parse tree of the source based on a Python grammar that combines elements of 2.x and 3.x's syntax. It takes note of exact indentation and comment so it will be reproduced exactly later.

  3. Each fixer has a pattern that describes the nodes it wants to match in the parse tree. The tree is traversed while asking each fixer if it matches the given node.

  4. If the fixer's pattern matches the node, the fixer is ask to transform the code. The fixer can manipulate the parse tree directly.

  5. A diff against the original source is printed to stdout and optionally written back to the file.

Over the past few weeks, I've written a couple of fixers. It's pretty intuitive once you get the hang of it, but writing good tests is very important because Python's flexible syntax produces many possibilities you fixer must deal with. I also refactored lib2to3, so plugging a different system of fixers is much easier for custom applications. I've also written some documentation on it's usage. I hope to start documenting the API and writing a guide for creating fixers soon, so other people can start making use of lib2to3.

On the Google Open Source Blog

It seems my testing project has made the Google Open Source Blog! Once again, I'd like to give a huge round of applause everyone who made it possible!