Web-Cat Eugenics

During the last few days I learned how to manage my way through the Python source code with some useful tools (IPython, DDD… - Thank you Jeff) and how to rebuild the new secured python interpreter.

After building Brett Cannon’s security work for securing the python interpreter on Linux, I got an error that cost me two days of debugging.

The error that I got from running the new “secure_python.exe” interpreter was: import of controlled_importlib failed!

Debugging the problem while stepping into python core functions started to look hopeless… So I decided that I should give a try and send an email to the one that actually wrote the code (make sense right?) but unfortunately I was answered that he has not touched that branch in about a year.

After googling the problem a while and reading some blogs I figured out that Brett Cannon actually used a Mac so why not giving a try and installing it on one. I headed to Keren Reid’s office and we tried together to install it on her machine. As expected it worked just fine…..

Trying to debug the error, and to understand from where the NULL that breaks everything is returned, started to seem like a trip to an infinite loop inside the deep Python source code of bytes set…… That why we decided to ask “the one that knows everyone”, Greg Wilson, what should we do next. Few seconds after asking him my mailbox had a new email waiting to be replied from David Wolever (that was staying in Brazil meanwhile…). David was so professional that in one hour he already analyzed the problem and told me what should be my next step. The problem was actually found where the functools library is imported (as the strace tool suggested) and an undefined symbol was found: PyExc_TypeError.

Meanwhile I didn’t notice that Jeff Balogh had tried to debug the problem himself and he actually got to the same conclusion but also to the solution.

Do you know the feeling of spending few days on a debugging a problem that could be solved in less than ten second?

Exactly, it is simpler than building Web-CAT:

Adding the “-Xlinker -export-dynamic” option while building secure_python.exe with gcc supplies system specific linker options which GCC does not know how to recognize. (sounds complicated? well, just add the option at the build_secure_python.sh file where secure_python.exe is built)

So after a lot of hours, a lot of source code and a lot of computer scientists, I can proudly announce:

And the winner is…………..

Jeff Balogh !!!!

After a very eventful meeting with Gene Amdur last week, we’ve finally figured out what to do for the rest of the summer.  Karen decided to switch Qi and I, where Qi will now be working on the Eclipse plug-in while I get the reins of Web-CAT, and everything else that comes with it.

We’ve decided to tread slowly, fixing minor bugs to get us familiarized with the innards of Web-CAT.  This week, I’ve fixed the bugs regarding releasing of assignment results and viewing permissions of students.  Apparently, students can see comments and marks added by the TA that is still being marked and has not been released yet (a big no-no).  Interestingly enough, even if marks added by the TA are not explicitly shown if assignments are unreleased, it is still included in the calculation of final mark shown when students view their feedback, which is just a matter of simple arithmetic to find out the “hidden” TA mark.

Next week, I would then start doing more major surgery on Web-CAT, now adding a complete and release mechanism to marking assignments, typical to the way instructors here are used to OLM.  This would provide capabilities for markers to change the status of the student as complete, indicating that it is ready to be released by the instructor for student viewing.  We would then transfer the release capability to the instructor, rather than the TA, of making the completed submission result available to students.  This would involve more changes internally as it would mean adding attributes to the tables that is accessible only through WebObject’s EOF - which is like SQLAlchemy in python, except that it’s meaner and eats children.

Another problem that I have is providing granular restrictions on components in a WebObject page.  As far as we know, Web-CAT only provides access level restrictions on a per-page level and not on individual components.  What this probably means is that we can’t have conditions on components like “display this button only to instructors”, but rather create a whole new page with just that one change and put restrictions accordingly on the page.  This still requires investigation that I’m looking forward to next week.

I read as much material as I could so I could start to have more
knowledge about the structure and the ideas behind Brett Cannon’s
work.

Jeff Balogh helped me a lot and introduced me to some tools for python
that made my life so easy in a second. Now I use IPython instead of the
regular python, it is a plugin that warps the regular distribution and
create new commands and much more features that makes a developers
life like heaven.

In addition I checked out two of Brett Cannon’s project from the python
svn reposatory: bcannon-objcap/, bcannon-sandboxing/.
Despite the names the sandboxing project actually deals with memory allocation
and usage and the objcap project is the security for python.

I spent some time also going over his personal blog and I found out some
post that are very interesting for us:
http://sayspy.blogspot.com/search/label/security

As you can see in one of the blog’s post nobody has tried the sandox except
Brett Cannon himself, although he mantioned that he finished the security work!

My plan for the next days is to continue to look at the code until I will understand
good enough the implementation.

Luke started to test the plug-in with some test cases that I transfered him and
also some of his own. Although he already found some bugs, all the bugs found are
currently not related to the plug-in but to Web-CAT. Everything will be recorded
but all the bugs that are not related to our work will currently remain unfixed.
If we will have additional time after the projects will be ready, we will go over the records
and fix the Web-CAT bugs (after talking to Stephen about it, of course).

Apparently, Eclipse has this weird dependency on Firefox through the SWT library.  After upgrading to Firefox 3 RC1 on Ubuntu 8.04 due to Firebug not working, Eclipse started throwing an SWT error with the following log in <workspace>/.metadata/.log file:

org.eclipse.swt.SWTException:
Failed to execute runnable (org.eclipse.swt.SWTError:
No more handles (java.lang.UnsatisfiedLinkError:
no swt-mozilla-gtk-3349 or swt-mozilla-gtk in
swt.library.path, java.library.path or the jar file))

Luckily, I’m not alone and bearfruit.org suggests solving the problem by typing:

sudo apt-get install xulrunner

or

sudo apt-get install xulrunner-gnome-support

on console, restart Eclipse, and problem solved.

First of all the python plugin for Web-CAT is ready to be tested!!!

(Luke will try to break it next week…)

A full feedback is generated including a colored html version of the student submissions!

Comments can be added to the files from the web browser by the TAs and instructors!

The html version of the student files is actually created so that Web-CAT will know how to refer to the data and  display it with all the inserted comments from the browser. That means that all the files submitted to Web-CAT servers will have to have the same structure.

Next week will be dedicated for research. I will need to learn more about how to sandbox python so that the files will be run in a secure environment on the servers.

Three major meetings had been held this week:

- The first one was with all the summer students/mentors and each one had 60 seconds to explain what he accomplished till know and what is his situation.

- The second meeting was with four TAs that used OLM in the past so that we could get a reasonable feedback for the usuability and interface problems.

- The third was with Gene Amdur, that generously came to give us some piece of advice. We figure out together a schedule for the rest of the project time so that we will be efficient as possible. In addition we got a lot of advice from the experience of a well known figure in the industry (As well stories about Gerg Wilson…).

Eran: trying to add syntax-coloured code view to python plugin, turns out it needs Web-CAT itself to generate the html page. 

Geofery: Generating ideas for Eclipse plugin’s GUI designs, and documenting the prospective API to talk to Web-CAT which Stephen is quick to implement,

Qi (me): Looking at the TODO list from last week’s meeting, going through a scavenger hunt inside Web-CAT’s two (currently buildable) components Core and Grader to see what we can do. So far we added a semi-working raw filebrowser on submission results page, and semi-fixed few UI annoyances in code viewer.

It would seem that things are picking up pace finally…

(Swing, Moderate)

Sometimes I’m happy, sometimes I’m blue,

My disposition is depended on you.

I never mind regressions from the nightlies,

As long as I have, a version in your eyes.

Sometimes I love you, sometimes I hate you,

But when I hate you, it’s because I love you

That’s how I am, so what can I do?

I’m happy when I’m connected to you

SVN, My best friend,

You’re the reason I commit so often,

How? I’m not here to tell you… because I don’t know yet, besides, this is not about Eclipse.

If you’ve spent the last week trying to make WebObjects+WOLips+Eclipse work on Ubuntu, like me, you’ve probably already tried following guides like this one, or this. After a while you find yourself staring at WOLips in Eclipse wondering why the frak does it not recognize all these delicious WebObjects frameworks you freshly extracted from XCode 2.5 with a 17″ Powerbook G4 that is still running good old 10.3.9. This might solve your problem…

(more…)

For the past week, I’ve been struggling with getting OLM installed in Ubuntu, while Qi struggles with getting Web-CAT built. Every time we figure out what’s wrong, it seems that the culprit is always the same: versions.

OLM documentation isn’t that bad, except that they forgot to include some of the dependencies involved. And it turned out that this is very important to make it run. The first mistake that I made was blithefully ignoring the current version of python Ubuntu was hooked on (which was 2.5) and proceeded to install it on the older version 2.4. Once I got to the point of installing modules, I then realize that using ‘apt-get install’ (which was another mistake on its own) installs the modules on the latest version. Turns out that you can specify which version of python you want to use by editing debian-defaults file and specify the version to use. This is when I’ve also discovered easy_install which makes life a bit rosy when installing modules, especially on a specific version. Just execute ez_setup.py to install it on a specific version of python and voila! you got python modules at your fingertips.

Once I got out of the apt-get hell, I then ran into the problem of compiling one of the modules that I need: PyXML. Turns out that it needs another python module (python-dev) that I couldn’t figure out how to install on 2.4 Edit: turns out you can do this with python<version>-dev. After giving up, I finally decided to manage my own python installation (building from scratch) and installed everything in it. I then override my $PATH with my custom python installation. At the end of the day, I finally got OLM up and running (even took me a while to figure out the default authentication to log in), with one exception: nothing was running. This was frustrating after a few days of wrestling through it.

After some laborious debugging, I finally tracked down the error: the version of TurboJson I was using (1.1.1) was in conflict with OLM. Even if the OLM doc specified to use TurboGears 1.0.2.2, installing TurboGears have other module dependencies and that it automatically downloads all the latest versions. The worse part is that OLM hides the error with its AJAX front-end so there was no way to see what’s going on without debugging (firebug not working well with FF3 is another story). Turns out latest version of TurboJson is throwing some AmbiguousMethod that can only be resolved by either changing the OLM source or downgrading TurboJson. So I chose the latter, and everything seems to be working like it should be.

With all this, I think it’s safe to say that we programmers still have a long way to go in resolving dependencies that do not break on every upgrade. The terms design-by-contract and program-to-interface always comes to mind on these occasions. For now the best remedy is to document everything, until someone comes up with a solution to make dependencies resolve itself on its own and make them, as Greg always says, either famous or filthy rich.