Losing my religion

It’s been a while since I haven’t presented a pet project on this site (or really, anything at all), and for a good reason: I haven’t done any. You see, 6 months ago I began a master program in computer science – not software engineering, and as you’ll see the two are quite different from each other. And ever since I started, I hadn’t had any idea of things I could develop in my spare time. I’d like to say it’s because I have too much work to do, but the truth is, I never worked less than since I’m here. Not because there is nothing to do, far from it, but because every day that passes see me losing more and more interest for the computer arts. I’ve lost my fascination for the source code, my thrill at the idea of building from scratch and more globally the curiosity toward the virtual world that characterized me some years earlier.

 

I can see some reasons for that: first, I, like a lot of people (particularly men, it seems), am a creature of passion. Ever since I was a kid I’ve been frequently victim to sudden strikes of interest toward a particular project or idea, spending a lot of time and energy on it at first, then gradually losing interest until I just kinda pushed the whole thing under the rug and forgot about it. I’ve been doing it for everything: hobbies, software projects, video games – hell, even romance in a way. So I now find myself wondering whether this whole software developer thing was just another fad, one that lived for a very long time sure, but still a fad. I don’t really buy it, but I know myself and that definitely sounds like the kind of things I’m capable of.

 

Another, more likely explanation come from the lack of inspiration raised by my studies. Last year, I was in what is called in France a professional bachelor programme. That is, a very vocational programme, aimed at producing ready-to-work technicians. That meant a school system that looked more like high-school than academia, with fixed teachers, regular hours, long days, and a small class of only 20 students. The classes focused on real-world technologies, involved actual developers coming to give lectures, and included plenty of hands-on lab work where we built databases, toyed around with android UIs, and studied the inner working of the JVM and other tools. Sure, we didn’t learn about big-O notation or algorithm theory or that kind of stuff, so some might find those studies undignified, but I find it was both practical, inspiring, and still challenging. I mean, it’s still software development. Just because it’s not entirely theoretical doesn’t mean we were breaking rocks with a stick in a carry.

By contrast, my new studies are about real, actual Computer Science, the likes of which you find taught on american campuses and criticized as useless and too removed from the real world on Hacker News. We’re talking automata, tree traversals and compiler construction here. Some might find it interesting, and if you’re pursuing in research in academia it could be the right choice for your, but for people who actually expect to work in software development, it’s almost entirely useless. I’ve heard people saying that it build your mind, but sorry, I don’t see it. I’ve been programming long before I’ve learned what goes into a compiler, and you don’t have to know all about timed non-deterministic automata networks to be proficient in design patterns, software architecture or other actually useful topics, all of which would be a much better use of time than what we’re “learning” now. As a result, my motivation for my studies have really dropped since the first time I started going to school, 20 years ago (we start that shit early over here).

 

No just… no

Anyway. I’ll have to do it whether I want it or not, since I can’t conceive dropping out, and I don’t have the budgets to add extra semesters to my studies, so I guess I’ll just have to man up and take it. The problem being that it’s what I’ve told myself for months and I’m still almost physically unable to focus on my studies, choosing instead to devote my attention to literally anything else (in this particular case, this blog post, while I should be reading and working on an assignment related to the stuff in that slide up there). My involvement with the VIS is particularly tricky in this regard, as it gives me a lot of things to do, while avoiding the guilt of not working. Hey, I’m still doing stuff! Yeah, except not doing it will not get me kicked out of the school.

I’m going to keep trying and hope that my motivation will come back, but so far, I’m not too confident. We’ll see.

Project: Progagil V8

Last year’s summer was by far the longest one for me. 5 months, from April when I finished my associate degree to October when I started my next. So obviously I started a kickass roadtrip across Europe!

Ha, ha, not really. In fact, I pretty much spent the 5 months at my parent’s house. My nice, cosy parent’s house by the coast, but still.  Fortunately, I wasn’t alone in this endless summer: An old friend of mine was here too, and he contacted me with an opportunity for a summer project. It turned out that his mother was heavily into dog sports, and especially “dog agility”, aka gymnastics (or army drills), for dogs: running, jumping through hoops, crawling inside tunnels, you name it.

There are contests of this thing, where people with way too much time on their hands run along their companions as fast as they can while trying to avoid making mistakes. Every dog sport club in France that wishes to organize an agility contest must be registered with the CNEAC, the “National Commission for Cynophile Education and Activity” (whose tag line is “For the betterment of the canine race” — as you’ll see, those guys kind of sounds like cartoon villains). They coordinate the ranking and contest of every single dog club everywhere in the country, so they must have a rock-solid IT system, right? Well, not really. They do, at least, force the canine clubs to use a single software to store the contests results.

Now this would normally be the point where I show you the face of state-of-the-art canine contest management software, except that it won’t work on my computer. Why ? Because my Windows 7 rig doesn’t support 16-bit instruction sets. Oh yeah, did I mention it’s a black and white text program ?

More like nethack

Either Zork, or dog contests. I’m not sure.

Yep, that’s a DOS program shoddily adapted for Windows. Progagil V6, as it is called, come straight from 1994, a year already blessed with the birth of Justin Bieber (no, really, google it).

Although it has been “maintained” over the years (this major version actually come from 2005, and minor bugfixes and addition might have been added no more than 2 years ago), the software is slow, buggy, cumbersome, requires a third-party (and commercial) software to use modern printers (it was designed with dot matrix printers in mind), stores its data in some sort of tricked out CSV files, and has a lengthy manual that contains the sentence “if you do exactly what I say, you shouldn’t encounter any problem” more times than any program should (Hint: it’s zero. The software adapt to the user, not the other way around).

Every time you start a contest you must spend several hours setting it up carefully, lest you ruin the whole endeavor at the tiniest mistake, and at the end you have to print dozens of results sheets with all sort of useless data and send it through slug mail to the CNEAC headquarter in Paris. “Kafkaian” doesn’t begin to describes it.

So the clubs rightfully complained that Progagil V6 was a huge pain in the ass, and in 2007, the CNEAC begrudgingly agreed that it was time to let the ole’ Progagil V6 go. I say “begrudgingly”, because, of course, it was the brainchild of a senior member of the association, who invested a lot of effort into it and didn’t want to see its baby tossed into the bin. But he eventually folded, and the association contacted a software vendor to try to make a new, non-insane software to manage dog contests.

2 years later, for undisclosed reasons (but I would be surprised if the fault was entirely on the developers’ side), the project failed, and now the CNEAC and the company are tied in a lengthy legal battle. Meanwhile, Progagil “15th anniversary edition” V6 was still in service.

In 2010, a new and younger (read : 50 years-old instead of 70) team of amateur programmers appeared in the ranks of the CNEAC and announced that they’ll do a new version themselves, using such cutting edge technology as PHP and SQL databases. The release of this shining piece of software was slated for summer 2010, then early 2011, then mid 2011… And not only was there no indication that the software was actually a thing (rumors stated that the man responsible for the new software had stopped everything in the spring and decided to restart from scratch), but the few reports of field tests of the software were disastrous. It was clear that Progagil “V7″, at is was temporarily called, would not be ready until a long time, if at all. And for the clubs, that meant several more months or years with a software that had seen the birth and death of everyone that was ever entered in its database (good thing it was canine sport).

Waiting for fry : still better than waiting for progagil

Dogs died waiting for this thing.

So there was, in fact, an opportunity right here. The clubs were pressuring the central administration for change. The association had already proved that they were willing to show some cash for a new version of the program. The only problem was convincing them that we were actually able to deliver, which would be arduous, since, you know, the last guys that told them that were a professional company of developers, while we were only two students on summer vacations. To be honest, even I was dubious of my own ability to actually finish the damn thing, since most of my other projects were left suspended forever in a half-finished state (see: the rest of this blog), and that clearly wouldn’t have been enough if the software was supposed to be used by hundreds of clubs in a near-professional setting.

So we instead decided to stay hidden and wait until we had an already completed version of the software. We would then surface, contact the CNEAC, show them our product (probably by demonstrating it to dog club owners first so we could have a “popular support”), and tell them “So you can pay us right here and now and it’s yours, or you can continue your infighting and dabbling in software development for some years while your clubs are getting more and more pissed off”.

Sure, it wasn’t the greatest plan. It was a shitty plan, even, as :

  1. We didn’t secure any exclusivity on the project, meaning that anyone (the software company, the original developers, but most of all the new guy) could release before us and almost certainly beat us, since they already had contacts inside the association, and
  2. It put us at a disadvantage if we ever came to the sale process. If negotiations went awry, they could simply tell us “Ok, no deal. We can wait until our in-house developer finish its own version. But you, what are you gonna do with your ultra-specialized software? Sell it to another dog club federation? Whoops, looks like we’re the only ones. So what now?”. Sure, we might have been able to contact foreign dog clubs, but this wasn’t something that we were quite ready to pursue, at least not until selling our first copy in France.

So, grabbing the V6′s data files, sheets for a future contest in our town and some volunteers’ complaints, we started working on our version of Progagil, the V8.

The program itself is not really interesting. Glorified CRUD with a bunch of views for entering, sorting and displaying parameters about a contest, some neat statistics and other features, but overall, it didn’t exactly change the world. There are some interesting point, though: this was my first time using SQLite, as I always used MySQL or other “heavy” DBMS before. Installing a MySQL server on every dog club’s secretary’s computer was obviously impractical… or so it seemed, but as you’ll see later, not everyone agrees that it’s the IT equivalent of installing a nuclear reactor in the backyard of the house you’re outfitting with heating equipment. But anyway. Working with SQLite was great. I loved the simplicity and the lightness of the database-as-a-file concept.

Another point of interest, which I still don’t know if I should be proud or ashamed of, is the sort-of-ORM I handcrafted for the program. At first I looked for existing solutions, but it was insane. Everything was either awfully complicated or out-of-date, and most of the ORM were engineered toward big, millions-of-LOC enterprise programs running on Oracle databases. After failing to adapt the NHibernate tutorial to SQLite (it was written for SQL Server), I scrapped the whole thing and decided that dammit, I was going to do it myself! And so did I, by using a mix of reflection and standard parametrized SQL statements.

It was remarkably simple to use; you just created your objects with either an existing id (if you wanted to retrieve data) or no id (if you wanted to create a new one), and it will commit everything to the database for you. Every accessor was then completed with the appropriate SELECT / UPDATE queries, so it was pretty transparent: you just used your object like you’ll do normally, and everything was replicated transparently. Naturally, it lacked every single feature that you can imagine, including ACID transactions, complex queries and a bunch of other basic SQL perks, like sorting or grouping (I remember considering using LINQ, but dropped it for some reason). Also, the performance was, well, not that great. As could be expected for a reflection-powered piece of dark arts hacked together by someone who never took a theoretical programming class and couldn’t find out the big-O of his algorithms if his life depended on it. But it worked.

// This method re-fill the object's fields from the DB.
protected void refresh()
{
    FieldInfo[] fields = this.GetType().GetFields(BindingFlags.NonPublic | BindingFlags.Instance);

    if (id != -1)
    {
        string txtQuery = "SELECT * FROM " + this.GetType().Name + " WHERE id = " + id;
        DataTable data = sql.runQuery(txtQuery);

        // Tis' be dark magic
        for (int i = 0; i < fields.Length - 1; i++)
        {
            //This get the "Cast" method, and define the type of T
            MethodInfo castMethod = this.GetType().GetMethod("Cast", BindingFlags.NonPublic | BindingFlags.Instance).MakeGenericMethod(fields[i].FieldType);

            //And this invoke it
            object castedObject = castMethod.Invoke(this, new object[] { data.Rows[0][getFieldName(fields[i].Name)] });

            fields[i].SetValue(this, castedObject);
        }
    }
}

I warned you it was dirty.

We toiled the whole summer until it started to look like we could make it fly. So we contacted a local contest organizer and asked them if we could use our system along with the official one, so we could try it out with real data, and maybe talk to the right people in the audience while we were at it.

Well, it turned out to be less of a commercial demonstration and more of a live debugging session, because, as Moltke the Elder famously said, no plan survives contact with the enemy. But it was fun, not to mention incredibly enlightening, to see everything that can go wrong where you’re on live current: bad data, duplicate data, missing data, unintended usages, unexpected usages, and of course, a whole bunch of “wow, this UI really sucks when you’re playing with more than 10 elements”.

After that event however, it was starting to look good. I wouldn’t say that I was proud of it, firstly because this is a goddamn dog contest management application, and also because the UI was ridiculous, with giant buttons and unnecessary progress meters, and the inner workings of the app weren’t exactly rock solid. I was (and still am) frustrated by the MVC model, which I think I never managed to implement correctly. Well I hope that I never did at least, because if that’s it, dear god, what a mess.

But anyway, by that point it was good enough to be at least presented to the CNEAC’s brass. Aaaaand of course, that’s when the news broke that the guys behind progagil V7 decided they had had enough of the vaporware label and actually released something. Or at least they started a beta. Or they will. Anyway, people at the CNEAC believed that this was finally over and were understandably reluctant to pony up some dough for a redundant software made by who-knows-who. And in fact, more than a year later, the “new” software has been deployed at least in some clubs, and the first reviews are, let’s put it gently, less than stellar. Mainly because it requires the installation of a whole WAMP stack on the club’s computer. No, really. Because it’s php, you see. The guy only knew php et MySQL and didn’t want to change his habits by chosing something maybe a bit more appropriate to the task at hand, so he’s now pushing this insanity to every single club in the country.

You know what, I think I better keep maintaining our version so we can sell it in 2027 when they’ll finally have had enough of this one.

Project: EDT Watcher

EDT Watcher (a rather nonsensical working name, “EDT” standing for “Emploi du Temps”, a timetable) is a Java library and an unfinished Android app designed to access the timetables of my home university of Nantes in a more user-friendly manner.

 

Nantes uses a web-based timetable system by Celcat for almost every department (engineering, humanities, medical school, etc). Those timetable are comprised of a large (~ 7000 lines for my former class, though we were not the busiest one) XML file detailing the lessons and events of the whole year (although it starts out empty and is then edited throughout the year as lessons are scheduled), and a bit of HTML/CSS and Javascript to make it easier to browse. Quite simple, and fine for most use cases (eg., looking at your next lessons or visualizing your week workload). However, it lacked some advanced features, and more importantly, didn’t display on mobile phones, at least not android phones.

So I started working on a small Java program to retrieve the XML file, parse it, and store the data in appropriate beans so it can be used later. Nearly the whole university is using Celcat’s system, which is a huge advantage: I start by regexin’ this page to get every department’s timetable’s sub-site’s url, and then I go to the department’s site’s group index (typically the url, + “/gindex.html”, for instance this is the group list for my department), and, if it’s not password protected, I can access the groups’ timetables. Now I can easily get the XML file for a selected group and process it.

After that, the group’s data is stored inside a Timetable object, containing an array of 52 Week objects, themselves containing an EnumMap mapping weekdays to Day objects, which hold the lessons in an ArrayList. These lessons comprises a list of teachers, rooms and modules (collectively grouped in a Resource class), a start and end times, and an ascending link to their Day.

I have to admit that it is not the greatest architecture I have ever designed, and it might probably be easier to do away with the Week and Weekdays abstraction and just list the days with their lessons in a collection, but it allow to easily perform operations on a week, and it’s the easiest way to map to the original XML files, which don’t mention days, but use weekdays and week numbers instead. Well, not even week numbers, but a 52-chars-long string with either “N” or “Y” symbolizing the position of the week in the year. Weird.

All in all, the original material seems to be of dubious quality, with esoteric formatting and a lot of redundant information, but I tried to do my best. Also, there was a lot of date and time stuff and the Java native tools in this category are… a bit lacking, so I used the popular Joda Time library instead, and it was a huge improvement.

Once I finished the whole parsing thing I added a command-line interface and a Google Agenda exporter, and then I started to work on the Android app, and, well, that’s where I stalled. I did a welcome screen, two sets of list to browse through departments and then groups, and a screen indicating the next lesson, its location and data (teachers, start and end time…)

It was actually pretty useful, and I used it often throughout the year to find my room without having to walk through half the campus to read the only timetable displayed in a corridor, but it lacks advanced features that I wanted to implement, such as a week / month timetable, statistics, and maybe an alarm clock based on your actual timetable.

 

As I said elsewhere, I’ll be leaving Nantes University soon, so I’m unsure about releasing it because it would means that I might get tangled up in support mails and possibly legal issues with the University and/or Celcat. However I still want to work on it, and it would be a shame to let it go to waste, since unlike most of my other projects, this one has real, demonstrated use cases. I’ll think about working on it during my last months here, and either release it in a user-friendly form if I think I can handle the support or find a successor / publish the source code on github before leaving. What do you think ?

Project: Pong Fight

Pong Fight (named like that because it was supposed to be AI vs AI) is a Java pong remake that I made during my sophomore year.

Part of our exams included creating 5 mini-projects illustrating the different skills that we acquired during the two years of our programme, including object-oriented programming, imperative programming, database integration, GUI-making and so forth. Most people made simple CRUD applications, and indeed this was the case of two of my 5 projects. The others included a very simple text encryption software, made 2 years earlier (thus being my first real project), and the Casio compiler that I already mentioned, which was even less completed then. And of course when I actually took the exam, the examiners, two old ladies, told me they “hadn’t time for games” and tested me on the filler CRUD software. I got a measly 13/20. What a bunch of dicks.

image du pong

A graphic designer I ain’t.

But back to Pong Fight. It was from a time where we transitioned from procedural programming to OOP, even though we used Java from the start. So yeah, for more than a year we had “static” and “public class” keywords floating around and had absolutely no idea what they did, except randomly generating errors. So when I started the project, everything was in the same class. After a while I divided it, but this was a bit like dividing the code in multiple files in C: every single var and method were static, and the whole thing was completely contrary to everything the language stand for. So when we eventually learned OOP, I tried to restructure it, but it was a massive endeavor, especially since I wasn’t exactly a Java expert then and didn’t want to destroy the project while refactoring (which I did. Multiple times).


/**
* Handles checkbox-style menu
*
* @param x (int) : x position of the click event
* @param x (int) : x position of the click event
* @param size (int) : number of elements in the menu
*/
public int checkboxMenu(int x, int y, int size) {
    for(int i = 0; i< size; i++) {
        if(x >= Main.settingsFile.getSizeX()/8 &&
           x <= Main.settingsFile.getSizeX()/8 + 40 &&
           y >= Main.settingsFile.getSizeY()/5 + i*50 &&
           y <= Main.settingsFile.getSizeY()/5 + 40 + i*50) {
             return i;
           }
    }

   return -1;
}

I made my own game “engine”, meaning that I had to reimplement everything from mouse events to menu widgets. It was fun.

The result is a bizarre accumulation of static elements, singletons and bona-fide classes and objects living in (relative) harmony. Another part of the game that was completely cobbled together was the features: at the beginning I wanted a simple pong game that two basic AIs could play together. But after a while I got caught in the project, and added player support, which meant I had to handle keyboard events. Then I added an (in-game) option panel, which meant that I had to handle mouse events and multiple screens. Then I added sounds. Then I added graphic options. Then I made it possible to resize the window. Then I added  the jewel in the crown: network support. I delved into the Socket javadoc and eventually managed to make my pong playable across two computer in our lab, and then added spectator support, so other computers could connect to a game and watch it. My fellow classmates (and more importantly, my teachers) were in awe, and I got 19/20 at our mock exam when I presented it.

Too bad I didn’t get to present it at the real thing.

Dicks.

Project: Casio Basic Editor

I guess most of us had one of these ridiculously outdated graphing calculators in high school or college. You know what I’m talking about: 2 colors? 64K memory? One of the handful of items that still runs on AA batteries? Yes, these ones. Now, I guess real geeks had a TI calculator of some sort, or maybe another, more street cred-worthy brand. Well, I didn’t. I had a Casio. Specifically, a Casio Graph 35+, better know in the rest of the world as a Fx-9750 Plus:

casio fx9750

Back then, "power graphic" didn't exactly mean what you'd expect.

It boasted a badass 128*64 screen, 763 features (of which I used maybe 30 in my whole education), the ability to program in a BASIC-like language, and absolutely zero computer connectivity.  It only had a proprietary socket that allowed you to connect your Fx-9750 to other Casio calculators. Like you would connect Gameboys to exchange pokemons. Only, you know, sadder.

Since then they added a USB port, so for the low price of $80, you can allow your computer to discover how it feels to be stuck in the nineties. But seriously, that would have been a great tool to export and import some of the aforementioned Casio code to a computer where you could have edited it, tested it and so forth. Sadly, nothing of the sort existed back then, so I decided that even though there was nothing I could do on the connectivity part (short of jury-rigging a usb-to-casio cable, which apparently is possible, though I wouldn’t risk it), I could try to act on the edit and test part.

So I fired up Eclipse and set to work on a Casio BASIC Editor, with a poor man’s IDE, a simplified emulator and a compiler (well, interpreter, technically). It was a fun project, but also a hard one, since I had no previous experience in compilers, languages and that sort of things. I was helped by a friend of mine, and at the end I managed to obtain something that wasn’t too shabby. I divided the application in several parts :

 

  • A system unit, simulating the memory and screen of the calculator,
...

public final static int SIZEX = 128;
public final static int SIZEY = 64;

public final static int NB_FILES = 6;
public final static int NB_LISTS = 6;

/*
* The user has access to 60 kB of memory.
* Each number in the list represent 10 bytes of memory,
* therefore the theoretical maximum size of a list
* is 6 000 numbers.
*/
public final static int NB_ROWS = 6000;

private int currentListFile = 1;

private Stack&lt;Double&gt; stack = new Stack&lt;Double&gt;();
private Stack textStack = new Stack();

private Vars var = new Vars();
private boolean[][] screen = new boolean[SIZEX][SIZEY];
private double[][][] lists = new double[NB_FILES][NB_LISTS][NB_ROWS];

...

 

  • A GUI, displaying the screen of the device, a code editor, and a debug console,

gui

 

  • And obviously, the compilation unit.

The parsing of the code is divided in three parts. First, the code goes through a lexical analyzer that removes the syntaxic sugar and other human-friendly features to reduce the code to its basic elements.

Then, the lexed code is parsed. In my case, it was basically a large switch structure that created a tree of the program. Each instruction was turned into a class, which defined its name,  behavior during execution, and more important, the kind of data or statement that it contains. So for instance, a statement like NUMBER (representing any kind of numeric constant) can only contains a number, but a statement like PLUS contains two parameters, which can be statements too. Some statements are even more complex and involve entire blocks of instructions (IF or LOOP, for instance). Finding and parsing the parameters of each statement is thus a bit complicated, especially since there are infix, prefix and postfix operators, control structures (including GOTOs), and ambiguous statements (the “-” sign can be used both to denote a subtraction, but also to indicate a negative number). So in the end, I obtain a list of statements that contains statements that contains statements, repeated ad infinitum.

Once the list is complete, it executes it. Each statement has an exec() method that represent what it’s supposed to do. Sometime it’s assigning a variable, sometimes it’s writing to the screen, but most of the time it’s retrieving, modifying and pushing numbers from and to the memory.

 

Let’s take an example :

0->A
A+1->A
Locate 1,1,A

This code initializes the variable “A” (there is only 28 variables, A-Z and two special symbols) to 0, then increments it, then prints it on the top left corner of the screen. Once lexed, this resolve to :

NUMBER(0) ASSIGN VAR(A) EOL
VAR(A) PLUS NUMBER(1) ASSIGN VAR(A)  EOL
LOCATE NUMBER(1) COMMA NUMBER(1) COMMA VAR(A) EOL

Numbers and vars are encapsulated in appropriate keywords and instructions are ended by an EOL token, but it’s little more than the uppercase version of the code above. The real magic happens during the compilation:

Assign ENumber(0.0) to EVar(A)
Assign Plus
 (EVar(A), ENumber(1.0)) to EVar(A)
Locate ENumber(1.0), ENumber(1.0), EVar(A)

There are three top-level statement here: the initial assignment, that takes the number parameter and assigns it to the var parameter; the second assignment, that takes a plus parameter and assigns it to the var parameter; and the locate statement, that outputs the var parameter to the two number parameters. Note that the plus statement contains itself two parameters, the var and the number.

The code tree is then executed, which means that “0″ is pushed onto the stack, then moved to the A var, the content of the A var and 1 are pushed onto the stack, they are retrieved, added, pushed to the A var, then 1, 1 and the content of the A var are pushed onto the stack, then used to write to the screen on the tile (1,1). And a beautiful “1″ appear on the corner of the screen.

 

I didn’t finish the project, sadly. Most features are implemented, including “if” statements and some loops, but there are still a bunch of issues that need ironing out, meaning it probably won’t work as intended for anything but the most trivial bit of code. The harder part is done though, so maybe I’ll get back to it one day to finish it, or at least clean it up nicely and release the source code on github.

If you’re interested,  feel free to send me an e-mail (you’ll find it on the main site). I’ll be happy to talk about it, or about similar projects that you or others have been working on.