So, recently a lot of stuff has been going on in my life – displacing the lower status stuff like hacking Bongo, painting the house, washing, shaving, that kind of thing. While I’ve tried to be online to cajole inspire others, that doesn’t really work – apparently my code talks better than I do. Ho hum.

More bad news – it’s looking increasingly likely that I’m not going to get to Ohio Lugfest in the States like I planned. The choice is coming down between “Ohio LugFest” and “New Kitchen”, and somewhat surprisingly my better half is demanding the improved culinary facilities. I’m still going to hit the States this September, though, I just won’t be able to stay quite as long – but I will try to make a point of seeing people while I’m over.

So, onto some good news. I don’t think I mentioned this anywhere except IRC because it’s not totally 100% golden yet, but Bongo will be at the UK Linux Expo 2008 which is happening this October (23rd-25th in London Olympia). We have a stand and I’ve been talking to some of the GNOME UK people about how we can help each other out. I’ve been to most of the UK Linux Expos, in London and elsewhere, and they’re generally extremely good fun and a great opportunity to meet up and make things happen.

Slightly more good news. I’ve finally got my head around this store patch which has been evading me. I’m going to save the technical details for another post later this week, but basically it comes down to a realisation that I had the other day. A number of times working on this I’ve realised my approach was wrong (and almost unworkable, sometimes), so I’ve had a couple of false starts – this weekend I realised that my ideas were wrong, which is worse.

The problem is this: we have store commands, which previously would do some relatively complex things behind the scenes: query through the SQL database, run some stuff in the Lucene index, that kind of thing. Now most of the complexity is basically just hitting the database – virtually all of the time, what we’re doing is translating store commands into some kind of SQL query. And sometimes – this is the harder bit – we also want to add some bespoke query parameters to our commands, to get back even more specific data. I wrote about the problem of parsing these bespoke queries in my last blog post.

What hit me, and seems totally obvious in hindsight, is that I was trying to translate store commands into SQL, these bespoke queries into SQL, and then kinda merge the two. It’s hard, and basically stupid. What I actually want to do is turn every store command into a bespoke query.

This sounds a bit silly, but it actually makes sense. The stuff that we specify in the store commands are basically all properties of the data we want back – e.g., COLLECTIONS won’t return documents, and is equivalent to a bespoke query with a “type=4096” parameter. And thanks to the wonder of boolean logic, if I turn our command into a query expression, I can AND it with the bespoke query, and it all joins up and works.

Initially, I worried that this could – in some circumstances – lead to sub-optimal expressions where we do things like duplicate columns, or express the same criteria twice, or something. Actually – I don’t think we care. SQLite has a query optimiser like any other SQL db, and because it knows which columns have indexes and other useful information, can probably do a damned better job of optimising than my code ever could. So, I’m not going to worry about that – I’m just going to do it and see how it turns out.