Project updates

Hi w00teh,

"Minor hiccups" -- you're so forgiving :-).

It's interesting that you would post this now, since I've been spending the last several days focusing on the performance of our server farm and trying to speed up the average update time. I've been thinking about this a lot lately.

You're right that the biggest problem we deal with (and it's a nice problem to have) is that there are always more projects coming into the system. Practically, this means that we have a continually increasing requirement for additional throughput, whether through better algorithms or simply more hardware.

If all we had to do was keep our current projects up-to-date, I'm pretty sure we could get down to daily updates without too much stress. I've been spending the last few days optimizing this side of the equation.

The real problem is that when a new project comes in, we have to download its entire history, and sometimes this takes a long time -- for a large project, this can literally take several weeks, during which time that server can do almost nothing else. We've killed some projects after literally a month of steady download. When several big jobs like that come in at the same time, it can reduce our available horsepower for updates to almost zero.

Apart from spending hundreds of thousands of dollars on new hardware, I'm not sure what strategy to use for this. We like our new users, and we want new projects on the system to have reports as soon as possible. We always assumed it was better to get a new project online quickly than to update existing projects. However, more and more we're starting to feel that real-time updates are a strategic goal for us.

I like the idea of setting specific project update intervals. We probably will attach a timestamp to each repository that says "Don't update me until ...". Probably, this timestamp would be estimated by looking back over the recent commit history, and picking either an hour, a day, a week, or a month. We could wire up a "Please update this project" button on the website to let users request a quicker update when they know there are pending changes. We could also use this timestamp to effectively turn off updates of a repository completely if the user knows that the repository has been "retired" or taken offline.

Regarding incremental updates, we've been doing things incrementally from the start. We couldn't survive without it. Not absolutely everything is incremental, but the really expensive parts are. If we literally updated a project every hour, yes, we might have some things to streamline, but for the most part an update costs only a tiny fraction of what a complete download costs.

I do think there's room in the future for triggered updates -- someone could install a post-commit hook on their source control server to notify Ohloh of changes and add themselves to the update queue. When we're fast enough to make this relevant, I think this would be pretty straightforward to implement.

In the far future of Ohloh, our current relational-database-centric architecture may be completely scrapped and replaced with a "map-reduce/couch-DB" type of universe where metrics are stored in documents instead of database rows. At that point, Ohloh could become completely decentralized, and metrics could be generated anywhere with access to the source control a la SETI@home, and then the final metrics document could be submitted to Ohloh. This allows highly distributed computation, lets us get behind firewalls, and possibly avoids the need to download code to Ohloh at all. If we had it to do over again, this is the road we would follow.

I'm still not sure what to do about new projects hogging all our server resources away from regular updates, but we're speeding up the updates. And your idea for specifying update intervals is probably coming next.

Just my own ramblings, thanks for your thoughts,

Robin

@Guiseppe -

Yes, this is a real problem for us: when Ohcount is updated, it often requires us to recount huge amounts of existing code on Ohloh. We don't have the horsepower for this right now, so projects are inconsistent in the version of Ohcount they've been counted with. We're recounting as we can, and will recount specific projects on request.

@Paolo -

Yes, Ohcount just works on directory snapshots. The work of parsing commit histories and generating diffs is done externally from Ohcount.

@Jumpin-Banana -

That's almost possible. The only problem is that our history parser works forward in history, not backward. We built it in the forward direction so that we could incrementally update as time goes on. I'm not sure what it would take to reverse the direction; an interesting idea.

Just FYI, we're experimenting with a change to the balance between downloading new projects and updating existing projects.

Starting last night, long-running downloads of any kind have only a 1/3 duty cycle: after a download runs for 8 hours, it stop and reschedules itself to resume in 16 hours.

The majority of projects on Ohloh can fully download in less than an hour, and these aren't really a problem for us. However, the really monstrous and slow source control histories will now take much longer to appear on Ohloh for the first time. This will prevent the server farm from getting bogged down by a few big slow downloads.

This has already shown its effect -- last night we updated about 4 times as many projects as the night before.

We are currently running about 4 days back, and getting faster. Let me know which projects are not updating and I'll take a look.