Forum Posts

Done. Thanks for discovering this!

Hi Max, Your questions have prompted a lot of discussion around here, and answering your questions created a long post. Rather than give exact mathematical answers (which I am happy to do later if ... [More] you are still interested), I'll explain the underlying principles and how they lead to the results you see. First, I checked for bugs, and I've confirmed that everything is working as designed (although perhaps not as expected!) Currently, the main shortcoming with KudoRank is sparse data. We're running a ranking algorithm on nearly 50,000 participants with only about 2000 people actively contributing data. The sparse data means that a large number of people have very similar scores. While the difference between a KudoRank 10 and a KudoRank 9 is pretty large, the differences are tiny after that. For the particular accounts you've highlighted the underlying scores are so close that it's a virtual tie, and a difference of several decimal points determines the ordering. As time goes on, problems from sparse data are diminishing, and primarily I want to exercise patience and let the data sort itself out. However, I think we have at least one design flaw which causes nearly everything you found -- which is: On the KudoRank wiki page, Rule 3 declares, "Giving away more Kudos dilutes your opinion". The intent of this rule is to prevent one person from stuffing the ballot box with thousands of Kudos. However, we actually have the opposite problem: a person who gives only a single Kudo has a very powerful impact on the results, as compared to a typical user who might hand out several dozen Kudos. We use a simple 1/n rule: if you hand out 2 Kudos, you pass along 1/2 of your influence to each recipient. If you hand out 10 Kudos, each recipient gets 1/10. This creates the undesirable scenario you found. Mr. 4513 manages to score very highly with only a single Kudo from someone with lowly KudoRank 1. Meanwhile, Mr. 4197 has received Kudos from several people, including someone with KudoRank 10 and someone with KudoRank 9, and yet does not manage to rank as highly. Surely this can't be right! The explanation is that Mr. 4513 is sole recipient of all Kudo influence from account 4360, getting 100% of the available boost. However, Mr. 4197 is receiving relatively small slices of influence from his Kudo senders. For example, account 1194 has given Kudos to 29 people, dividing his influece so many ways that even though he has KudoRank 10, his net influence on any one kudo recipient turns out to be smaller than a single, undivided contribution from someone with KudoRank 1. I think this is broken, and it isn't something that will correct itself as we collect more data. Some possible fixes: Cap the 1/n rule, giving n some minimum value. For instance, you can never give more than 10% of your influence to any individual. This would prevent one KudoRank 1 person from launching another KudoRank 1 person all the way up to KudoRank 9 with a single action. Remove the 1/n rule completely, which lets you give your full Kudo influence to as many people as you choose. This means that, as expected, a Kudo from someone with KudoRank 10 would be worth much more than a Kudo from someone with KudoRank 1, no matter how many other Kudos those senders might be passing around. I like this option because it simplifies, rather than complicates, the system. Finally, understanding Mr. 4267 comes down to understanding how we integrate stacks in the calculation. When you stack a project, that's equivalent to giving a single Kudo to that project, to be shared by all of its contributors. The Kudo is split up proprotionately among the contributors based on their number of commits. If the project has only a single contributor, that contributor gets the whole Kudo, and it's mathematically equivalent to giving that contributor a regular Kudo directly. This means that Mr. 4267 has received 'nearly' the equivalent of 5 Kudos, because WiX has been stacked 5 times and it is 'nearly' a solo project. How does this put him 100 ranks higher than Mr. 183? Well, that's thanks to the 1/n rule, which dilutes the kudos given to Mr. 4267 and Mr. 183 such that 4267 comes out on top -- but just barely. Their scores are actually quite close. OK, this post is getting quite long, so I'll stop for breath and invite feedback. I've come away with the conclusion that the 1/n rule needs to be addressed in some way, although some thought needs to go into the exact change. Let me know if I've answered your questions well enough, and I'm very interested to hear your thoughts. Thanks, Robin [Less]

Hi w00teh, "Minor hiccups" -- you're so forgiving :-). It's interesting that you would post this now, since I've been spending the last several days focusing on the performance of our server farm ... [More] and trying to speed up the average update time. I've been thinking about this a lot lately. You're right that the biggest problem we deal with (and it's a nice problem to have) is that there are always more projects coming into the system. Practically, this means that we have a continually increasing requirement for additional throughput, whether through better algorithms or simply more hardware. If all we had to do was keep our current projects up-to-date, I'm pretty sure we could get down to daily updates without too much stress. I've been spending the last few days optimizing this side of the equation. The real problem is that when a new project comes in, we have to download its entire history, and sometimes this takes a long time -- for a large project, this can literally take several weeks, during which time that server can do almost nothing else. We've killed some projects after literally a month of steady download. When several big jobs like that come in at the same time, it can reduce our available horsepower for updates to almost zero. Apart from spending hundreds of thousands of dollars on new hardware, I'm not sure what strategy to use for this. We like our new users, and we want new projects on the system to have reports as soon as possible. We always assumed it was better to get a new project online quickly than to update existing projects. However, more and more we're starting to feel that real-time updates are a strategic goal for us. I like the idea of setting specific project update intervals. We probably will attach a timestamp to each repository that says "Don't update me until ...". Probably, this timestamp would be estimated by looking back over the recent commit history, and picking either an hour, a day, a week, or a month. We could wire up a "Please update this project" button on the website to let users request a quicker update when they know there are pending changes. We could also use this timestamp to effectively turn off updates of a repository completely if the user knows that the repository has been "retired" or taken offline. Regarding incremental updates, we've been doing things incrementally from the start. We couldn't survive without it. Not absolutely everything is incremental, but the really expensive parts are. If we literally updated a project every hour, yes, we might have some things to streamline, but for the most part an update costs only a tiny fraction of what a complete download costs. I do think there's room in the future for triggered updates -- someone could install a post-commit hook on their source control server to notify Ohloh of changes and add themselves to the update queue. When we're fast enough to make this relevant, I think this would be pretty straightforward to implement. In the far future of Ohloh, our current relational-database-centric architecture may be completely scrapped and replaced with a "map-reduce/couch-DB" type of universe where metrics are stored in documents instead of database rows. At that point, Ohloh could become completely decentralized, and metrics could be generated anywhere with access to the source control a la SETI@home, and then the final metrics document could be submitted to Ohloh. This allows highly distributed computation, lets us get behind firewalls, and possibly avoids the need to download code to Ohloh at all. If we had it to do over again, this is the road we would follow. I'm still not sure what to do about new projects hogging all our server resources away from regular updates, but we're speeding up the updates. And your idea for specifying update intervals is probably coming next. Just my own ramblings, thanks for your thoughts, Robin [Less]

Hi Michael, Updates are on the way.

Hi kowey, I really think it's great that there is interest in adding Darcs to ohloh_scm. Honestly, everyone here at Ohloh is excited about the prospect. However, I want to be very careful to set ... [More] expectations appropriately. This is a real crunch time for us, and we have no scheduled time available to work on this. I suspect that a functioning Darcs adapter can be knocked out in a single evening. That's the easy part. The real price comes later, in debugging the endless variety of strange crashes and server performance sinks that arise in production deployment. There's no "bare minimum" implementation that is useful, and if undertaken with that attitude it simply will not happen. We need some confidence that we aren't going to to be stuck solving a lot of problems after the rollout, and that we'll get good help from the community when we encounter them. Regardless of our own scope intentions, the public is going to feed every Darcs repository in existence to this adapter, and will expect it to work -- even when given extremely large and/or invalid repositories. You'll be amazed at the variety of abuses people have committed with source control systems. Also, this adapter code has to be well-behaved (low memory, low CPU), even when given extreme repositories, so that it will not interfere with our existing services. This means that the adapter has to be very robust and efficient right from the starting gate, or it's not likely to be deployed. So I'm not saying "no", I'm saying "yes if you really, really mean it and are willing to do some work to help us". If after my stern warnings you're still up for it, I'm happy to slice out some time over the next few weeks and give it a shot. Here are some development pointers: A similar OS will probably help a lot. While ohloh_scm should run on most any POSIX OS, we run Ubuntu 8.04 on our servers and OS X 10.5 on our development machines. Windows is a non-starter. It will take some effort to get the ohloh_scm test suites passing on your development machine -- you have to install 4 different source control systems on your box, and few outside of Ohloh have ever run this code. Passing the unit tests are the first milestone. I am happy to help with this step over IRC. Once you have the test suites passing, I would take a look at the Bazaar or Mercurial adapter (whichever is most similar to Darcs) and use it as a model. I would literally just copy and paste the entire lib/scm/adapters/hg directory and all of the test/unit/hg_* files, start renaming them to darcs, and go from there. For the test suites, you'll need to prepare a couple of small sample repositories that exercise all of the key features, including whatever branching and merging possibilities may arise. Good coverage is essential -- I'll be a real stickler about this. Finally, I know you all are Darcs users, but somehow you'll have to learn to like Git long enough to submit your patches to me, preferably via GitHub :-). I'm happy to answer any questions, explain the big picture of how Ohloh uses this library, and help the process along. I hope there are some Darcs lovers who are willing to commit some time to this! Thanks, Robin [Less]

I've rescheduled the updates. I'll keep an eye on it. Thanks, Robin

Hi Metal3D, An update for project Pholow is running now.

Great!

Hi kost, Sorry, there is currently no way for Ohloh to skip a revision.

Hi ymombrun, I've done some investigation, and can confirm that this is a bug/limitation in Ohloh. The fix won't be simple, so we might be stuck with this behavior for a while. If you're interested ... [More] in the technical details: The commit you've referenced contains a change to an XML file (solr-engine/src/test/resources/request.xml) which adds 6615 lines of new XML and removes 2245 existing lines. This is more than enough to trigger our "large commit" flag. The weblab project on Ohloh is properly configured to ignore this file during our analysis, so it reasonably follows that this XML file should not trigger a "large commit" alert. However, that's not quite the way it works today. Ohloh users can change the project's ignore settings at any time, and more than one Ohloh project may share the same repositories -- each with different ignore settings. For performance reasons, we therefore make the "large commit" decision regardless of any one project's ignore settings. Moreover, when ignore settings on a project are changed, we don't recompute these flags. Clearly this isn't ideal, and it doesn't match what users might reasonably expect. I've opened a bug ticket for this, and I'll talk to the team and see what can be done about this. Thanks, Robin [Less]