Something is wrong in kudo ranking land

I've had my doubts a long time, but today I finally found a great example where the current kudo rankings simply doesn't reflect how any human being would do the rank. I claim that this is a bug or a flaw in the ranking algorithm. See here:

(I don't know any of my two example persons so I'm not actually biased, and I'm not involved in any of the projects these guys are involved in.)

See today's kudo rank #56 and #57. dsaff and Larry Ewing. dsaff being ranked higher, both being level-10.

dsaff is ranked #56 with only 173 commits in one single project (JUnit). He has received one single kudo from a leven-1 person.

Larry Ewing is ranked #57 with 6800 commits in 10+ projects (several of them being well known and well used). He has receied 23 kudos, and three of them from level-10 people.

Any human being looking at these two would claim that Larry is the more prominent, more experienced and thus should have the higher rank of these two. In fact, I would even say that seeing dsaff ranked as a level-10 with 173 commits is another indication the ranking algo is flawed.

I think that the resistance to change (which will primarily come from ppl whose rank drops due to the new algorithm) could be mitigated by some transparency.

I'd like to see the changes happen sooner rather than later, but as long as you guys put up a page explaining the changes and why you made them (possibly with a basic overview of the new algorithm) I would assume that the reception would be pretty good.

I'm not sure I agree those are the only variables you have to play with. You also have for example amount of commits, the time the project has been around and more.

If a project gets added to many stacks, that is a useful marker for how good and useful a project is. But the question is how to value each contributor in a particular project.

If a popular project is made with very few commits over a short time, the effort doesn't seem to be as big and kudo-awarding compared to a popular project done with thousands of commits over decades. That is, persistence and staying-as-a-contributor is kudo-awarding in my eyes.

And I agree with MotiveForce that most users most likely will accept a totally new rank if the algo is remade and explained properly.

I started to contribute to kde and just signed up on ohloh to get some statistics of my work. Just claiming to be a kde developer led to kudo rank 7 for me. That's ridiculous. How about having to earn you're share of the stack kudos. With now something about 60 commits on kde i don't think i deserve such a kudos ranking.

For me that rendered the kudos worthless. Sorry.

Mike

Maybe it's irrelevant, but the things that get added to "stacks" are things that developers use. No non-developers are going to come to ohlohlcatz and add their software to their stacks. They just aren't going to do it.

So any software which does not target developers won't appear in stacks (e.g. game programs will never appear in stacks.)

So, e.g. John Carmack's work would be rated excessivly low in ohloh's view, since it rates "stacks" heavily, and though his work is in reality, in a lot of peoples "stacks" those people aren't on ohloh making this known.

Consequently, "stacks" are biased towards a certain sort of program -- the sort of program which developers use, or work on and usee, (e.g. vi, gcc, linux kernel, etc...)

I think everybody can agree that in the current form Kudos are completely useless. Just one example: ian%hixie.ch (Contributor to Firefox) actually changed 135 lines of HTML code in 8 commits a few years back (he was fixing some spelling errors in documentation). Nothing more. I'm sure he will be very happy to hear that this makes him a TOP-2% Open Source developer world wide (Place 2400/125000).

Without having looked at the algorithm: A simple solution might be to simply take the ln of the stacks value. That would still disadvantage unknown projects (that exist maybe just to build kudos) but it would also still give a bonus for a higher stacks-value.

Well... Hixie is actually a bad example there, since he's the editor of HTML5 and numerous other web specs and actually *is* a top open source contributor in a nonstandard sense. Not that this means the ranking system works, it just got lucky in this case.

Not really: There are at least 4 ian%hixie.ch accounts, so just because that may match to some real person that deserves the rank for OTHER reasons does not make the ranking of any of the 4 individual accounts any better.

I agree with the previous proposal that "persistence and staying-as-a-contributor" is one of the good indication of the level of participation within a project.

A related argument:

Automatic Kudos for a few short term commit on a popular project might open the door to manipulation (e.g. making a few trivial changes just to get Kudos).

In my opinion, it is better (and safer) to have a few automatic Kudos for long term and consistent contributors. It is less likely that someone will succeed into manipulating Kudos for a given project over a period of years...

\Mario