Kudos rank calculation flawed?

Hi there,

today I was randomly looking through the Kudos rankings. I figured that looking at people with high Kudos might reveal some interesting new projects etc. Doing so I observed something that to me (after reading the "How is Kudos rank calculated" description) makes no sense. Essentially, I would have expected the following to hold: "If a person gets Kudos from others, and participates a lot in many high-profile projects, (s)he will have a higher Kudos rank than somebody who got no Kudos from others, and contributes less to fewer, lower-ranked projects".

However, that doesn't seem to be the case. I could use my own profile for comparison now (which, to be honest, was my initial reason to write this post -- yes, I am a Karma whore :-), but that's not necessary. To give an example, consider these two persons:

• 4267 (rank 494)
• 4197 (rank 588)
• 183 (rank 595)

I don't understand how Mr 4267 can be 100 ranks higher than Mr. 183). That seems to make no sense -- but maybe I am just missing something?

The most extreme example, however, is this guy: 4513 (rank 400) who has done nothing (well nothing visible on Ohloh at the time of this writing), except for getting Kudos from somebody on Kudos level "1". What?!?

Would be nice if somebody could try to explain these particular examples, and maybe the Kudos ranking description could be clarified a bit. :-)

Ed: linkified post.

Hi Jason,

thanks for the reply, I now look forward to maybe hearing from Robin. I know how difficult it is to get a ranking formula "balanced" (SourceForge.net took years for it, and the original formula was totally flawed). Esp. when you use an iterative definition as you do, or as Google's pagerank uses, it's a hell of a time to get those tuned, and in the end you'll always get some data which is "surprising". There's a reason this kind of algorithms comprises an interesting research field of its own in comp.sci. and math ;-). So good luck on this one.

(Oh and thanks for linkifying my post.)

Hi Robin, thanks for the info, that answers most of my questions. Being a mathematician I am always interested in mathematical details ;-). I dealt a bit with the theory of "fair" voting systems in the past, which has quite some relations to the problem at hand. In particular I was reminded a bit of Arrow's theorem.

For the 1/n rule, let me point out another possibility (I am not trying to promote it, just pointing out another possibility): You could use a "non-linear" scaling, e.g. make the influence of a Kudos proportional to 1/log(n+1) (base 2 logarithm used here). This way giving a single kudos counts fully, giving 16 counts 1/4th, etc. And then of course you could combine this with a capping scheme, too... And other formulas are possible as well, of course. In fact, 1/sqrt(n) might be a good one, too. This would in fact be in line with some results in election theory which imply that the total influence of a group among several groups (think of the population of one state out of a federation of states, like in the US or in the European Union) grows proportional to the square root of its size, instead of growing linearly as one would naively assume. But I didn't do anything to properly research the connection to this result from election theory, it's more a gut feeling, so don't take it too seriously (but it might be worth trying this out anyway).

The alternative of not doing any scaling of the kudos a person gives (i.e. each counts full) is appealing because of its simplicity and clarity. Your voice counts fully, no matter what you do. However, this is not without problems, too, as televoting or webpolls show (or, for that matter, democracies): A few dedicated persons willing to make sure there "voice" is heard can spoil the system. In our case by spending kudos on lots of persons. Even "kudos bombing" would be possible: Create lots of fake accounts, give kudos to each other, and then use this accumulated Kudos power to boost other people (why on earth you would want to do that is beyond me, but my lack of imagination won't stop anybody ;-).

However, this is just randomly playing with the parameters, without real understanding of what's going on. It might be worth to do a bit research and look at the literature whether there are some applicable research results and/or discussions of similar problems...

As for the project kudos: This also seems "bad", but is likewise probably not easy to fix (and possibly even impossible to fully "fix", see ). It "seems" queer that a single contributor to a low profile project would get more kudos than a high profile contributor on a project with many other contributors. But I do understand now what's going on there. This is indeed a difficult problem. Wonderful, I love difficult problems, esp. those which aren't mine ;-).

Cheers, Max

Very interesting discussion, thanks for sharing these thoughts. I'm also very interested in popularity/reputation metrics, but I'm wondering whether you guys have considered the effects of making the specs of the kudo algorithm (partly or completely) public. Most projects I know of keep their ranking algorithm strictly secret to prevent spam (whereas some reverse engineering is tolerable).

Um, dumb question (or clarification rather)

When you stack a project, that's equivalent to giving a single Kudo to that project, to be shared by all of its contributors.

Okie, hypothetical user Joe Bloggs has a kudos rank of 10. Joe has been really impressed by 5 people, and has given each of them kudos, giving each 1/5th of his influence. Joe then hits the project search and stacks 5 projects he couldn't live without.

Now the clarification.... Which of the following is correct?

• Joe is now giving 1/10th of his influence to each of the 5 people and the 5 projects.
• Joe is now giving 1/5th of his influence to each of the people, and each project stacked receives 1/5th of his influence, regardless of how many projects have been stacked.
• As last option, but 1/6th (5 people and a spare "person" for the projects)
• Joe is giving 1/5th influence to the 5 people, and a separately calculated 1/5th to each of the 5 projects

And so on through a few other possible permutations.

The way I see it, Ohloh users fall in to two categories ... those that go round stacking every project they use (which is how I interpret the original intention of the feature), and those who just stack the ones they can't live without or consider their favourites (I fall in to this category).

If the number of projects you've stacked is used in calculating the amount of influence given to people, then the latter group have a massively more effective influence since the former group might stack 50 projects while the latter group may only stack 5. This ends up allowing people to cheat the system and get better kudos by keeping their stacks to a minimum.

If, on the other hand, projects receive a fixed amount of influence (whether that be based on kudos given, kudos rank, or arbitarily fixed), then the lack of voting "value" as opposed to giving kudos opens the possibility of mass project creation/stacking to boost kudos, and also tends to result in the reasonably predictable result that certain projects and people are almost guaranteed to swamp the top of the list (although that isn't a bad thing if that was the aim of the behaviour).

Um ... I've forgotten where I was going with this ... I blame being tired... I'll shut up now :)

IMHO, kudos given to a project should be (at least automatically) weighted proportionally to the size (weight) of the project. Weight of a project should, in its turn, be proportional to its activity (commits per time), quality (documentation), code size, famousness, ... For your last question, yes, I'd appreciate if I could weight my own stack. I use some open source softwares because there aren't another thing better, but I don't think they are as good as they could be.

Another potential problem I just noticed today (as I experienced it myself): Get a couple of your friends to sign up, say three. Then, each of you gives kudos to each other. Don't take too many. Now each of you got kudos from three people, so you get a nice boost in the current system (due to the current "sparse data" situation). Hence you get a nice kudos boost. Which increases the worth of your kudos, which means that your buddies get a bigger boost, which means you get an even bigger boost in the end.

I think it's important to balance the system against this kind of use. This is, in this situation the diminishing effect giving kudos to three people should be balanced against the boost these people get, and the back-boost you get from them in turn.

How much does the kudos rank factor into the kudos I give, anyway? Is the kudos of a rank 9 person 9 times higher than that of a rank 1 person? Hm.

Maybe this has already been covered (or implicitly assumed in the mathematical background), but it sounds from the preceding discussion like the value of "kudo" in general does not change from person to person. The value granted can be diluted by granting multiple kudos but that value is the same for people of different ranks. It seems to me that the opinion of a high ranking developer matters much more than the opinion of a low-ranking developer, such that a high ranking developer giving out a modest amount of kudos should not devalue their opinion so low that it is below even the lowest ranked developer. Is it just a matter of scale - can the existing system be fixed (or at least ameliorated) by simply increasing the value of kudo proportional to ranking? I think there may also be another phenomenon in effect - I would hazard a guess that higher ranked developers are also more popular and familiar with other higher-ranked developers, and therefore are more predisposed to actually grant kudos than a low-ranked, infrequent, developer just doesn't happen to really know anybody else.

Andy:

I think your major concern is that our current system might discourage people from creating large stacks. And you're right that we'd like users to stack all the projects they use -- this gives us the richest data, allowing us to find better suggestions for you and people with more similar stacks.

My concern is more that the data it is producing is effectively weighted heavily away from the people using the system as intended.

Example, mon amigo w00t has 5 items stacked, and is giving kudos to 6 people ... I'd guess he's created his stack on the basis that it contains projects he's directly involved iAlson or (in the case of PHP) relies heavily on. As a result, the influence he's giving is pretty heavy. Compare this with mr webs who has stacked more projects than I wish to count. Since he doesn't seem to have bothered kudos w**ing, he has none in and none out ... meaning he is having an infinitesimally small influence on them.

The developer user who influences people and projects s/he is directly involved in is bound to end up with ridiculous values. The end user that stacks all the projects s/he actually uses (theoretically increasing the project population in the process) has next to no influence on the results.

Personally I don't see this as a problem for the projects by stack counts (which is an interesting and fairly accurate metric that relates as closely to actual user counts as does running /list on freenode). Unfortunately the kudos ranks for users end up completely out of wack.

Proposal: Allow you to weight the value you place on the items in your stack. Perhaps this is just ratings, but make it easy to rate all your projects right from your stack window. This suggestion was sent to us by a user and I thought it was a good idea at the time. But now with your comments, I think it might be useful in itself but also might encourage more people to stack more of their projects.

So I could stack, say, PHP and Putty ... but since I can't really live without PHP but I'm phasing out my use of putty (bye bye windows so bye bye putty), I can give PHP 90% of the kudos and Putty 10%?

I'd probably actually stack more projects if we had separated stacks. Then I could build a stack for my operating base (Gentoo, PHP, Apache and Subversion maybe), one for the tools I use (Firefox, Vim, webdev toolbar, etc), one for my more recreational use (inkscape and gimp spring to mind) .. and one for projects I'm actually involved in coding.

Might be interesting to see an overal list of projects sorted by ranking ... personal stacks ... maybe it would make sense to have some sort of personal importance rating? Then we could make it so it runs distro, webserver, scripting language?

Would probably make sense to have personal importance seperated from our rankings of projects...

DotMG:

IMHO, kudos given to a project should be (at least automatically) weighted proportionally to the size (weight) of the project. Weight of a project should, in its turn, be proportional to its activity (commits per time), quality (documentation), code size, famousness, ...

That sounds painfully complicated. Plus weighting by activity isn't paticularly fair on smaller projects. If you stack KDE or a distro, it'll probably gobble up all your influence. Quality and famousness aren't paticularly good choices for metrics, since they are largely personal opinions.

Max Horn:

I think it's important to balance the system against this kind of use. This is, in this situation the diminishing effect giving kudos to three people should be balanced against the boost these people get, and the back-boost you get from them in turn.

Logically speaking, the rate at which the kudos rank increases will slow down and eventually start decreasing again. It's something of a parabolic curve? Math terms kinda need my head working lol. As the users join, the projects list gets more populated, and the actual commiting and kudos receiving person count increases, people already on the kudos ranks will tend to loose places unless they gain more kudos.

Robin:

As the system works today, if you give out 5 kudos and 5 stacks, you'd be giving 1/10th of your influence to each. I don't like this. I'd rather that you were giving a fixed amount of influence to all 5 of your kudos, and another (probably smaller) fixed amount of influence to each of your stacked projects. I don't want to penalize people for participating and stacking lots of projects.

Yeah, it would make much more sense to have a separate kudos "pools" for user to user kudos and user to project kudos, especially when you combine that with the aforementioned project kudos weighting. Seems kinda counter intuitive having the most people with the least "activity" on the site having the greatest influence on ranks.

Yes, a small group can "log-roll" eachother up the scale -- You can become a 10 with a just a couple of bogus accounts put together in a chain. I need to give it some thought, and it seems counter-intuitive, but I suspect that removing the 1/n scaling factor would actually make this harder to do because it will elevate the scores of "real" people. I need time with paper and pencil on this one.

Sadly I suspect this will give you a headache. Cheating the system with loops can't really be blocked as loops, because all kudos giving eventually produces a looped set of nodes. Looking for loops below a certain size doesn't help because that means you need to arbitrarily pick a size at which a loop should be considered fair ... but then again, I'm sure there are an awful lot of reciprocal kudos links in the database ... person A gets kudos from person B, so he gives it back. You could possibly look for closed loops, where a group of people have created a web of kudos gives that doesn't cause kudos to leave that group, by way of project stack or direct kudos grant, but that still runs the risk of penalising groups of people that don't have much interaction outside their communities. You possibly place a "tax" on kudos loops shorter than, say, 3 or 4 nodes... but that'll probably hit everyone in the database in some way, so the value of it is debatable.

I think valid accounts bouncing each other up the kudos ranks is something anyone will struggle to find a good way of dealing with, since there isn't a decent way of identifying those users. Targeting bogus accounts might be an easier way to remove some overinflated kudos from the ecosystem heh.

It's kinda like trying to rig an automated system to find an lock multis on the mmorpgs that don't allow people to have more than one account. Any user smart enough to have created multiple accounts that are not obviously the same (different email addies, access via proxy's, etc etc) is going to be able to maintain gaming patterns that are not sufficiently different from natural clan behaviours to be targetable by any sort of automated tool. scratches his head

Actually, I don't think it's impossible to get around "loop abuse", i.e. people pimping up each other by forming a complete graph. The problem here is that so far the Kudo ranking has been developed "from the guts", and not by applying rigorous mathematical arguments. This is prone to break / not work, as sometimes "common sense" just fails to find the "right" solution, or even comes up with solution that seem "correct" but actually can be shown to be worse than most other algorithms. Which is why I proposed that one should at least try to look at the literature, I'd not be surprised if there weren't already some papers etc. dealing with similar problems.

As for the stacked-project-kudos, it makes for an example were one could challenge the whole approach. Right now, you have taken an "user centric" view point -- you check how many projects got kudos from a person, and then assign to each some value. But IMO a "project centric" view is also possible, and actually is the one I assumed in use (until I learned in this thread that I was wrong), and which lead me to my "wrong" assumptions in the first post on this thread. Namely, if I look at a project page, and see that 500 people stacked it, I think "Wow, lots of people use it" and would expect that it gets a big boost from this, compared to a project with say 10 people who stacked it. And for this I do not care at all what the Kudos rank of the people who stacked it is.

So if 100 anybodies use project X, while project Y is only used by the the top 4 (according to Kudos rank) developers -- then I definitely would rank X higher. Probably with 10 vs. 4, too. I guess below that I could accept a certain higher influence of the "high profile" devs, but I wouldn't complain if all users were treated equally on this. At most, I'd say a Rank 10 dev should get maybe 20-50% more influence, I'd say (from my guts, too, I admit :-).

Going further, assume project A and B are stacked by 10 people, but the 10 stackers of A each stacke 100 projects, while those 10 stackers of B only stacked B and nothing else. Then I would expect that A and B get the exact same boost. I simply see no justification to do it differently... :-)

Just my two cents, and again, I didn't sit down to work out on a rigorous theory behind this, so I don't claim to be "right" or anything, just stating my thoughts here.

Hi Robin,

thanks once more for the clarifications and explanations, esp. the link to PageRank, and also about the zero sum principle. That makes a whole lot sense.

As for projects not receiving Kudos: Aye, you are right of course (no surprise, you are undoubtedly the expert ;) they don't. And I very badly expressed myself there. My point was more like this: If I stack a project, I assign a certain amount of kudos to each developer of the project. Another way to view this is that I assign some kudos to the project, which is then split between the developers of the project. So, while technically speaking I assign Kudos to the project members, not the project itself, from my point of view I didn't do any such thing, I gave Kudos to the project (I stacked it, thus marking it as "useful to me"). If I now look at the profile of a developer of that project, I do not see the indirect Kudos; I only see the direct kudos, and the size of the stacks. Hence if there are two people without any direct kudos, but each is member of a (different) project which has been stacked 100 times -- then I'd expect them to have equal Kudos rank. And am irritated if they diverge by several hundred ranks.

So, as a conclusion, I guess what I really want to say is this: Naively spoken I would prefer if stacking a project would assign a fixed amount of kudos to the project resp. it's team members, treating all "stackers" equal. However, in light of your words on zero sum games, this might not be possible. Still, it would solve two problems: The "can't understand indirect kudos when looking at person overview page", and the "discourage people from making huge stacks" problem.

As for you developer A vs B question: Tricky one, really. In the end it boils down to defining arbitrary weights, resp. a philosophical one: "Which color is bigger? green or yellow?" well you just can't compare colors that way. Neither can you really compare "stacking" vs. "Kudos", as the relation is different for everybody. It will always require a somewhat arbitrary decision to weld together the two concepts and make them comparable by force ...

If being stacked by 100 people is well above average, then I'd feel this should give a quite big kudos boost. But not as much as direct kudos from 10 "important" devs -- direct kudos is the true kudos, it means "yes, this guy did great work". While stacking a project means "Hey, this project is great", and only by indirection implies the devs (or at least some of them) can't be totally wrong either, given that they developed this great app.

I think this is also one of the core difficulties with KudosRank: It mixes two different concepts and tries to fit them under a single umbrella. More, if you also try to factor in the "activity level" of a project and its member by considering the commits counts...

Cheers, Max

P.S.: Any chance you guys could allow for a little bit wider columns in this forum? It just doesn't seem to be made for people like Robin and me who prefer writing "too long" posts ;-)

Actually, I don't think it's impossible to get around "loop abuse", i.e. people pimping up each other by forming a complete graph.

The problem is doing it in a way that doesn't impact legitimate users giving legitimate kudos.

So if 100 anybodies use project X, while project Y is only used by the the top 4 (according to Kudos rank) developers -- then I definitely would rank X higher.

If the top 4 people on this site shared a common project in their stack, I'd definitely be interested in looking at it to see what makes it special.

Going further, assume project A and B are stacked by 10 people, but the 10 stackers of A each stacke 100 projects, while those 10 stackers of B only stacked B and nothing else. Then I would expect that A and B get the exact same boost. I simply see no justification to do it differently... :-)

There is an argument that says that since the people stacking project A are stacking so many projects, each project in the stack has less importance in their eyes than the over all. Similarly, if you give kudos to 100 people, there is a logical leap that says that those 100 people have less "kudos" in your eyes than if you felt only 10 deserved your kudos. Think of it as you having £100 to give away ... if you split it between 100 people, then those 100 people probably have less need of it in your eyes than if you split it between just 2 people.

Yes that argument is arguably flawed, but that's beside the point. The more critical issue is that ohloh is trying to create opposite trends for user kudos and project kudos, but treating them both with the same system. Specifically, the former is aimed at generating a trend of giving kudos to those we feel most deserve it, while the latter is aimed at giving kudos to every project we use.

If two people kudo eachother, they form a symmetrical system where each takes as much as he gives, and they end up with the same score they began with.

Not quite .... logically, A -> B -> C -> D -> A will cause all 4 accounts to have far more kudos than they started with, because an account with no kudos given to it must still have a non zero amount of kudos to give, and giving kudos doesn't decrease the amount you personally have. Thus each time kudos scores are recalculated, the actual kudos value of each account should work out about the same but increase relative to the general population, which each increase being of a decreasing amount.

Kudos are person-centric, and don't have anything at all to do with ranking projects. Projects don't "get a boost" from kudos.

Kudos and stacks are two separate features. The KudoRank algorithm itself doesn't know anything about stacks, and stacks definitely don't know anything about kudos.

Uh, wait a sec, I thought that having 5 kudos gives and 5 projects stacked resulted in each being given 1/10th of your influence ... that's not very separate.

Also, we feel that being a key developer on a project with a lot of stacks ought to be worth something in the KudoRanks.

Indeed, but stacking a lot of projects shouldn't decrease the value of your opinion on another person.

If being stacked by 100 people is well above average, then I'd feel this should give a quite big kudos boost. But not as much as direct kudos from 10 "important" devs -- direct kudos is the true kudos, it means "yes, this guy did great work". While stacking a project means "Hey, this project is great", and only by indirection implies the devs (or at least some of them) can't be totally wrong either, given that they developed this great app.

Interesting thought ... you could derive kudos from projects based on the number of people stacking that project relative to other projects... although I think I prefer the separate pool of kudos and the the weighted distribution discussed above as a solution.

Oh and yay for bigger columns :D

OK, I just noticed a new wonderful example which highlights some problem with the KudosRank:

Take a look at Mr. 3409. He is ranked 565th at the time I am writing this. He neither received nor gave any direct Kudos. However, he performed 98% of the work on the single project he is member of. This project was stacked by four people -- one Mr. 3409 himself (he's a 9er), and three level 1 guys.

Compare this to e.g. my buddy Mr. 803, right now only at 1344, he's on level 8. Yet he got direct Kudos from three people, and some stack Kudos. Just weird.

I assume the system handles the case where Mr X is in project Y and stacks it, so that he doesn't give himself Kudos that way?

Does the system