excerpt from a discussion about reputation points

I was recently "quoted" on another person's blog. I think it's a first for me. In a blog post about reputations, Gia Lyons referenced some of my points in a discussion on the Jive community forums. I've excerpted my posts from the discussion if you feel like skipping the larger debate. Thanks Gia, my first honor ;-)

From a discussion on this link, my posts:

First Post:

My background is with Jungian psychology for definitions of my terminology, so please bear with me if its new to you.

When trying to design a reputation system, its important to recognize that it's ultimately derived from one's subjective opinion, despite the fact that it has objective pieces and parts. Context is also important.

Objective in Jungian terms means "shared and compromised" to summarize a complicated concept. Subjective means unique and personal. Ultimately the things that I think matter in terms of a person's reputation are going to be different than yours, but ultimately I probably will, in some manner, "poll" people who also know the shared/objective third person, and ask others in shared objective terms, such as "politeness," helpfulness," "participatory-ness," or some other tag or label.

So when we create a measurement, such as "was Joe Schmoe's answer helpful" we're creating a shared, objective measurement. That doesn't speak to Joe Schmoe's tendency to answer questions in the most derisive and patronizing manner possible (and from my personal experience, when dealing with OpenSource forum support systems, that's usually how it happens!).

Now Don Doozer also tends to be neck and neck with Joe Schmoe with the helpfulness of his answers, and he's a lot nicer. It turns out he's an IT guy for a non-profit organization that deals with quite a few computer illiterate people, so there's a background to why. There's nothing in the rating system to give him "nice" points, because I'm depending on a third party (a shared objective intermediary) to create a "niceness" meter.

Now I could create my own niceness meter label to apply to people, but it only contains my ratings, and doesn't benefit from wisdom of other people's opinions of Don's, and thus far, there's no way for me to share it. Now a system could allow for me to publish (share objectively) my subjective metering system, but that's not always desirable (particularly since I don't want Joe to know that I think he isn't nice). One is entitled one's opinion, but you're also entitled to keep it to yourself; and I'm sure we wish more people would exercize that second entitlement sometimes!

But let's say I have a peer group of people with whom I'd like to share my niceness meter, and whose opinions of niceness I hold in high regard. I decide I'd like to incorporate their opinions as well. But I'd like to keep my opinion as the primary meter, and only let their collective opinion "tweak" the result.

But there's a person in my peer group who I know is a poor judge of character, and actually thinks people who are rude are actually being mercifully nice. I'm not a fan of his opinion, despite the fact that he's a great peer in the current context. What's actually happening is that we don't objectively share the definition of "niceness." Do I have the ability to tweak the formula to incorporate my (low) opinion of his (inappropriate) opinion?

But wait, let's say there's a meter that I haven't considered (timeliness of response, for example) that others have considered. Is there a way to "suggest" that meter, or subconsciously incorporate it?

It turns out that if I incorporate that last meter, some dark horse candidate comes around the bend and ends up winning the Kentucky Derby, and that this dark horse happens to have an extremely high reputation in the opinion of both Don and Joe!

This is getting pretty complicated isn't it? Well, so are reputations.

Ultimately, any reputation system that depends on "global" objective/communal/shared systems is going to be gamed and ultimately fail in the relevance game. Once you know the rules, its easy to break them. But its impossible to game a system when it isn't known by the gamer, because that system is private, subjective and personal.

Reputations algorithms are ultimately subjective and therefore any attempt to quantify them must be customizable by the person depending on their results. In order to be customizable, they must be transparent. In order to be useful, they must be understandable by the end user who may not understand algebraic equations.

Reputation is going to be one of many Next Big Things in the next generation of Social Software, and unless it takes context, subjective and objective concepts into account, it will boil down to devolved popularity contests.

Second Post

My first attempt at a reply was lost to the ether, but I think my "keep me logged in" check box should stick this time.

Your kudos and concerns about recommendations are both spot-on. For example, I've been stuck at 90% on my LinkedIn profile because I lack two more recommendations. Now I actually have an issue with that, because it's not my fault - it isn't my responsibility to have a recommendation; the onus is on others to recommend me, not for me to solicit recommendations. I hate grovelling, brown-nosing and otherwise acting as a salesman. For me, the incentive (for the goal of 100%) should be the other way around - recommend 3 others.

It may seem silly to use a Likert scale numeric metric for a recommendation, but it does have some value. If it's one of a few (or several) metrics used in the final rep algorithm, you'll be able to tweak its influence in the final "score" (presuming you have a way to view the 'rithm transparently and can change it). How you tweak that Likert score can be influenced by the commentary that accompanies it.

It's pretty easy to give someone a "high five" when you're enthusiatically entranced by them, but if you're being overly gushy in your commentary (in someone else's opinion), that enthusiasm can be tempered by one's opinion of their opinion.

Let's presume it's a +10/-10 scale with commentary (I prefer 10 point scales over 5, because it allows for more nuance). The final outcome of the score would be affected by

1) my (subjective) relationship "score" (if any) with the recommender.
2) my (subjective) relationship "score" (if any) with the recommendee
3) my (subjective) opinion of the recommendation based on the commentary
4) the (objectively) collected (subjective) opinions of the recommendation
5) pick a metric, any metric.
6) and for their final rep score, how much recommendations factor into the final number

and of course we could go on ad infinitum with other factors, and none are really necessary. Having both a number and commentary provides for both subjective (free-form space to gush) and objective (a shared standard of measurement) factors.

Privacy about recommendations is important as well. There's not much incentive for me to give a negative recommendation if my name is attached - I don't necessarily think I need to own up to my comments, if my aims are altruistic and I'm trying to give constructive criticism. But privacy also allows for libelous commentary. Private recommendations could be allowed with a moderator/broker approval, if keeping things civil is an important aim, and you can even allow for hand-picked brokers/moderators. You could always subjectively "turn off" anonymous commentary but still allow for it.

Again, this is getting exponentially complicated, because it is. Ultimately a remote web-hosted SAAS platform (an objective/shared space) can't handle it, because it's infinitely subjective and requires (IMHO) a directly accessable desktop application. A peer-to-peer (or persona-to-persona) system is ultimately the only way to handle it. That's not a reason for SAAS companies to become despondent however. They can assist in many ways, by enabling the basic metrics and standards, by (objectively) "aggregrating" metrics on my behalf, by storing or processing via HAAS'n'SAAS. I think the best thing that SAAS companies can do is contribute to OpenSource initiatives that enable end-users to view and tweak rep systems, and make their own inhouse rep systems compatible with others and accessable via APIs. Short-sighted companies will fight this, but they'll ultimately lose.
~~~~~~~~~

Some of my first thoughts on this topic can be found on this blog post from June of '07. http://internetpsyche.blogspot.com/2007/06/reputation-systems-and-4-views.html