excerpt from a discussion about reputation points

I was recently "quoted" on another person's blog. I think it's a first for me. In a blog post about reputations, Gia Lyons referenced some of my points in a discussion on the Jive community forums. I've excerpted my posts from the discussion if you feel like skipping the larger debate. Thanks Gia, my first honor ;-)

From a discussion on this link, my posts:

First Post:

My background is with Jungian psychology for definitions of my terminology, so please bear with me if its new to you.

When trying to design a reputation system, its important to recognize that it's ultimately derived from one's subjective opinion, despite the fact that it has objective pieces and parts. Context is also important.

Objective in Jungian terms means "shared and compromised" to summarize a complicated concept. Subjective means unique and personal. Ultimately the things that I think matter in terms of a person's reputation are going to be different than yours, but ultimately I probably will, in some manner, "poll" people who also know the shared/objective third person, and ask others in shared objective terms, such as "politeness," helpfulness," "participatory-ness," or some other tag or label.

So when we create a measurement, such as "was Joe Schmoe's answer helpful" we're creating a shared, objective measurement. That doesn't speak to Joe Schmoe's tendency to answer questions in the most derisive and patronizing manner possible (and from my personal experience, when dealing with OpenSource forum support systems, that's usually how it happens!).

Now Don Doozer also tends to be neck and neck with Joe Schmoe with the helpfulness of his answers, and he's a lot nicer. It turns out he's an IT guy for a non-profit organization that deals with quite a few computer illiterate people, so there's a background to why. There's nothing in the rating system to give him "nice" points, because I'm depending on a third party (a shared objective intermediary) to create a "niceness" meter.

Now I could create my own niceness meter label to apply to people, but it only contains my ratings, and doesn't benefit from wisdom of other people's opinions of Don's, and thus far, there's no way for me to share it. Now a system could allow for me to publish (share objectively) my subjective metering system, but that's not always desirable (particularly since I don't want Joe to know that I think he isn't nice). One is entitled one's opinion, but you're also entitled to keep it to yourself; and I'm sure we wish more people would exercize that second entitlement sometimes!

But let's say I have a peer group of people with whom I'd like to share my niceness meter, and whose opinions of niceness I hold in high regard. I decide I'd like to incorporate their opinions as well. But I'd like to keep my opinion as the primary meter, and only let their collective opinion "tweak" the result.

But there's a person in my peer group who I know is a poor judge of character, and actually thinks people who are rude are actually being mercifully nice. I'm not a fan of his opinion, despite the fact that he's a great peer in the current context. What's actually happening is that we don't objectively share the definition of "niceness." Do I have the ability to tweak the formula to incorporate my (low) opinion of his (inappropriate) opinion?

But wait, let's say there's a meter that I haven't considered (timeliness of response, for example) that others have considered. Is there a way to "suggest" that meter, or subconsciously incorporate it?

It turns out that if I incorporate that last meter, some dark horse candidate comes around the bend and ends up winning the Kentucky Derby, and that this dark horse happens to have an extremely high reputation in the opinion of both Don and Joe!

This is getting pretty complicated isn't it? Well, so are reputations.

Ultimately, any reputation system that depends on "global" objective/communal/shared systems is going to be gamed and ultimately fail in the relevance game. Once you know the rules, its easy to break them. But its impossible to game a system when it isn't known by the gamer, because that system is private, subjective and personal.

Reputations algorithms are ultimately subjective and therefore any attempt to quantify them must be customizable by the person depending on their results. In order to be customizable, they must be transparent. In order to be useful, they must be understandable by the end user who may not understand algebraic equations.

Reputation is going to be one of many Next Big Things in the next generation of Social Software, and unless it takes context, subjective and objective concepts into account, it will boil down to devolved popularity contests.

Second Post

My first attempt at a reply was lost to the ether, but I think my "keep me logged in" check box should stick this time.

Your kudos and concerns about recommendations are both spot-on. For example, I've been stuck at 90% on my LinkedIn profile because I lack two more recommendations. Now I actually have an issue with that, because it's not my fault - it isn't my responsibility to have a recommendation; the onus is on others to recommend me, not for me to solicit recommendations. I hate grovelling, brown-nosing and otherwise acting as a salesman. For me, the incentive (for the goal of 100%) should be the other way around - recommend 3 others.

It may seem silly to use a Likert scale numeric metric for a recommendation, but it does have some value. If it's one of a few (or several) metrics used in the final rep algorithm, you'll be able to tweak its influence in the final "score" (presuming you have a way to view the 'rithm transparently and can change it). How you tweak that Likert score can be influenced by the commentary that accompanies it.

It's pretty easy to give someone a "high five" when you're enthusiatically entranced by them, but if you're being overly gushy in your commentary (in someone else's opinion), that enthusiasm can be tempered by one's opinion of their opinion.

Let's presume it's a +10/-10 scale with commentary (I prefer 10 point scales over 5, because it allows for more nuance). The final outcome of the score would be affected by

1) my (subjective) relationship "score" (if any) with the recommender.
2) my (subjective) relationship "score" (if any) with the recommendee
3) my (subjective) opinion of the recommendation based on the commentary
4) the (objectively) collected (subjective) opinions of the recommendation
5) pick a metric, any metric.
6) and for their final rep score, how much recommendations factor into the final number

and of course we could go on ad infinitum with other factors, and none are really necessary. Having both a number and commentary provides for both subjective (free-form space to gush) and objective (a shared standard of measurement) factors.

Privacy about recommendations is important as well. There's not much incentive for me to give a negative recommendation if my name is attached - I don't necessarily think I need to own up to my comments, if my aims are altruistic and I'm trying to give constructive criticism. But privacy also allows for libelous commentary. Private recommendations could be allowed with a moderator/broker approval, if keeping things civil is an important aim, and you can even allow for hand-picked brokers/moderators. You could always subjectively "turn off" anonymous commentary but still allow for it.

Again, this is getting exponentially complicated, because it is. Ultimately a remote web-hosted SAAS platform (an objective/shared space) can't handle it, because it's infinitely subjective and requires (IMHO) a directly accessable desktop application. A peer-to-peer (or persona-to-persona) system is ultimately the only way to handle it. That's not a reason for SAAS companies to become despondent however. They can assist in many ways, by enabling the basic metrics and standards, by (objectively) "aggregrating" metrics on my behalf, by storing or processing via HAAS'n'SAAS. I think the best thing that SAAS companies can do is contribute to OpenSource initiatives that enable end-users to view and tweak rep systems, and make their own inhouse rep systems compatible with others and accessable via APIs. Short-sighted companies will fight this, but they'll ultimately lose.
~~~~~~~~~

Some of my first thoughts on this topic can be found on this blog post from June of '07. http://internetpsyche.blogspot.com/2007/06/reputation-systems-and-4-views.html

Tags have reputations too

Original post date: 9/8/07

Tags, or labels, have finally come to the forefront of memetic space, but there's still a problem.

Until recently, we've been stuck with folders, a hierarchical way of organizing all of our information. It's essentially meant that we've been stuck in two dimensions, a Flatland perspective of semantic space. Tags have the potential to turn everything into N dimensional space, or in database terms, truly relational.

One presumed aspect to tags so far has been the attitude that all tags are created equal, and they're not. This is especially true given that tags have synonyms, that words fade in popularity, and that connotation beats out denotation every time (triumph of the subjective in each of these cases).

Remember when card catalogs were the way we looked for books in the library? Remember looking for a subject using one word, and finding a card that said "See " or "See also ?" Turns out there isn't some objective method that catalogers use to decide which term will be the Primary term to list books under said subjects. Yes, there is a vast thesaurus librarians use, but which term they use in the end is completely subjective. Well, people do the same thing:

Is it
Web 2.0?
Social Web?
Dynamic Web?
Platform Web?

Heres another example, and one that will seem controversial and offensive to some (apologies ahead of time to those who think some of these words should never be uttered again), but it's invaluable as an example

Negro
Negroe
African
African-American
Afro-American
Colored
Of Color
Nigger
Nigguh
Nigga
Nig

Now tell me all of the above don't have their own reputations, and *that they're different for everyone*. Another important aspect to consider- there's a good chance that if you rank a certain tag high, that your close peers will as well. If you're a bigot, so will probably be your friends. Theres also a pretty good chance that youre not going to want to publish your personal ranking of tags too, but that you'll share them with your peers. Pretty damn difficult for a central database like Delicious to handle if you ask me (an objective solution to a subjective problem).

So why ask them too? Tags, like friends and peers, should be private, ranked, personal and rewarded for subjectivity. There will NEVER be a central Rogets Thesaurus for Tags, and we should give up trying to create one. Reward the subjective and the objective will eventually come.

How do we enable this? Allow users to create a personal thesaurus, for one. If I come a cross a blog that is labeled African-American and I want to actually label it Afro-American (for whatever reason), it should allow for it and do it automatically for me. Secondly, allow someone to apply a value to a tag (that way high ranking tags will more likely bring you relevant information).

Allow for labeling tags as parts of speech, so that we can manipulate them further.


Allow tags and folders to be interchangeable. If I tag something with a few tags, allow me to rank the tags in order of importance (for that item, or in general), and then allow me to designate if the top ranked tag is the Broader term (top level folder) or the Narrower term (the deepest folder). This allows for parallel folder structures and virtual folder views. It's a helluva lot easier to move things around too.