Because I didnt want to post at the other link, I just figured I'd post this here:
This is on the issue of "biased judging" and just ratings in general.
I read in the other thread someone's idea (cant remember who) of one person installing the mod, then a couple of other people playing with it to test it. Personally, I am against that system a bit. I know I'm not directly involved in any of this, so just take these suggestions for what they're worth; I'm not trying to be a downer or anything.
What I really think would work best is a core group of testers, probably 5-7 people. When mods get submitted, I really think a minimum of three people should test the mod, with a max of probably around five people (any more than that and reviews are just going to get long and drawn out). Testers should have to go through every step themselves, from initial installation to customization to testing the mod out as an admin, moderator, member, guest (should the mod work differently for different user levels). If you aren't comfortable with doing all of that, you aren't really experienced and ready enough to test mods, IMHO. This also provides a greater testing area on different platforms and different users, so you don't have just the installation opinion for one person installing it.
After the installation, I think the rating should be a concensus between all the testers (Id imagine that was the way it was going to be anyway, but Im typing it again just so all my thoughts are layed out and I dont lose anybody). Additional testing, such as compatibility issues, can most likely be handled by one or two individuals, as most mod are written with the idea of (and even used to state) "mod assumes a clean installation of YaBB."
As far as reviews go, take a look at this link:
http://art.counter-strike.net/art.shtmlDon't pay attention to the links without thumbnaiils, but Weeks 10-12 have little "read" links under all the pictures. This is where the reviewers (which happen to be for art, in this case) post their individual thoughts and additional comments. Something like that would be nice on the mod review, so that users could read what mod reviewers actually thought, and could read a variety of comments. That way if John Doe reviewed, say, the email mod, but doesnt need email much, so he wrote a pretty basic review, but Jane Doe uses email ALL the time and really likes it, she could post why she especially likes the mods. It would just help give a very rounded approach to what each tester thought of the mod.
Anyway, those are some of my ideas from now. Just remember, whatever you end up doing, I'll think it's cool. Just wanted to give you guys some ideas and thoughts to brainstorm on. And if you need any help... well, I actually programmed a little PERL last night, so you never know... maybe I'd be able to pitch in sometime.
- Dave