How should staff be evaluated?

[Superfluous]

So I'm scrolling through some of your guys' posts on the end of year voting thread, and I'm wondering why you're voting for the staff members that you're voting for (myself included). I can think of four reasons why you'd vote for someone as staff member of the year in x category:

1. You have some set of criteria by which you've gauged that some staff are performing better than others (activity, attitude, judgement, engagement, etc.); essentially, you are an informed voter
2. You're voting for staff other people before you have voted for and aren't particularly well informed yourself
3. You're voting for friends
   
4. You're voting randomly

I'm most interested in the first situation and how it pertains to how we rate staff performance internally.

In the past, we used a modscore system. This would tally up various mod actions and give each staff member a score every two weeks; those with consistently low scores were told to pick it up and demoted if they didn't. You can think of modscore as roughly: [# of warnings issued] + 2 * [# of bans issued] + 3 * [# of disputes closed] + 4 * [# of scam reports closed]. I'm not including everything we counted, but these are the basics. This system has advantages (e.g. staff can incentivize certain mod activities by offering more score for them) and drawbacks (e.g. not all activities generate modscore; notably, pardons were horribly neglected for this reason), but it has fallen into disrepair now that we've switched to XenForo. I think there may be issues with counting threads locked across various forums (though we can count total posts in all forums, so not sure how accurate this is anymore).

Our current system is to have anonymous, monthly votes on other staff members' performance: We get a blank slate with staff members' names on it, and rate them from 1 to 10 in a few categories. Voting distributions are then shared between staff members, and low scorers are told to improve. I personally think this system is worse than modscore and am wondering if we can improve it somehow, which is part of why I'm posting this thread here.

Obviously, you guys care about different things when it comes to moderation which is why you're collectively voting for so many different staff members for x of the year. What can we do to change the way we evaluate staff to better align with the things that you guys value? In my mind, this is the best way to make sure that we're doing a good job. So, I'm curious to hear what you think... What's best? Some variation of modscore? Some different voting scheme? Something else altogether (not evaluating at all, counting just time spent online, some type of reputation system for good/bad staff actions, some hybrid of various approaches)?

 

[Wonderland]

Stick to the evaluation being internal. There is enough dick sucking as it is around here.

 

[Tyler]

This is really a rough question, Internal feedback is useful, and probably the best system we could have but provides possible negatives.


I cant think of a better alternative at the moment.

 

[FlyingToast]

I think the bigger question is what are staff supposed to do?

What's best? Some variation of modscore? Some different voting scheme? Something else altogether (not evaluating at all, counting just time spent online, some type of reputation system for good/bad staff actions, some hybrid of various approaches)

From this quote above, it seems like there isn't really a clear, well defined purpose for staff roles atm. Historically we had sectionals / globals / admins, but those lines have been blurred more than ever in these past few months. I think before you can rate / score eachother, you really need to figure out what you want everyone to actually be doing. Makes me think of a teacher who is grading an assignment without making a rubric beforehand.

Mod logs tracked your progress to see if you were meeting the minimum quota for what you're supposed to be doing. How is this possible with an anonymous voting system that isn't based on concrete metrics? Why is meeting the bare minimum bad in an unpaid volunteer position?

If it was up to me, I'd go with Jack's proposed system of adding actual discussion activity into modscores, then try your best to balance the benefits that you perceive to come from staff interactions with fair values for mod actions and run a mod-log system again. Give people different areas to shine in, but still offer a concrete metric in which you can track eachother by. The current system sounds like some survivor style voting the weakest link off the island. Reward people for offering to volunteer their time to run this site, don't chastise them for not doing more. This site is lacking in many areas, but discussion and staff presence within discussions outside of the spam forum easily could be improved. Let those who can promote discussion, and in turn promote activity, be rewarded.

I strongly disliked dealing with scam reports / disputes, I couldn't stand to be "the judge and jurry". Despite this, I absolutely loved promoting discussion within my sections, welcoming people to this site, and smashing through rasp reports. I didn't really have an avenue to shine in terms of metrics with modlogs, nor would I currently based on how I'm imagining these questionnaires / review threads are set up. Despite this, I felt like I never necessarily did a "poor job", I just was never tracked in the areas in which I felt I excelled in best.

 

[Superfluous]

I think the bigger question is what are staff supposed to do?



From this quote above, it seems like there isn't really a clear, well defined purpose for staff roles atm. Historically we had sectionals / globals / admins, but those lines have been blurred more than ever in these past few months. I think before you can rate / score eachother, you really need to figure out what you want everyone to actually be doing. Makes me think of a teacher who is grading an assignment without making a rubric beforehand.

Mod logs tracked your progress to see if you were meeting the minimum quota for what you're supposed to be doing. How is this possible with an anonymous voting system that isn't based on concrete metrics? Why is meeting the bare minimum bad in an unpaid volunteer position?

If it was up to me, I'd go with Jack's proposed system of adding actual discussion activity into modscores, then try your best to balance the benefits that you perceive to come from staff interactions with fair values for mod actions and run a mod-log system again. Give people different areas to shine in, but still offer a concrete metric in which you can track eachother by. The current system sounds like some survivor style voting the weakest link off the island. Reward people for offering to volunteer their time to run this site, don't chastise them for not doing more. This site is lacking in many areas, but discussion and staff presence within discussions outside of the spam forum easily could be improved. Let those who can promote discussion, and in turn promote activity, be rewarded.

I strongly disliked dealing with scam reports / disputes, I couldn't stand to be "the judge and jurry". Despite this, I absolutely loved promoting discussion within my sections, welcoming people to this site, and smashing through rasp reports. I didn't really have an avenue to shine in terms of metrics with modlogs, nor would I currently based on how I'm imagining these questionnaires / review threads are set up. Despite this, I felt like I never necessarily did a "poor job", I just was never tracked in the areas in which I felt I excelled in best.

Great post, thanks. I think the hope may have been that internal ratings would help capture those intangible aspects of moderation better than modscore, but I don't think this has been the case, especially since internal ratings are not guided (i.e., a "7" doesn't really mean anything, so my idea of a "7" could be your idea of a "4"; this obviously leads to problems when people are given aggregate ratings).

Though internal ratings could be adjusted to make them better, my preference would still be to expand modscores to capture some of these things as best we can. I think "posts in community/help/support sections" could be a category. "Time spent online" might work as well. Pardons should count for something. Total mod actions should be logged as they were in the past (if possible). I don't know if we're technologically able to do all of these things, but I imagine that we probably are. Maybe this is a question for @Matthew more than anyone?

 

[BumBum]

On the real, I believe that optional anonymous surveys for each individual staff member could go a long way, especially if you reward users for giving genuine constructive feedback. Rewards don't have to be substantial, just enough to give people reason to truthfully fill out these surveys.

 

[Ex]

@Superfluous i think staff reviews should be both internal and external.

The internal part being the mod score, scam reports , post reports etc should all stay the same.

The external part should be the Community deciding on if that staff member is fair and consistent with their actions, maybe not a monthly thing but possibly quarterly.

PLUS: Staff members should be decently active. What type of example is it to have staff members that rarely post / do anything with the community.

 

[ShipTheFlip]

My opinion comes from how in-depth they go in the way they handle reports, pardons, etc. So for example @malakadang and @Sythe are the gold standard.

 

[Hahanerd]

The evaluation should be internal. We have enough active staff members to determine whether or not someone on the team is slacking and they can be the judge of character for that.

 

[Sonia]

Aside my lengthy weird post.

Both internally and externally moderation should be evaluated.

You are moderating a site which is directly impacting users. Therefore you need to get the community to review you as a staff member. Take the feedback for all staff as a somewhat small review, but most tends to be biased or inaccurate depending on how each community member comes across dependent staff, based on time-zones etc. Not all staff see every other staff I presume, therefore internal review can also be biased.

Basically use the external evaluation to define community interaction (I mean if you want to evaluate how staff relate to community?). This is something I think should definitely be done, definitely agree with @Jack stating this about having staff use their sections and activity within their sections be more of a way to review staff, same as FT.

It's not just about the pardon system and the scammers section. For some reason the current system is so hell bent on only focusing staff efforts on moderation score, when in reality if they interacted with the community you would see a better impact of reviewing staff over all. I am so bored of seeing staff pay attention to people getting scammed and pardoned, when half of them don't give much attention to the community that is here already, and the sections that we use etc. No wonder why we're all sour.

I'm probably over-reacting, but basically that is the point.

If they act within the community, then we know you are there.
If they don't actively relate to the community we don't know you are there.

This should also enable knowing when a staff member is active enough or not.

The internal secretive review staff members v staff can be biased, because staff will see everything each other staff member does from a staffs perspective, but from a community perspective some people don't do anything.

Also agree with @SuF when he wanted more sectionals to focus on individual sections within the forum. It makes sense along with the review of staff.

@Sythe literally should get people on the CDT just to dedicate themselves more to sections, because clearly you don't want staff to do that job.

 

[Jack]

@Superfluous look at the thread I made in SL with my proposed changes to modscore. @Matthew was saying that they were waiting on a xenforo patch which should have been happening within a few weeks back when I was admin so that modscore could be re-implemented properly, I assume it's just now been deserted in favor of solely anonymous peer reviews. Many people begrudged modscore as it did not tell a whole story. Switching to relying on solely a subjective review system is not a good idea either as it allows bias and uninformed decisions to be made (we naturally wouldn't scour every inch of the site to see the full extent of one's contributions and it allows for cliques etc to form).

The issues that existed with modscore primarily lay in the fact that it did not address pardons or anything outside of strictly moderating issues. One of the biggest issues in that and is something that I've been passionate about my whole time on sythe, even before my tenure on staff is that of participation in the community. Like it or not, staff members are leaders in the site and can help drive discussion with their participation. Do you remember how you felt when you were new and participating in sections and all of a sudden a moderator was replying to you and participating, welcoming you into the community? I've never agreed with the whole ivory tower approach where all of a sudden a moderator would solely pour all of their time into reports and the like, no longer participating in the community that they're supposed to be in tune with and serve.

Ultimately my proposed additions went something along the lines of adding modscore for all posts on pardons (serves ultimately rewarding all phases of a pardon - those who post their initial findings/previous reports or pardons, those who vote since a prerequisite to voting is a bit of independent research, those who lock pardons, those who copy pardons for USL voting). Also modscore would be rewarded for posts within discussion sections and extra modscore would be rewarded for creation of threads within these sections. I don't remember the values I had assigned to these actions but I know I had them proportioned relatively well to the existing values for scam reports etc so as to not create avenues where staff members could "cheat" higher modscore and it would not throw the importance hierarchy of actions out of balance either while still being substantial enough to still be an incentive.

 

[FireZ]

From this quote above, it seems like there isn't really a clear, well defined purpose for staff roles atm. Historically we had sectionals / globals / admins, but those lines have been blurred more than ever in these past few months.

its to the point theres hardly a difference between a global and a sectional

 

[FlyingToast]

its to the point theres hardly a difference between a global and a sectional

And the point of the rotary is that "at any time" Global <------> Admin. Like I said, there isn't really a well defined set of duties for each role; globals have been merging into admin duties, whilst sectionals have also merged into globals. It's hard to know if you're doing a good job when you don't know what you're supposed to be doing.

 

[Superfluous]

@Superfluous look at the thread I made in SL with my proposed changes to modscore. @Matthew was saying that they were waiting on a xenforo patch which should have been happening within a few weeks back when I was admin so that modscore could be re-implemented properly, I assume it's just now been deserted in favor of solely anonymous peer reviews. Many people begrudged modscore as it did not tell a whole story. Switching to relying on solely a subjective review system is not a good idea either as it allows bias and uninformed decisions to be made (we naturally wouldn't scour every inch of the site to see the full extent of one's contributions and it allows for cliques etc to form).

The issues that existed with modscore primarily lay in the fact that it did not address pardons or anything outside of strictly moderating issues. One of the biggest issues in that and is something that I've been passionate about my whole time on sythe, even before my tenure on staff is that of participation in the community. Like it or not, staff members are leaders in the site and can help drive discussion with their participation. Do you remember how you felt when you were new and participating in sections and all of a sudden a moderator was replying to you and participating, welcoming you into the community? I've never agreed with the whole ivory tower approach where all of a sudden a moderator would solely pour all of their time into reports and the like, no longer participating in the community that they're supposed to be in tune with and serve.

Ultimately my proposed additions went something along the lines of adding modscore for all posts on pardons (serves ultimately rewarding all phases of a pardon - those who post their initial findings/previous reports or pardons, those who vote since a prerequisite to voting is a bit of independent research, those who lock pardons, those who copy pardons for USL voting). Also modscore would be rewarded for posts within discussion sections and extra modscore would be rewarded for creation of threads within these sections. I don't remember the values I had assigned to these actions but I know I had them proportioned relatively well to the existing values for scam reports etc so as to not create avenues where staff members could "cheat" higher modscore and it would not throw the importance hierarchy of actions out of balance either while still being substantial enough to still be an incentive.

Another great post, appreciate the input, and I agree that we can (theoretically) reformat modscore to capture more of what staff should be doing along the lines you outlined here. I don't see a reason NOT to do this... incentivizing community integration with modscore for community-related actions seems perfectly reasonable to me. And, at the very least, it would be a super-helpful companion to the current system of anonymous voting.