I've been reviewing some of the pages of ORES' mistakes (e.g. it:Progetto:Patrolling/ORES and some of our review of anomalies in training data (e.g. Phab:T171497). I think we'll need a nice pattern by which multiple judgments can be submitted about a single wiki entity (revision, page, user, etc.), a conversation can take place and consensus recorded. In order to manage all of this in the JADE/Schema, we've split up the notion of a "judgement" from an "endorsement" and we've borrowed the concept of "preference" from Wikidata.
Let me illustrate with an example. Let's say that I want to judge the quality of en:Special:Diff/327645238. I'd create a new "judgement" with damaging=true and file an "endorsement" of the judgement with my comment "Removing cleanup tags without fixing the problems." Because (in this example) I was the first person to submit a judgement/endorsement, that judgement would be marked "preferred".
At a later date, Awight is reviewing some anomalies in our training set and comes across this judgement. He reviews the diff and comes to a different conclusion. So he creates a new "judgement" damaging=false and files an "endorsement" of that judgement with the comment "Good removal of cleanup tags". He's feeling pretty confident, so he moves the "preference" to his non-damaging judgement and starts a new discussion thread tied to this wiki entity titled "Cleanup was performed in past edits" where he points out that the cleanup had already occurred before the edit in question.
I get a notification that an entity I judged has a new discussion thread. After reviewing the edit again and seeing Awight's reasoning, I still disagree, but I don't feel strongly enough to dispute the new preferred judgement. I could change my endorsement and edit my comment to reflect Awight's, but I don't want to. So I leave my endorsement on damaging=true and leave the "preference" alone (so it's still pointing to damaging=false)
Does this make sense? Should it be possible to have two different "preferred" judgments or should we limit it to just one that captures consensus? Does it make sense to allow individuals to keep their endorsements where they like and set the preference bit separately? What do you think?