Talk:ORES/FAQ

FAQ discussion page for ORES.

Help answer ORES expert level questions...

Here are some questions we would still like to answer! Feel free to share answers or ask something that hasn't been covered in the FAQ!

What hosts are used for ORES and where do they fit in the architecture?

Where are the deployment repositories? ===

How do I install the ORES package? 

How do I develop ORES? 

How to start a dev instance? 

How to submit a PR to this repository?

How do I use ORES' API client? 

What's a ScoringSystem and how does it work?  SRodlund (WMF) (talk) 01:53, 14 November 2017 (UTC)

In what sense does ORES WP10 model measure the "quality" of an article? Does it differentiate between high and low quality sources? Good and bad writing? Can I use it for grading Wikipedia editing assignments?

We're incorporating more WP10-based info into the Wiki Education Dashboard these days... I actually just deployed a new feature that links to this very FAQ for background... and these are the types of questions that academics are likely to ask. More explanation of what features do and do not go into the models would be most welcome. Sage (Wiki Ed) (talk) 22:36, 21 November 2017 (UTC)

Ooh. Good Q. I'm happy to have a more substantial writeup about the wp10 model. Halfak (WMF) (talk) 15:02, 22 November 2017 (UTC)

Sage, it's not much, but check out what I've added to ORES#Assessment_scale_support. What do you think? I think we could write a whole page about what the model directly and indirectly accounts for. But I figured this was a good start. EpochFail (talk) 16:07, 29 November 2017 (UTC)

Yeah, good start! Sage (Wiki Ed) (talk) 17:59, 29 November 2017 (UTC)

Another question TODO answer:

• Which models are used by each known consumer? For example, what is using wp10 predictions? Adamw (talk) 18:44, 18 January 2018 (UTC)

scoring-internal@wikimedia.org

The page asked that people contact the team through said email address, but attempting to do so gives "550 Address scoring-internal@wikimedia.org does not exist". I've removed the address from the page for the time being. Julle (talk) 17:57, 15 January 2018 (UTC)

Thanks for noticing—I'm replacing with the correct address, scoring-internallists.wikimedia.org Adamw (talk) 14:20, 16 January 2018 (UTC)

Explanation of ArticleQuality.js script

I just talked through the ArticleQuality script and how to interpret it, with another Wiki Education staffer, and we thought something similar would be good to include in the FAQ.

We were looking at the result for https://en.wikipedia.org/wiki/Rachel_Carson: FA (4.7), and the question was, what does the 4.7 mean? (47% likely to be an FA? 4.7% likely to be a FA?)

The answer I gave is that it's based on the probabilities that ORES gives for the possible quality levels: ps://ores.wmflabs.org/v3/scores/enwiki/?models=wp10&revids=877812769

So it says FA because FA is the most likely result (at 42%), but 4.7 means the weighted average prediction is actually closer to GA, with the scale going from 1 (definitely a stub) to 6 (definitely an FA). Sage (Wiki Ed) (talk) 19:06, 16 January 2019 (UTC)

New question: What information does ORES use to evaluate an edit?

Does it just use the content of the edit itself? Does it consider whether the editor is logged-in or not? The age of the editor's account? The permissions of the account? The time of day the edit was made? Kaldari (talk) 15:55, 23 December 2020 (UTC)

@EpochFail@ACraze (WMF) - I bet y'all know the answer to this. Kaldari (talk) 16:06, 23 December 2020 (UTC)

Yes to all of the above. Check out https://ores.wikimedia.org/v3/scores/enwiki/123457/damaging?features EpochFail (talk) 02:36, 24 December 2020 (UTC)

You can experiment with the local gradients for each feature by injecting counterfactuals: ORES/Feature injection

For a more systematic way of exploring how the features lead to a given prediction, there are libraries which vary each feature and show its impact on the output. For example: https://github.com/adamwight/ores-lime/blob/master/Explain%20edit%20quality.ipynb Adamw (talk) 15:23, 24 December 2020 (UTC)