The Assistive Technology Automation Subgroup of ARIA-AT Community Group just discussed this issue. Meeting minutes are available on w3.org and also included below.

<details> <summary>The full IRC log of that discussion</summary>

<Sam_Shaw> TOPIC: w3c/aria-at - #945 - Rethinking the wording for assertion verdicts <Sam_Shaw> https://github.com/w3c/aria-at/issues/945 <Sam_Shaw> jugglinmike: The working mode doesn't have the term verdict yet, but its one we intend to add. <Sam_Shaw> jugglinmike: The working mode refers to verdicts as supported, not supported etc <Sam_Shaw> jugglinmike: automation refers to the verdicts as acceptable, omitted, contradictory <Sam_Shaw> jugglinmike: I have a proposal for a new set of terms <Sam_Shaw> correction: automation refers to the verdicts as good output, no output, incorrect output <Sam_Shaw> the proposed new terms are acceptable, omitted, contradictory <Sam_Shaw> js: I like the new terms you proposed. In terms of bubbling up the results, I wonder if no support, partial support, supported is clearer <Sam_Shaw> MK: Thats why I wanted to use numbers <Sam_Shaw> MK: Partial support could mean anything between a little support to almost fully supported <Sam_Shaw> JS: I agree but if something is 90% supported, the remaining 10% could still make it unusable <Sam_Shaw> MK: I agree, unless we have multiple layers of assertions we don't need numbers. We also want to be diplomatic <mmoss> present+ <Sam_Shaw> MK: I think for your solution is pretty solid <Sam_Shaw> MK: We just need to decide if we extend the use of these terms, or bubble them up <Sam_Shaw> jugglinmike: Yes bubbling up we need to consider, the case where a feature is all supported except one, its not supported. For verdicts that can be in three states, understanding why its partially supported is tough. I'm not sure if bubbling can work if we are looking for a percent score <Sam_Shaw> MK: Yeah supported needs to be binary <Sam_Shaw> JS: I think we need all three states <Sam_Shaw> MK: What do the responses tell us? Either there is some support there or there isn't. Then the reasons is because someone tried, or someone didn't try to support <Sam_Shaw> MK: If you measuring something using a percentage, then it needs to be binary <Sam_Shaw> JS: for the reports, are there three levels to two of support? <Sam_Shaw> MK: Any level of support beyond assertion is a percentage. <Sam_Shaw> MK: At the AT level, the test level, at the AT level, all will be a percentage <Sam_Shaw> MK: So we would say, using Mikes terminology, At the assertion level if the response is omitted or contradictory then that counts as a 0. If its acceptable then it counts as a 1. <Sam_Shaw> MK: We could do other reports we could run that say what percent is contradictory, which percent is omitted <Sam_Shaw> MK: I don't know that we need to bubble up these terms in the reports we have now <Sam_Shaw> MK: We don't need terms for working mode, its just level of support <Sam_Shaw> jugglinmike: I do think the working mode uses supported not supported. <Sam_Shaw> MK: I can get rid of that <Sam_Shaw> MK: I have some other issues for the working mode, particularly 950, I think we need to work on another iteration of the working mode and share it with the community <Sam_Shaw> MK: We could have a binary state for assertions, and get rid of contradictory <Sam_Shaw> JS: I agree, but we should rewrite the terms <Sam_Shaw> JS: Lets add this to the agenda for the CG meeting thursday <Sam_Shaw> jugglinmike: What I'm hearing is, we like the terms I proposed, but we may not need three terms <Sam_Shaw> JS: It will make the testing easier if we just have two states/terms <Sam_Shaw> MK: Okay but if this task isn't on the critical path, I want to be conscious of that <Sam_Shaw> JS: This could speed up the process <Sam_Shaw> MK: But its not a blocker, we can talk about enhancements in the near future <Sam_Shaw> Michael Fairchild: Is there a third state where we publish a report with some of the data missing? <Sam_Shaw> JS: No, really, but we need to consider this. <Sam_Shaw> JS: If there is a situation where only 50% of test have been completed, what does that look like for a percent supported? <Sam_Shaw> MK: We made a decision to change the working mode, and to get rid of the three output terms <Sam_Shaw> MK: The question before we change the UI, is do we go from 3 to 2 states? Acceptable, not, contradictory

</details>

Hi there, @frehner! I see that you received some direction from Kevin over in gh-3738; did you want to incorporate that here in this patch? There's no rush--I just don't want to merge this prematurely.

Thanks! This normative change has consensus, but we'd like to wait until this comment is resolved before merging the tests.

<details> <summary>Context: explanation of the term "verdict"</summary>

Currently, the process of "running a test" involves two steps:

1. collecting AT responses, and
2. interpreting whether the collected AT responses satisfy the given assertions

The result is a series of "verdicts" (we'll be adding a definition along these lines to the glossary soon).

Historically, we haven't discussed these as separate operations because we've expected every tester to perform both. It will soon be necessary to consider them as distinct steps because we will support automated AT response collection, but we will not support automated verdict assignment.

</details>

Issues gh-936 and gh-937 describe new coordination opportunities which will only be possible once we recognize that "running a test" is a two-step process.

The more we consider "AT response collection" and "verdict assignment" as separate tasks (that is, if we embrace the new possibilities in those issues), the more I wonder whether the two need distinct verification steps, where verdict assignment does not begin until AT responses have been corroborated.

In other words: should we ask Testers to assign verdicts before we're confident in the AT responses which they're interpreting?

Thanks @mcking65! I'm going to close this as "not planned" based on your feedback alone, given that we've been encouraging folks via other channels to share their perspective here for over a month.

Thanks, @mcking65! There are two places in the Working Mode where Testers are assigned to Test Plans, so I'm wondering where that scenario applies. Do you envision it occurring before a "Draft" Test Plan advances to the "Candidate" phase? Is it something that might happen when reporting on "Recommended" Test Plans?

We know that a human tester will one day need to assign verdicts to AT responses collected by the automated system.

Yes. However, to mitigate the need to do this more than is necessary, I believe we agreed the initial implementation of the automated system will assign verdicts when the AT response is identical to a previously analyzed response to the same command in the same test.

I do recall our discussion about verdict reuse, though I'm not sure if it needs to happen in the automation subsystem. Since reuse seems like an optimization which is orthogonal to the capability discussed here, I'll refrain from going into detail in this thread. (This will certainly come up as we continue to refine the design of the automation system, but if you'd like to continue the discussion right away, I'm happy to open a new issue.)

In practice, I have found it difficult to look at the content in the response field and assign verdicts without experiencing the test case itself. Perhaps that could change with practice. Regardless, if my job is to assign verdicts to previously collected responses, I think I would always do a better job if I were to open the test page, run set up, and run the test for most of the commands.

This reminds me of a discussion from a recent CG meeting. To the extent that "you had to be there" to explain the assignment of a verdict, that seems like a problem for the public's consumption of ARIA-AT's reports. If we could more concretely describe the kinds of situations for which the AT response data is insufficient, that might uncover improvements to ARIA-AT's processes/systems which ultimately promote transparency.

Do you think we could include an agenda item for the next CG meeting to ask the Testers present about their experience along these lines?

An understanding of these use cases (plus any others) will help us refine both ARIA-AT App and the nascent automation system. This brings me to the question I posed in the title of this issue: should one tester be able to assign verdicts to AT responses collected by another?

We have a form of this in the admin role. I don't see much value in extending the current UI to support the additional use case of having some CG members only collect responses.

Thanks!

#1904 added another case where we do nothing, a dataIsOlder early exit. That most likely leads to additional cases where we fail to update the data even though there are contradictions.

@foolip was this addressed by the refectoring in gh-2629?

The Glossary defines the term "test plan run" as follows:

Test Plan Run

Execution of all tests in a test plan with a specific assistive technology at a specific version using a specific browser at a specific version, e.g., run all tests in the checkbox test plan with JAWS X and Chrome Y. X and Y include minor version numbers.

An "execution" is an ephemeral concept, seeming to me more like an event than an object with static properties.

However, the way the Working Mode applies the term doesn't support this interpretation. It describes participants interacting with Test Plan Runs as if they were documents:

• Test Admins "add" Test Plan Runs to the Test Queue
• Testers "execute" Test Plan Runs
• Test Plan Reports include Test Plan Runs

Meanwhile, Testers are also expected to "run the tests in the draft test plan" directly, apparently beyond the context of any Test Plan Run.

The implementation of ARIA-AT App offers yet another interpretation. There, the entity named TestPlanRun is essentially a set of test results that are attributed to a specific Tester.

Although there may be an alternative design which obviates the need for the "test plan run" concept altogether, I recognize that such a change would be disruptive to the community group's ongoing work.

Instead, I believe the following changes would avoid rocking the boat but still enable forward progress on future extensions to the Working Mode:

1. redefine the "Test Plan Run" as "an entity describing one Tester's progress collecting results for all tests in a test plan with a specific [...]"
2. remove any wording that has Testers interacting with test plans, and replace it with instruction for Testers to complete Test Plan Runs to which they have been assigned

(These changes are very closely related to a previously-reported omission regarding the handling of incomplete Test Plan Runs, but I think it would be wise to address this more subtle consistency issue, first.)

The update-bcd.ts script has eight "TODO"-style code comments, but one of them omits an explanation of the task which motivated it:

https://github.com/foolip/mdn-bcd-collector/blob/723b755a1f8717b3e9088778ca954574686856d7/update-bcd.ts#L290-L295

The comment introduced in 2020 via commit d5ff5fe2. Its position implies that something more ought to be done when encountering a browser version with no support information, but I haven't been able to infer what that is.

I've been gathering together a list of enhancements for that script, and I don't want to miss whatever functionality motivated that comment.

In this project, an assertion is judged in terms of an AT response to produce a "verdict." That verdict can be one of three values. The Working Mode uses one set of words to describe those values, and the app uses a different set of words:

This inconsistency can make it difficult to talk about the process and the implementation. Further, half of the terms are susceptible to misinterpretation:

• "incorrectly supported" is something of an oxymoron. The word "incorrectly" subverts the meaning of "supported."
• "no output" is too expansive. It's meant to describe the case where there is no output related to the assertion under consideration, but new Testers could be forgiven for thinking it only applies where there is no output at all (i.e. when the screen reader is completely silent).
• "incorrect output" is likewise overly broad because it describes the "output" as a whole. For instance, given the output "bananas" and the assertion "the role 'button' is conveyed", a new Tester might reasonably say "incorrect output" because that output is incorrect.

I am proposing a new set of terms to be used by both the Working Mode and the App:

I agree with the intent of this issue, but I think the semantics of the in operator make it unsuitable for conformance testing. Since the prototype of the Location prototype is the Object prototype, a test like 'toString' in Location.prototype is also subject to false positives.

Case in point: the toString in question is specified as an "own" property of the Location instance, so if that particular test references Location.prototype, it should probably assert the absence of an "own" property (though that may be too prescriptive--I don't know HTML's policy for host-defined methods).

Sure thing!

<!-- 🙌 Thanks for contributing to MDN Web Docs. Adding details below will help us to merge your PR faster. -->

Description

Demonstrate the API that the page is documenting

Motivation

As originally written, the example demonstrates a different API (Object.prototype.toString)

Additional details

<!-- 🔗 Link to release notes, vendor docs, bug trackers, source control, or other places providing more context -->

Related issues and pull requests

<!-- 🔨 If this fully resolves a GitHub issue, use "Fixes #123" --> <!-- 👉 Highlight related pull requests using "Relates to #123" --> <!-- ❗ If another pull request should be merged first, use "Depends on: #123" -->

<!-- 👷‍♀️ After submitting, go to the "Checks" tab of your PR for the build status -->

<!-- 🙌 Thanks for contributing to MDN Web Docs. Adding details below will help us to merge your PR faster. -->

Description

Correct return value of location.toString()

Motivation

Accurately describe the API's behavior

Additional details

<!-- 🔗 Link to release notes, vendor docs, bug trackers, source control, or other places providing more context -->

Related issues and pull requests

<!-- 🔨 If this fully resolves a GitHub issue, use "Fixes #123" --> <!-- 👉 Highlight related pull requests using "Relates to #123" --> <!-- ❗ If another pull request should be merged first, use "Depends on: #123" -->

<!-- 👷‍♀️ After submitting, go to the "Checks" tab of your PR for the build status -->

const checkDate = async function (t, dateSelector, day) {

This line creates a function named assertAriaRoles and stores it in a variable named checkDate. The name is somewhat confusing in this context because this file already has a function stored in the variable assertAriaRoles.

This doesn't demonstrate the bug fix (it passes on the main branch). Starting from 2020-02-29 would do it, though

    const startDate = '2/29/2020';
    const targetDates = [
      '2/28/2019',
      '2/28/2018',
      '2/28/2017',
      '2/29/2016',
      '2/28/2015',
      '2/28/2014',
    ];

  const d = String(day.getDate()).padStart(2, '0');

The code works as written, so you can take or leave this suggestion (I just feel it communicates your intent better).

That's a far more informed theory than I've been able to come up with! It also sounds like a great starting point for when we start working on the long-term fix.