Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature]: single aria snapshot for different engines/browsers #34483

Closed
rashmipriya12 opened this issue Jan 26, 2025 · 4 comments · Fixed by #34537
Closed

[Feature]: single aria snapshot for different engines/browsers #34483

rashmipriya12 opened this issue Jan 26, 2025 · 4 comments · Fixed by #34537
Assignees
Labels

Comments

@rashmipriya12
Copy link

🚀 Feature Request

In the latest release v1.50 aria-snapshot can be generated using toMatchAriaSnapshot({ name: 'snapshot.yml }) but this expects different snapshot for chromium, firefox and webkit on windows and linux.
The aria snapshot will not be different for the above different scenarios, can aria-snapshot work with single yml file please. Thanks

Example

No response

Motivation

This will reduce the effort of copy pasting same regex in different files

@agg23
Copy link
Contributor

agg23 commented Jan 27, 2025

You are getting separate snapshots because you are using separate projects for each browser, which will create separate files by default. You can use snapshotPathTemplate in your config to control how the snapshots are generated.

@rashmipriya12
Copy link
Author

You are getting separate snapshots because you are using separate projects for each browser, which will create separate files by default. You can use snapshotPathTemplate in your config to control how the snapshots are generated.

Thanks for suggesting this. I tried it, however we do have some visual diff testing using toHaveScreenshot(). The screenshot requires different browsers configuration as few elements like scrollbar render differently. The accessibility tree does not have the same issue as it is same across browser

@agg23
Copy link
Contributor

agg23 commented Jan 28, 2025

You're right. This is something we need to improve on.

For now, you can use the explicit inline Aria snapshot mode:

await expect(page.locator('body')).toMatchAriaSnapshot(`
  - heading "todos"
  - textbox "What needs to be done?"
`);

@rashmipriya12
Copy link
Author

You're right. This is something we need to improve on.

For now, you can use the explicit inline Aria snapshot mode:

await expect(page.locator('body')).toMatchAriaSnapshot(`

  • heading "todos"
  • textbox "What needs to be done?"
    `);

Thank you!!

@dgozman dgozman added v1.50 and removed v1.51 labels Jan 29, 2025
DennisRasey pushed a commit to DennisRasey/forgejo that referenced this issue Feb 1, 2025
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [@playwright/test](https://playwright.dev) ([source](https://github.com/microsoft/playwright)) | devDependencies | minor | [`1.49.1` -> `1.50.1`](https://renovatebot.com/diffs/npm/@playwright%2ftest/1.49.1/1.50.1) |

---

### Release Notes

<details>
<summary>microsoft/playwright (@&#8203;playwright/test)</summary>

### [`v1.50.1`](https://github.com/microsoft/playwright/releases/tag/v1.50.1)

[Compare Source](microsoft/playwright@v1.50.0...v1.50.1)

##### Highlights

microsoft/playwright#34483 - \[Feature]: single aria snapshot for different engines/browsers
microsoft/playwright#34497 - \[Bug]: Firefox not handling keepalive: true fetch requests
microsoft/playwright#34504 - \[Bug]: update snapshots not creating good diffs
microsoft/playwright#34507 - \[Bug]: snapshotPathTemplate doesnt work when multiple projects
microsoft/playwright#34462 - \[Bug]: updateSnapshots "changed" throws an error

#### Browser Versions

-   Chromium 133.0.6943.16
-   Mozilla Firefox 134.0
-   WebKit 18.2

This version was also tested against the following stable channels:

-   Google Chrome 132
-   Microsoft Edge 132

### [`v1.50.0`](https://github.com/microsoft/playwright/releases/tag/v1.50.0)

[Compare Source](microsoft/playwright@v1.49.1...v1.50.0)

#### Test runner

-   New option [`timeout`](https://playwright.dev/docs/api/class-test#test-step-option-timeout) allows specifying a maximum run time for an individual test step. A timed-out step will fail the execution of the test.

    ```js
    test('some test', async ({ page }) => {
      await test.step('a step', async () => {
        // This step can time out separately from the test
      }, { timeout: 1000 });
    });
    ```

-   New method [test.step.skip()](https://playwright.dev/docs/api/class-test#test-step-skip) to disable execution of a test step.

    ```js
    test('some test', async ({ page }) => {
      await test.step('before running step', async () => {
        // Normal step
      });

      await test.step.skip('not yet ready', async () => {
        // This step is skipped
      });

      await test.step('after running step', async () => {
        // This step still runs even though the previous one was skipped
      });
    });
    ```

-   Expanded [expect(locator).toMatchAriaSnapshot()](https://playwright.dev/docs/api/class-locatorassertions#locator-assertions-to-match-aria-snapshot-2) to allow storing of aria snapshots in separate YAML files.

-   Added method [expect(locator).toHaveAccessibleErrorMessage()](https://playwright.dev/docs/api/class-locatorassertions#locator-assertions-to-have-accessible-error-message) to assert the Locator points to an element with a given [aria errormessage](https://w3c.github.io/aria/#aria-errormessage).

-   Option [testConfig.updateSnapshots](https://playwright.dev/docs/api/class-testconfig#test-config-update-snapshots) added the configuration enum `changed`. `changed` updates only the snapshots that have changed, whereas `all` now updates all snapshots, regardless of whether there are any differences.

-   New option [testConfig.updateSourceMethod](https://playwright.dev/docs/api/class-testconfig#test-config-update-source-method) defines the way source code is updated when [testConfig.updateSnapshots](https://playwright.dev/docs/api/class-testconfig#test-config-update-snapshots) is configured. Added `overwrite` and `3-way` modes that write the changes into source code, on top of existing `patch` mode that creates a patch file.

    ```bash
    npx playwright test --update-snapshots=changed --update-source-method=3way
    ```

-   Option [testConfig.webServer](https://playwright.dev/docs/api/class-testconfig#test-config-web-server) added a `gracefulShutdown` field for specifying a process kill signal other than the default `SIGKILL`.

-   Exposed [testStep.attachments](https://playwright.dev/docs/api/class-teststep#test-step-attachments) from the reporter API to allow retrieval of all attachments created by that step.

-   New option `pathTemplate` for `toHaveScreenshot` and `toMatchAriaSnapshot` assertions in the [testConfig.expect](https://playwright.dev/docs/api/class-testconfig#test-config-expect) configuration.

#### UI updates

-   Updated default HTML reporter to improve display of attachments.
-   New button for picking elements to produce aria snapshots.
-   Additional details (such as keys pressed) are now displayed alongside action API calls in traces.
-   Display of `canvas` content in traces is error-prone. Display is now disabled by default, and can be enabled via the `Display canvas content` UI setting.
-   `Call` and `Network` panels now display additional time information.

#### Breaking

-   [expect(locator).toBeEditable()](https://playwright.dev/docs/api/class-locatorassertions#locator-assertions-to-be-editable) and [locator.isEditable()](https://playwright.dev/docs/api/class-locator#locator-is-editable) now throw if the target element is not `<input>`, `<select>`, or a number of other editable elements.
-   Option [testConfig.updateSnapshots](https://playwright.dev/docs/api/class-testconfig#test-config-update-snapshots) now updates all snapshots when set to `all`, rather than only the failed/changed snapshots. Use the new enum `changed` to keep the old functionality of only updating the changed snapshots.

#### Browser Versions

-   Chromium 133.0.6943.16
-   Mozilla Firefox 134.0
-   WebKit 18.2

This version was also tested against the following stable channels:

-   Google Chrome 132
-   Microsoft Edge 132

</details>

---

### Configuration

📅 **Schedule**: Branch creation - "* 0-3 1 * *" (UTC), Automerge - "* 0-3 * * *" (UTC).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOS4xMzYuMSIsInVwZGF0ZWRJblZlciI6IjM5LjEzNi4xIiwidGFyZ2V0QnJhbmNoIjoiZm9yZ2VqbyIsImxhYmVscyI6WyJkZXBlbmRlbmN5LXVwZ3JhZGUiLCJ0ZXN0L25vdC1uZWVkZWQiXX0=-->

Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/6756
Reviewed-by: Earl Warren <[email protected]>
Co-authored-by: Renovate Bot <[email protected]>
Co-committed-by: Renovate Bot <[email protected]>
humaton pushed a commit to fedora-infra/forgejo-deployment-images that referenced this issue Feb 18, 2025
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [@playwright/test](https://playwright.dev) ([source](https://github.com/microsoft/playwright)) | devDependencies | minor | [`1.49.1` -> `1.50.1`](https://renovatebot.com/diffs/npm/@playwright%2ftest/1.49.1/1.50.1) |

---

### Release Notes

<details>
<summary>microsoft/playwright (@&#8203;playwright/test)</summary>

### [`v1.50.1`](https://github.com/microsoft/playwright/releases/tag/v1.50.1)

[Compare Source](microsoft/playwright@v1.50.0...v1.50.1)

##### Highlights

microsoft/playwright#34483 - \[Feature]: single aria snapshot for different engines/browsers
microsoft/playwright#34497 - \[Bug]: Firefox not handling keepalive: true fetch requests
microsoft/playwright#34504 - \[Bug]: update snapshots not creating good diffs
microsoft/playwright#34507 - \[Bug]: snapshotPathTemplate doesnt work when multiple projects
microsoft/playwright#34462 - \[Bug]: updateSnapshots "changed" throws an error

#### Browser Versions

-   Chromium 133.0.6943.16
-   Mozilla Firefox 134.0
-   WebKit 18.2

This version was also tested against the following stable channels:

-   Google Chrome 132
-   Microsoft Edge 132

### [`v1.50.0`](https://github.com/microsoft/playwright/releases/tag/v1.50.0)

[Compare Source](microsoft/playwright@v1.49.1...v1.50.0)

#### Test runner

-   New option [`timeout`](https://playwright.dev/docs/api/class-test#test-step-option-timeout) allows specifying a maximum run time for an individual test step. A timed-out step will fail the execution of the test.

    ```js
    test('some test', async ({ page }) => {
      await test.step('a step', async () => {
        // This step can time out separately from the test
      }, { timeout: 1000 });
    });
    ```

-   New method [test.step.skip()](https://playwright.dev/docs/api/class-test#test-step-skip) to disable execution of a test step.

    ```js
    test('some test', async ({ page }) => {
      await test.step('before running step', async () => {
        // Normal step
      });

      await test.step.skip('not yet ready', async () => {
        // This step is skipped
      });

      await test.step('after running step', async () => {
        // This step still runs even though the previous one was skipped
      });
    });
    ```

-   Expanded [expect(locator).toMatchAriaSnapshot()](https://playwright.dev/docs/api/class-locatorassertions#locator-assertions-to-match-aria-snapshot-2) to allow storing of aria snapshots in separate YAML files.

-   Added method [expect(locator).toHaveAccessibleErrorMessage()](https://playwright.dev/docs/api/class-locatorassertions#locator-assertions-to-have-accessible-error-message) to assert the Locator points to an element with a given [aria errormessage](https://w3c.github.io/aria/#aria-errormessage).

-   Option [testConfig.updateSnapshots](https://playwright.dev/docs/api/class-testconfig#test-config-update-snapshots) added the configuration enum `changed`. `changed` updates only the snapshots that have changed, whereas `all` now updates all snapshots, regardless of whether there are any differences.

-   New option [testConfig.updateSourceMethod](https://playwright.dev/docs/api/class-testconfig#test-config-update-source-method) defines the way source code is updated when [testConfig.updateSnapshots](https://playwright.dev/docs/api/class-testconfig#test-config-update-snapshots) is configured. Added `overwrite` and `3-way` modes that write the changes into source code, on top of existing `patch` mode that creates a patch file.

    ```bash
    npx playwright test --update-snapshots=changed --update-source-method=3way
    ```

-   Option [testConfig.webServer](https://playwright.dev/docs/api/class-testconfig#test-config-web-server) added a `gracefulShutdown` field for specifying a process kill signal other than the default `SIGKILL`.

-   Exposed [testStep.attachments](https://playwright.dev/docs/api/class-teststep#test-step-attachments) from the reporter API to allow retrieval of all attachments created by that step.

-   New option `pathTemplate` for `toHaveScreenshot` and `toMatchAriaSnapshot` assertions in the [testConfig.expect](https://playwright.dev/docs/api/class-testconfig#test-config-expect) configuration.

#### UI updates

-   Updated default HTML reporter to improve display of attachments.
-   New button for picking elements to produce aria snapshots.
-   Additional details (such as keys pressed) are now displayed alongside action API calls in traces.
-   Display of `canvas` content in traces is error-prone. Display is now disabled by default, and can be enabled via the `Display canvas content` UI setting.
-   `Call` and `Network` panels now display additional time information.

#### Breaking

-   [expect(locator).toBeEditable()](https://playwright.dev/docs/api/class-locatorassertions#locator-assertions-to-be-editable) and [locator.isEditable()](https://playwright.dev/docs/api/class-locator#locator-is-editable) now throw if the target element is not `<input>`, `<select>`, or a number of other editable elements.
-   Option [testConfig.updateSnapshots](https://playwright.dev/docs/api/class-testconfig#test-config-update-snapshots) now updates all snapshots when set to `all`, rather than only the failed/changed snapshots. Use the new enum `changed` to keep the old functionality of only updating the changed snapshots.

#### Browser Versions

-   Chromium 133.0.6943.16
-   Mozilla Firefox 134.0
-   WebKit 18.2

This version was also tested against the following stable channels:

-   Google Chrome 132
-   Microsoft Edge 132

</details>

---

### Configuration

📅 **Schedule**: Branch creation - "* 0-3 1 * *" (UTC), Automerge - "* 0-3 * * *" (UTC).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOS4xMzYuMSIsInVwZGF0ZWRJblZlciI6IjM5LjEzNi4xIiwidGFyZ2V0QnJhbmNoIjoiZm9yZ2VqbyIsImxhYmVscyI6WyJkZXBlbmRlbmN5LXVwZ3JhZGUiLCJ0ZXN0L25vdC1uZWVkZWQiXX0=-->

Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/6756
Reviewed-by: Earl Warren <[email protected]>
Co-authored-by: Renovate Bot <[email protected]>
Co-committed-by: Renovate Bot <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants