Skip to main content

Appt Evaluation Methodology (Appt-EM)

Current version: 1.1.0, last updated on 4 July 2023.

The Appt Foundation has created the Appt Evaluation Methodology (Appt-EM) to determine the accessibility of apps in a uniform and reproducible manner.

Worldwide legislation for digital accessibility sets requirements for apps. Most legislation refers to the Web Content Accessibility Guidelines 2.1 (WCAG 2.1). The recommended evaluation method is the Website Accessibility Conformance Evaluation Methodology (WCAG-EM).

However, both the WCAG and WCAG-EM are written for websites and therefore not easily applicable to apps. Appt-EM is written specifically for apps and is based on WCAG-EM.

Basic conditions

To successfully use Appt-EM it is assumed that there is a good understanding of WCAG and accessibility features of Android and iOS.

Evaluation procedure

This chapter describes the steps and activities of the evaluation procedure. At each step you perform several activities. You may have to take one or more steps back when you have gained new insights.

The steps of Appt-EM are similar to those of WCAG-EM:

  • Step 1 – Define the evaluation scope

  • Step 2 – Explore the app

  • Step 3 – Select a representative sample

  • Step 4 – Audit the selected sample

  • Step 5 – Report the evaluation findings

Step 1 - Define the evaluation scope

In this step, the scope of the evaluation is determined. To determine the scope, it must be clear what is within and outside the scope of the research.

Step 1a - Define the scope of the app

Determining the scope starts by determining the URL where the app can be downloaded. Then, an overview of which screens are available must be gained.

Determine operating system

Apps are often available for Android and iOS. Determine on which operating system you are conducting the research. A separate audit must be done for each operating system. We recommend providing a separate report for each operating system.

Determine installation method

An app can often be downloaded from an app store and then has a unique download link. It is important to copy the full download link, including all URL parameters. There may be different versions of an app available per country. In addition, there may also be different variants available, such as a 'lite' version.

If an app has not (yet) been published, it can also be made available via an installation file. The installation file is called an Android Application Package (.apk) on Android and an iOS App Store Package (.ipa) on iOS. In this case, include the download link and/or the full name of the installation file.

Determine version number

You can find the version number in the app store. After installation, you can also find the version number in the settings of your device. The version number is often also stated somewhere in the app.

Determine screens

Apps usually do not have a sitemap with an overview of all screens. In addition, screens usually do not have a unique URL. Therefore, record the path you took to get to the screen. Use the names of the screens for this, for example: Home → Profile → Edit. If a screen does not have a name, you will have to come up with a descriptive name yourself.

You can determine the available screens within an app in various ways. Usually, it comes down to navigating through all the screens of the app.

Step 1.b – Define the conformance target

Determine the conformance level being tested: level A, AA or AAA. The AA conformance level is widely used. European legislation also refers to this level for government organizations.

Step 1.c – Define the scope of the hardware/software

Determine hardware scope

Depending on the hardware, an app may respond differently.

  • Determine the devices

  • Determine the screen dimensions*.*

Determine software scope

Depending on the software, an app may respond differently. It is important to note the version number of the operating system.

Apps have accessibility features built into the operating system. Accessibility features vary by manufacturer. Assistive technologies such as the screen reader, voice control, switch control, and keyboard control often have their own version number, make a note of this.

Finally, note relevant system settings that affect the behavior of the app, such as language and location.

We recommend at least testing with larger text and the screen reader. Additionally, you can test with keyboard, voice control, switch control, and other available accessibility features.

  • Determine operating system version number

  • Determine accessibility features and their version numbers

  • Determine relevant system settings

  • Determine which tools and accessibility features to test with.

Step 2 – Explore the app

This step determines the use, purpose and functionality of the app. For this, the screens need to be examined in more detail.

Step 2.a – Identify frequently used screens

Determine if there are screens that users encounter multiple times when using the app. And determine which screens users spend the most time on.

In any case, choose screens that contain functionality that is essential to use the app, such as registering, logging in, resetting password, etc. Screens that are directly accessible from the main screen of the app are often also important to investigate.

Step 2.b – Identify essential functionality

Determine which flows are essential in the app. A flow consists of several steps or screens that you have to go through to perform a task. By describing the important flows of the app, you can then test all screens of these flows.

Step 2.c – Identify frequently used elements

Determine what kind of elements are used in the app. The table below lists common Android, iOS, Flutter, and React Native elements.

ElementAndroidiOSFlutterReact Native
ImageImageViewUIImageViewImageImage
VideoMediaPlayerAVPlayerVideoPlayerVideo
AudioMediaPlayerAVPlayerAudioPlayerAudio
TextTextViewUILabelTextText
ButtonButtonUIButtonTextButtonButton
LinkURLSpanNSAttributedStringTextSpanText
Input fieldEditTextUITextFieldTextFieldTextInput
CheckboxCheckBoxUISwitchCheckboxSwitch
NavigationToolbarUINavigationBarAppBarNavigationBar
TabsTabLayoutUITabBarAppBarBottomTabs

Step 2.d – Identify technologies relied upon

Determine which techniques were used to develop the app.

  • Programming languages, for example Swift or Kotlin

  • Frameworks, for example SwiftUI or React Native

  • Libraries, for example Alamofire or Expo

Step 2.e – Identify other relevant screens

Determine whether there are specific functionalities or screens for users with disabilities. Also add screens explaining the app or contact information to the sample.

Step 3 – Select a representative sample

Select the screens that are part of the research. You can choose to examine all screens of the app. Or you can take a representative sample consisting of a structured sample and randomly selected screens.

Step 3.a – Include a structured sample

With a structured sample you select screens that are a good reflection of all screens of the app.

Select the screens as follows:

  1. The main screens as determined in step 2.a

  2. The other relevant screens as determined in step 2.e

  3. If the screens from steps 2.a and 2.e are not sufficient for a representative structured sample, add these screens:

    • Screens with essential functionality from step 2.b

    • Screens with common elements from step 2.c

    • Screens using the techniques from step 2.d

Step 3.b – Include a randomly selected sample

Randomly choose one or more screens to check whether the structured sample accurately reflects the app. The number of random screens must be at least 10% of the number of screens in the sample.

For example: with 20 screens in the sample, you must add at least 2 randomly selected screens for checking.

Step 3.c – Include flows

See if any screens from steps 3.a and 3.b are part of a flow. Add the screens of the detected flows to the selection.

Apps mainly consist of multi-screen flows. So, you may eventually have to test (almost) all screens.

Step 4 – Audit the selected sample

Audit the screens according to the chosen conformance level from step 1.b.

Perform the following steps:

  • Step 4.a Evaluate all screens

  • Step 4.b Assess all flows

  • Step 4.c Compare structured and random samples

Note: Unlike websites, it is important to include screenshots when dealing with apps. If you reload the screen, the content may be different the next time, making it irreproducible without a screenshot.

Step 5 – Report the evaluation findings

An audit report must meet certain formal requirements. See Chapter 5 of WCAG-EM for further explanation.

For apps there are deviations and recommendations which are explained below. As a result, the reports meet the minimum requirements necessary to reproduce the findings.

Deviations

Names and Definitions

An app has specific names and definitions. For example, there is no web page, but a screen. Use the correct names and definitions for apps.

Report path

Apps usually don't use URLs to identify screens. Instead of the URL, report the path you took to get to the screen.

Recommendations

Provide solutions and code samples

The WCAG Success Criteria definitions only include solutions and code samples for websites. Provide solutions and code samples for apps, for example by referencing relevant pages from the Appt knowledge base. You can use our WCAG shortlinks, e.g. appt.li/1.1.1 refers to our explanations for success criterion 1.1.1.

Provide screenshots

With apps, it's important to add screenshots to the report. Apps are much more dynamic than websites making it difficult to reproduce the error. A screenshot shows exactly what the app looked like at the time of reporting.

Appendix I: Accessibility requirements from the EN 301 549 v3.2.1

The European Directive EN 301 549 V3.2.1 (2021-03) contains accessibility requirements for software, including mobile applications (apps). Apps are listed in the category 'Open functionality'. Chapter 11 includes 44 success criteria from the Web Content Accessibility Guidelines 2.1 (WCAG 2.1). This is 6 less compared to websites.

Closed and Open Functionality

The European Directive refers to "Open functionality" and "Closed functionality". The given definition for "Closed functionality" is: "Computers that do not allow end-users to adjust settings or install software are functionally closed." Since many accessibility tools can be used in the settings of the mobile phone, we see mobile phones with the corresponding apps as "Open functionality".

Success criteria with deviation

Chapter 11 includes all Level A and AA success criteria from WCAG 2.1. In total, 6 success criteria are not applicable. With 13 success criteria there are small adjustments in the notes or definitions. The thinking and purpose behind these success criteria is the same, only the term “webpage” has been replaced by “software”.

Note: Apps only have to meet 44 success criteria instead of 50.

An overview of the deviations:

The table below contains a complete overview of the comments per success criterion.

Success criterionLevelComment from EN 301 549
1.1.1 Non-text ContentANo deviations
1.2.1 Audio-only and Video-only (Prerecorded)ANo deviations
1.2.2 Captions (Prerecorded)ANo deviations
1.2.3 Audio Description or Media Alternative (Prerecorded)ANo deviations
1.2.4 Captions (Live)AANo deviations
1.2.5 Audio Description (Prerecorded)AANo deviations
1.3.1 Info and RelationshipsANo deviations
1.3.2 Meaningful SequenceANo deviations
1.3.3 Sensory CharacteristicsANo deviations
1.3.4 OrientationAANo deviations
1.3.5 Identify Input PurposeAANo deviations
1.4.1 Use of ColorANo deviations
1.4.2 Audio ControlASmall adjustments in the notes or definitions
1.4.3 Contrast (Minimum)AANo deviations
1.4.4 Resize TextAANo deviations
1.4.5 Images of TextAANo deviations
1.4.10 ReflowAASmall adjustments in the notes or definitions
1.4.11 Non-text ContrastAANo deviations
1.4.12 Text SpacingAANo deviations
1.4.13 Content on Hover or FocusAANo deviations
2.1.1 KeyboardANo deviations
2.1.2 No Keyboard TrapASmall adjustments in the notes or definitions
2.1.4 Character Key ShortcutsANo deviations
2.2.1 Timing AdjustableASmall adjustments in the notes or definitions
2.2.2 Pause, Stop, HideASmall adjustments in the notes or definitions
2.3.1 Three Flashes or Below ThresholdASmall adjustments in the notes or definitions
2.4.1 Bypass BlocksANot applicable to apps
2.4.2 Page TitledANot applicable to apps
2.4.3 Focus OrderASmall adjustments in the notes or definitions
2.4.4 Link Purpose (In Context)ANo deviations
2.4.5 Multiple WaysAANot applicable to apps
2.4.6 Headings and LabelsAANo deviations
2.4.7 Focus VisibleAANo deviations
2.5.1 Pointer GesturesASmall adjustments in the notes or definitions
2.5.2 Pointer CancellationASmall adjustments in the notes or definitions
2.5.3 Label in NameANo deviations
2.5.4 Motion ActuationANo deviations
3.1.1 Language of PageAASmall adjustments in the notes or definitions
3.1.2 Language of PartsAANot applicable to apps
3.2.1 On FocusANo deviations
3.2.2 On InputANo deviations
3.2.3 Consistent NavigationAANot applicable to apps
3.2.4 Consistent IdentificationAANot applicable to apps
3.3.1 Error IdentificationANo deviations
3.3.2 Labels or InstructionsANo deviations
3.3.3 Error SuggestionAANo deviations
3.3.4 Error Prevention (Legal, Financial, Data)AASmall adjustments in the notes or definitions
4.1.1 ParsingASmall adjustments in the notes or definitions
4.1.2 Name, Role, ValueASmall adjustments in the notes or definitions
4.1.3 Status MessagesAANo deviations

For the full explanation, see Chapter 11 of the European Directive EN 301 549 V3.2.1 (2021-03).