WARP: Web Accessibility Reporting Project Ireland 2002 Baseline Study
4 November 2002 [Document also available in PDF Format.]
- Executive Summary
- 1 What is Web Accessibility?
- 2.1 The W3C WCAG Accessibility Guidelines
- 2.2 Delimiting the Irish Web Space
- 2.3 Sampling the Irish Web Space
- 2.4 Delimiting Individual Web Sites
- 2.5 Sampling Individual Web Sites
- 2.6 System Architecture
- 2.7 Accessibility Measures and Benchmarks
- 2.8 Experimental Runs
- 3 Processing Platform Requirements
- 4 Irish Web Accessibility Benchmarks
5 Defect Type Analysis
5.1 WCAG Priority 1
g9(90.6% of sites)
g39(34.0% of sites)
g38(33% of sites)
g240(26.4% of sites)
g10(18.2% of sites)
g21(10.7% of sites)
g20(0.6% of sites)
- 5.2 WCAG Priority 2 Defects
- 5.3 WCAG Priority 3 Defects
- 5.1 WCAG Priority 1 Defects
- 6 Future Work
- 7 Conclusion
- A. Complete listing of initial target sites
bobbyFull Support Diagnostics vs. WCAG Checkpoints
- C. Template
- D. Failed
- E. Unsatisfactory
- F. Successful
- G. Failed
- H. Successful
If anybody asks me what the Internet means to me, I will tell him without hesitation: To me (a quadriplegic) the Internet occupies the most important part in my life. It is my feet that can take me to any part of the world; it is my hands which help me to accomplish my work; it is my best friend--it gives my life meaning.
- Dr. ZhangXu (Zhangxu & Aldis, 2001)
The technology of the Internet holds tremendous promise to significantly improve access to information, goods, and services for many people with disabilities. Properly engineering web sites can interoperate with dedicated assistive technologies to flexibly address a wide range of disabilities. Almost overnight, it has become possible for a blind person to read papers, magazines and books, without assistance, and on the same day they are published; for a person with impaired mobility to shop for groceries, and pay her bills without even leaving home; for a deaf student to attend a "virtual" lecture, with sub-titling and text transcripts.
The key is in the design of web sites so that they facilitate--rather than obstruct--access by users with disabilities. This is not rocket science: the basic requirements have been codified since 1999 in the Web Content Accessibility Guidelines (WCAG) 1.0 published by the World Wide Web Consortium (W3C, 1999), and more recently endorsed by the European Commission (2001) and the Irish National Disability Authority (2002). These define a series of checkpoints which, if satisfied by a web site, will ensure that it has a high likelyhood of being accessible to the widest possible variety of users. This is good for the disability community; but it is also good for the general community of web users: it is well established that universal design frequently results in products and services that are more usable for all. And in the world of the web, where another site is only ever a key-click or a button-press away, improved usability must be a key priority for all web site operators. It is the proverbial win-win situation.
Nonetheless, though this is the promise: it is already under threat.
WCAG defines three conformance level: WCAG-A is a minimum standard which a site must meet to be considered accessible for any significant disability groups; WCAG-AA is a "professional practice" standard, which all sites should meet to be accessible to a broad range of disability groups; finally WCAG-AAA is a "gold standard" of maximum accessibility which some sites may choose to aim for--for example, sites with a particular remit to serve disability communities.
This report documents a study of over 159 separate web sites, operated by Irish organisations, and spanning a wide range of activities, information, and services. These were assessed for a set of characteristics correlated with the WCAG guidelines. This set is not exhaustive: it cannot determine that any site positively meets the guidelines; but failure on any of these tests definitively demonstrates failure against the guidelines.The key results are that, of this sample, at least 94% failed to meet even the minimum WCAG-A standard; and 100% failed to meet the professional practice WCAG-AA standard. Furthermore, at least 90% of sites failed to meet minimal conformance with generic technical standards for web interoperability.
This should be a "wake-up call"--for government, for public agencies, for private companies, organisations and individuals. It signals that, despite Ireland's justifiable pride in its economic and technological development, despite very laudable goals in documents such as the E-Europe Action Plan (European Commission, 2000,2001), the current commitment to accessibility of the Irish web for users with disabilities is, at best, aspirational--and, at worst, cynically inadequate.
On the other hand, this study also indicates that significant progress could be made quickly, and with comparatively little effort--if decision makers will recognise the significance of the issue and take action accordingly. Detailed analysis of the survey data identifies a number of specific, pervasive, web design flaws which can significantly obstruct accessibility by users with disabilities; but which could be drastically reduced or eliminated if web site designers and content authors were simply provided with appropriate tools and training.
The report discusses both the strengths and limitations of its particular methodology; and lays out a programme of further work to extend, clarify, and refine our ability to monitor the evolving state of Irish web accessibility--as an essential tool for informed policy formation. Now, of course, there is a need for leadership; for public policy initiatives, particularly in education and training; and for concrete incentives and supports to organisations wishing to improve services for users with disabilities.
Finally, there must surely also be a role for legislation and regulation--to ultimately guarantee and vindicate the rights of all citizens to equal treatment in a digital democracy.
1 What is Web Accessibility?1
The power of the Web is in its universality. Access by everyone regardless of disability is an essential aspect...
- Tim Berners-Lee, W3C Director and inventor of the World Wide Web. (W3C, 1997)
The explosive arrival of the Internet/World Wide Web2 as a mass communication medium makes it a little difficult to see it yet in full perspective, or to properly gauge its impact. For most of us it is at least a somewhat useful tool, giving more immediate access to many information sources and services. It is also often awkward and frustrating-with slow responses, complex and confusing interactions, incompatible software requirements, and all the other characteristics of any immature technology.
But for many people with disabilities the web is more than just another technological toy-it can offer the potential for significant improvement in their access to products and services that the more abled community take for granted. Here are a few brief examples:
- A student who is deaf might enroll in an online class where lectures and other audio materials are made available with sub-titling and full text transcripts etc.
- A person who has restricted mobility can compare prices across online shops, and order a wide variety of goods for home delivery; or (virtually) visit a bank to pay bills, arrange a loan etc.
- A blind user can now have books, newspapers and magazines "read" to them on demand--by accessing them through a web browser equipped with automatic text to speech synthesis.
Other more elaborated and detailed scenarios are available in an illuminating guide to How People with Disabilities Use the Web (W3C, 2001). However, as that document also makes clear, while the web opens up very promising new possibilities for supporting and including users with disabilities in the benefits of the emerging information society, these outcomes are by no means automatic: they rely on the services and facilities being designed in an inclusive way that supports and facilitates all categories of users.
To give a more familiar example: wheelchairs are an assistive technology with the potential to dramatically improve the mobility of people with specific physical disabilities. However, if buildings are designed without ramps or accessible lifts and doors, then the effectiveness of a wheelchair is, of course, dramatically reduced.
In the same way, assistive technologies such as speech synthesizers, alternative keyboard and pointing devices, voice recognisers etc., can all help people with various disabilities to access computer devices in general, and to access the Internet in particular. But if web services are not designed in such a way as to facilitate and inter-operate with these assistive devices, then their benefit may be sharply reduced.
Here is a simple--and pervasive--example. Many web sites are enriched with small graphical images or "icons" for various functions, such as searching and navigation. This immediately presents a potentially serious obstacle for a blind user, who, of course, cannot see these icons. While a speech synthesizer can automatically read out any text on the page, it cannot in general recognise or describe arbitrary images in any meaningful way. Fortunately there is a relatively trivial way of resolving this. The underlying HTML format of web pages makes provision for attaching "alternative" or ALT text to images. This is generally hidden if the image is displayed or visible to the user; but it can be picked up and spoken by the speech synthesizer for a blind user. The extra design effort involved is minimal--but dramatically improves the usability of the site for the affected users.3
There are many other ways in which web sites and services can be designed in an inclusive way that facilitates access by all users regardless of disability or the need for particular assistive technologies. These have been codified in various guidelines formulated by the World Wide Web Consortium (W3C)--in particular, the Web Content Accessibility Guidelines version 1.0, WCAG 1.0 (W3C, 1999).
While compliance to such guidelines is essentially voluntary at the moment, this is likely to evolve on an ongoing basis. For example, the Irish Information Society Commission has stated that "... [w]here organisations are designing web-sites care must be taken to ensure that they are accessible to as broad a section of the population as possible" (Irish Information Society Commission, 2000). More recently, the e-Europe Action Plan (European Commission, 2000,2001) makes a commitment to the adoption and (ultimately) application of the WCAG guidelines to all public sector web sites in Europe.
To inform policy and promote action on this agenda it is important to have information about the state of accessibility of web sites, and to actively monitor trends in accessibility over time. This is a large scale task which should ultimately involve a range of different instruments, methods and approaches, which will themselves evolve over time.
This report documents an initial, baseline survey of the overall accessibility of Irish web sites. The particular approach taken has both strengths and weaknesses: however it is hoped that it can initiate a process of engagement, debate, and action by all Irish providers of web services, not merely to "accommodate" users with disabilities, but to reach out and positively include them--as of right--in the emerging Information Society.
The starting point for this study is the W3C set of Web Content Accessibility Guidelines version 1.0, WCAG 1.0 (W3C, 1999). The question being researched is then the extent to which Irish based web sites conform to these guidelines.
It is important to note that this question is already at one remove from actual user experience. That is, instead of direct testing of web sites by users with various disabilities, we are using the WCAG guidelines as a proxy indicator of accessibility. In particular, we are here relying on the prior work in devising these guidelines to assure their quality and relevance to real accessibility: but we note that the guidelines themselves are under ongoing review, and that research which actively investigates the correlation between WCAG conformance and accessibility is an essential complement to the work presented here.
WCAG consists of 14 separate guidelines, each of which has an associated set of one or more individual checkpoints. There are a total of 65 checkpoints which are classified into three priority levels (W3C, 1999, Section 4):
- Priority 1:
- A Web content developer must satisfy this checkpoint. Otherwise, one or more groups will find it impossible to access information in the document. Satisfying this checkpoint is a basic requirement for some groups to be able to use Web documents.
- Priority 2:
- A Web content developer should satisfy this checkpoint. Otherwise, one or more groups will find it difficult to access information in the document. Satisfying this checkpoint will remove significant barriers to accessing Web documents.
- Priority 3:
- A Web content developer may address this checkpoint. Otherwise, one or more groups will find it somewhat difficult to access information in the document. Satisfying this checkpoint will improve access to Web documents.
WCAG conformance levels are then defined on the basis of these priorities (W3C, 1999, Section 5):
- Conformance Level A:
- all Priority 1 checkpoints are satisfied.
- Conformance Level AA:
- all Priority 1 and 2 checkpoints are satisfied.
- Conformance Level AAA:
- all Priority 1, 2, and 3 checkpoints are satisfied.
Thus, to fully assess conformance with WCAG potentially requires an assessment of conformance with each and every checkpoint by each and every web "page" of the site(s) under consideration. In general, this requires detailed testing and evaluation, against each checkpoint, by an expert, human, tester. This clearly makes large scale, exhaustive, conformance testing a slow and potentially costly activity. Whereas, for the purposes of the current study, it is essential to be able to scale up to examinations of large collections of web sites (each of which can individually consist of many separate pages).
However: there are a number of checkpoints (or aspects of checkpoints) where it is possible to test conformance "mechanically"--in effect, by submitting the page(s) to an automated, software, testing agent. By using such automated assessment, it is possible to devise a survey methodology that can be effectively and efficiently scaled up to large collections of web sites.
Of course, the price of limiting consideration to a subset of the checkpoints (or ever, certain aspects of certain checkpoints) is that the results are intrinsically incomplete. In particular, it is impossible, using such a methodology, to declare that any site positively conforms to WCAG at any particular level; rather, the strongest potential results will be of the negative form, namely that specific sites definitely do not conform.
Nonetheless, notwithstanding this limitation, for the purposes of policy formulation and implementation, it is much preferable to have available some concrete, comprehensive, data relating to web accessibility on a large scale, even if this is incomplete, and requires careful interpretation. Moreover, the use of such automated assessment instruments carries with it the positive merit of being absolutely objective. Thus, it provides a firm foundation for valid comparisons--between sites, between regions, and between different points in time.
The objective of this study is primarily to inform and promote web accessibility policy in Ireland. However: by its nature, the World Wide Web is a globally distributed network. Thus, a web "site" (or "server") can be physically situated in a location which is geographical arbitrary, relative to the locations both of the operating organisations and of the users of those sites. Thus, it is not immediately obvious what criteria to apply in classifying sites as "Irish", nor how to employ such criteria in locating or identifying such sites.
The general criterion that has been applied here is to classify sites as Irish if (and only if) the organisation responsible for the site is incorporated in this jurisdiction (i.e., would be legally bound by the laws and courts of Ireland). This is applied regardless of the physical locations of the servers or the users.
This criterion is based on the view that only such sites are likely to be directly affected by Irish policy making and initiatives: however, it must be recognised that this does give a perspective which is then somewhat skewed relative to that of a "typical" Irish user. Such users will very often access sites which are not Irish in this sense, but which do offer information, services, or tangible goods to Irish users, and whose accessibility thus does directly impact on their web experiences.
In any case, this is only a criterion: we must now attempt (somehow) to generate listings of candidate sites matching this criterion.
One natural approach to this problem is to start with information from the Internet's own Domain Name System (DNS). This is the distributed database which contains administrative and management information about every named host computer which is connected to the Internet.
The DNS is organised as a hierarchy; the highest level
contains the so-called Top Level Domains, or TLDs. These are
subdivided into "geographical" and "non-geographical". There
are defined geographical codes for each internationally
recognised, independent legal state; that for Ireland is the
".ie". The non-geographical TLDs
".net" etc. While these are most appropriate
for use by organisations operating across national
boundaries, there is no technical constraint enforcing this.
In practice, individual organisations may opt to use a
non-geographical TLD for a variety of more or less pragmatic
reasons (including simple cost--registration of
".ie" names continues to be comparatively more
expensive than names with non-geographical TLDs).
Thus, while it is possible to automatically extract, from
the DNS, a listing of all names within the
".ie" TLD, and it is reasonable to assume that
more or less all of these are "Irish" in our sense, this
will certainly not represent a complete or exhaustive
catalogue of such names. However, since the DNS also
contains additional administrative information, including,
in particular, the address of the registered owner for each
name, it is possible to attempt to probe the
non-geographical TLDs for names associated with Irish
A further significant difficulty in attempting to generate a listing of web sites simply through querying of the DNS, is that the although the DNS contains host names for all publicly accessible web sites, it also contains many other host names which do not correspond to web sites at all. This would include, for example, names which have been registered simply to protect trademarks, or in anticipation of future use. At any given time, the DNS also contains many names which are, in effect, defunct--i.e., which are no longer in use (if they ever were in use) but whose registrations have not yet expired. However, it is possible to automatically address some of these difficulties by the simple expedient of tentatively assuming that a given host name does correspond to a public web site, and then attempting to connect to it on that basis. If this succeeds, it is generally possible to also automatically extract some further technical and administrative information directly from it, which can help to decide whether it represents a functioning web site; conversely, if this connection fails, one can (still tentatively) conclude that the host name does not correspond to a web site.
These general techniques have been used by
develop at least rough estimates of the extent of the Irish
web "space". The following summary data was reported as of
20 March 2002:
- The total number of domain names registered with a TLD
- Of these, approximately 6,000 seemed to have corresponding, functioning, web sites (described as web sites with "Title, Descriptions, Keywords").
- 41,756 domain names with TLDs of
".net"are described as "confirmed Irish-owned"; a further 35,731 domain names in these same TLDs are classified as "Irish-owned, pending confirmation".
- Data on detailed probing of these domain names for
functioning web sites was not provided. However,
if similar proportions applied to those obtaining
".ie"TLD, this would suggest somewhere between 9,000 ("confirmed Irish-owned") and 17,000 ("confirmed + unconfirmed Irish-owned) web sites.
In total then, based purely on DNS data, the Irish web space, across the main relevant TLDs, would appear to consist of between 15,000 and 23,000 individual web sites. However, allowing for the exhaustive nature of the DNS, this is likely to be, if anything, an over-estimate.
A quite different approach to assessing the scope of the Irish web space is through analysis of manually constructed directories of web sites. In contrast to the automated, mechanical character of the DNS approach, entries in such directories must generally be originated, and perhaps explicitly reviewed or evaluated, by human "editors". Their coverage is therefore likely to be significantly less exhaustive (i.e., providing, if anything, an under-estimate of overall scale); but entries are likely to much more consistently correspond to genuine web sites providing some substantive information or service.
One of the largest such directories is that maintained by the Open Directory Project (ODP). Indeed, the ODP provides the core data which is used as the basis for a number of other well known, commercially oriented, Internet directories, such as Google. The ODP is an open, collaboratively maintained, global directory of web sites, organised by a variety of categories, including geographical location.
In total, the ODP geographical category for "Ireland" contains 7,1476 entries (as of 21st April 2002). Of course, not all of these would necessarily be "Irish" in our sense, but the great majority can reasonably be assumed to be. While this number is rather smaller than the 15,000-23,000 estimate based on raw DNS data, it is of a similar order of magnitude. Thus, finally combining the two approaches, suggests an overall estimate of perhaps 10,000 distinct sites as the current size of the Irish web.
This data of the previous section provides a useful gross indication of the overall scale of the challenge of surveying the Irish web space (for accessibility, or any other purpose).
In the context of the current study, it was clear that attempting an exhaustive study would be technically very demanding; and that even direct (unbiased) sampling based on this data would probably not be appropriate. Web sites differ from each other in many ways: size, function, utility, popularity etc. It is inevitable that a relatively much smaller subset of the total sites identifiable via the methods of the previous section are actually significant to the needs of typical users.
Ideally, one might therefore attempt to identify the most relevant Irish sites by consideration of relative "traffic" or "activity". However, such an analysis is very difficult in the general case. There is considerable debate as to what the most appropriate measures of activity levels should be. Objectively certified usage data is not widely available. And even where usage data is available it is typically intended to support commercial objectives (specifically, Internet based marketing), whereas, many important web services are provided by public sector or otherwise non-commercial organisations (which may be therefore be seriously under-represented in such data).
Accordingly, the pragmatic approach taken here was to create a sample of the Irish Web based on largely subjective judgments of experienced web users (drawn from within the project team). The ODP category for Ireland was used as a starting point, but sites were also identified from a variety of other sources, including other directories, public advertising etc.
A target sample size of approximately 200 distinct sites was set for this current (initial) study. This was judged to be large enough to give a reasonably wide distribution across sectors and service types and serve as a proof of principle of the methodology and technology being used. Scaling up to significantly larger surveys can be considered subsequently, in the light of the experience with this initial baseline.
The following informal categories were taken into account in selecting specific sites:
- Government and other public sector sites: being publicly funded, there is an intrinsic onus on such sites to respect accessibility for all citizens; furthermore, this is explicitly reflected in the accessibility objectives of the e-Europe Action Plan (European Commission, 2001).
- Political parties: directly influence policy, regulation, and legislation.
- Agencies, state or voluntary, with particular responsibilities for services to users with disabilities.
- Educational institutions: both because of the significance of their services to users with disabilities, and their potential influence on future decision makers.
- Media organisations: again their services may be particularly significant for users with disabilities, and they also strongly influence public policy formation.
- Major Irish companies (PLCs): both because of their direct significance in their own commercial fields, and also their wider influence on practice and standards in the commercial sphere.
- Travel related organisations: physical mobility is a significant issue for people with a wide range of disabilities, so access to information on travel services can be particularly important.
- Telecommunications, IT and Web design/hosting companies: because of their particular impact on the engineering and design of web sites generally.
- A variety of other less focussed categories were also considered: entertainment, sports, trade unions, legal, medical etc.
The final list of target sites contains 214 entries; it is presented in full in Appendix A. It should be emphasised that this listing is not presented as a "statistically representative" sample; extrapolations to the overall Irish web space must therefore be considered very carefully. Nonetheless, and subject to further critical testing in future studies, it is conjectured that overall patterns from this sample can usefully guide national policy development and implementation.
Just as the web itself is globally distributed and hyperlinked, and lacking clear boundaries, the scope of a "single" web site is typically hard to define--both in principle and in practice.
It is generally reasonable to suppose that all components of a single web site should be managed by a single organisation, so that hypertext links pointing outside of that organisational responsibility do not belong to the same site--although even this principle can be violated where an organisation outsources certain aspects of its services. A more serious problem is that the delimiting of a "single" organisation can itself be problematic. Thus a large organisation will often have subdivisions, subsidiaries, partnerships, projects etc., each of which may have their own associated web information or services. In principle, for our purposes, these should probably be considered as separate web sites if they are subject to separate technical design and maintenance (and thus may be significantly different in terms of accessibility). However, in practice, there is a virtual continuum of degrees of independence; and no reliable technical means to measure this variation externally.
As with the delimiting of the overall Irish web space, concrete, pragmatic, choices had to be made here.
Accordingly, the scope of a single web "site" has been arbitrarily defined relative to its identified top-level host name (this normally corresponds to what is called the site "home page"). Thus starting with a host name of the form, say:
an initial, or "root", Uniform Resource Locater (URL) is generated as:
The page at this URL will, in general, contain hypertext links to other pages. These hypertext links are classified as falling within the "same" web site if, and only if, their URLs contain exactly the same host name. Those secondary pages may, in turn, contain further links; and the same criterion is then applied to classify these as falling inside or outside the site. In principle, by applying this procedure recursively, the complete scope of the web site (at least, as defined in this particular way) can be mechanically identified.
While this is presented as a reasonable, pragmatic, approach to delimiting single web sites, it does, of course, have some significant limitations. In particular, URLs with a slightly different host name may well properly refer to the same web site. Continuing the example above, one might encounter URLs of the form, say:
which might provide a search facility for the site, and should surely be classified as belonging to the site. However, they would be excluded by our criterion. On the other hand, of course, if one instead required only the "organisation specific" fragment of the host name (the "domain name") to be preserved, one might well include URLs which should really be excluded; for example:
If this corresponded to information or services under completely separate management, it should be classified as outside the original site.
There is no generally reliable method to automatically distinguish these cases; but the choice we have made, being the more restrictive, is likely to ensure that all URLs classified as belonging to a site, do indeed do so--even if it is not exhaustive in probing the limits of the site.
While the method described above provides a criterion for delimiting individual web sites, it does not necessarily follow that one should include the complete scope of every site, identified in this way, in the current study. In particular:
- The automated accessibility tests at the core of this study are applicable only to content in HTML format. Accordingly, content of other media types cannot usefully be processed.
- Assuming that a single site is subject to coherent design and management, accessibility characteristics are likely to be relatively consistent across the site. Accordingly, while it is certainly useful to analyse some significant selection of pages from each site, there is likely to be an effect of "diminishing returns" in identifying further accessibility issues. Accordingly, it would is reasonable to set some overall limit(s) to how much data is processed from a single site.
With these points in mind, the particular approach taken here was as follows:
- Only HTML format pages or resources should be processed.
- Only URLs specifying the
http:protocol or "scheme" should be retrieved. This is a somewhat technical issue, but this is the most common web protocol, and the great majority of HTML content will be addressable in this way; whereas, URLs for other media types will often specify alternative, specialised, protocols. Moreover, while the
http:protocol reliably declares the media type of the resource being accessed (thus allowing filtering for HTML content only), other arbitrary protocols generally do not.
- A limit is imposed on the amount of data processed from any single site. There are two "natural" ways of expressing such a limit: either in terms of the number of separate pages, or in terms of the aggregate quantity of data (in KBytes or MBytes). Given that web sites may differ significantly in the average sizes of individual pages, it was judged that using an aggregate data quantity limit should be preferred. Given that this was an initial, proof of principle, study, and having regard to available resources and typical web page sizes, this limit was set at 200 KBytes per web site.4 Thus, the total amount of raw HTML data processed across all sites would have a maximum of approximately 200×200 or 40 MBytes. This limit could, of course, be varied in future studies.
- A limit is imposed on the "depth" of hypertext linking which is admitted. Very roughly speaking, it is reasonable to suppose that the most "significant" pages of the site (at least from the perspective of the providing organisation) will be linked reasonably directly to the top, or home, page. Accordingly, it was decided that pages which are more than 3 links "away" from the home page should not be processed. Again, this limit could be varied in future studies.
As with the overall sampling of the Irish web "space", the sampling procedures described here are pragmatic, and are not presented as "statistically representative". Thus, results from each such sample cannot be reliably extrapolated to complete sites. However, there is no basis for supposing that accessibility characteristics would be randomly and independently distributed across a site anyway--on the contrary, if anything, there are likely to be very strong correlations. Thus, "statistically representative" sampling is probably not even meaningful in principle.
The overall system for conducting the accessibility survey consists of the following major components:
bobby: WCAG Assessment Tool. This tool mechanically analyses individual HTML pages, or groups of such pages, for conformance with a (subset of) the WCAG guidelines.
pavuk: Web capture "robot". This is a programme for automatically capturing a local copy of a web site, on the basis of specified delimiting criteria.
postgresql: RDBMS system. Records of the web pages captured, and the analyses carried out, are recorded in a database.
- Custom co-ordination and integration tools. These are
programmes developed for the specific purposes of the
study; they sequence the operations of the
bobbytools, injecting all results into the database; programmed queries on the database then allow further analysis and summarising of the overall results. These were generally written in
are described in more detail in the following sections.
There are a number of software products now available to
carry out automated assessments against (subsets of) the
WCAG guidelines. These have a variety of strengths and
weaknesses, but are functionally very similar (by
definition, as they are largely driven by the WCAG
guidelines themselves). For the purposes of this study we
have chosen to adopt one of the most widely deployed of
bobby, originally developed by the
Center for Applied Special
Technology (CAST), and now distributed and maintained by
However, it should be noted that
common with most products in this area, is primarily
focussed on assessment as a prelude to "repairing" of
correcting accessibility problems. Thus it has not been
designed with the special purposes of large scale surveying
(without a direct linkage to repair) in mind, and is not
necessarily particularly well suited to this purpose. This
issue will be returned to in the conclusion of the
The particular version of
bobby used here is
Bobby Worldwide (Core v4.0). This is available both
in an ASP form5(hosted by
Watchfire and controlled via a web interface), and in a
standalone form (hosted on a local server). The latter
provides greater functionality (specifically, the ability to
analyse batches of locally stored pages, and to generate
machine readable, XML reports) and is therefore preferred
for the current study. All further detailed discussion
refers to this version.
bobby implements 91 distinct tests or
diagnostics, each of which maps onto a specific WCAG
checkpoint.6 A number of
bobby diagnostics map onto (different aspects
of) the same checkpoint. The
are classified into a number of different "support"
categories, as follows:
bobbyautomatically checks this guideline.
bobbyperforms some checking and presents line numbers of the HTML code to verify for errors.
- Partial Once:
- Similar to partial but
bobbydoes not present line numbers.
- Ask Once:
bobbydoes not do any checking, the guideline is presented as a reminder to you.
- Summary Ask Once:
- An Ask Once only presented in the Summary report since it only applies to a group of pages (such as Use Consistent Navigation).
For all categories other than Full, further
evaluation would be required by a human assessor to
determine WCAG conformance. Accordingly, in the work
bobby is restricted to
implementing just those diagnostics with Full
support. There are 25 such diagnostics, which map onto
(aspects of) 20 distinct WCAG checkpoints, including some at
all three priority levels. The complete table of these
bobby diagnostics, with WCAG mappings and
priorities, is presented in appendix B.
The command line form of
bobbycl, was used. Custom
scripts were used to invoked
bobbycl once for
each web site in the survey. In each case,
bobby was provided with a list of of local
files to be analysed, corresponding to a local (sampled)
image of the target site (the capture of this local image is
discussed in section 2.6.2 below). The results were emitted by
bobby in the form of an XML formatted file.
This is designed for ease of subsequent machine processing.
In our case, the controlling
perl script then
parsed this output file (using the
perl module) and recorded the required data in
As already discussed, it is necessary in this study to delimit individual web sites, according to some more or less pragmatic technical criteria. The core element of this is a procedure whereby links are extracted from a given page, and each of these is classified for membership of the intended site; the process is then recursively applied to those pages etc.
bobby package has integrated support for
this form of recursive site mapping; however, it provides
only limited control and customisation. In particular, it
does not provide for site sampling limited by data quantity;
nor for configuration of sampling by media type.
Furthermore, when using this integrated mechanism,
bobby downloads web pages for immediate
analysis, directly over the Internet, after which the
downloaded pages are simply discarded. This is
unsatisfactory for our purposes. In general, for test and
debug reasons alone, it is necessary to be able to review
specific pages against the corresponding
generated reports; furthermore, in the general case (not
addressed in the current study) there may be other
additional analyses which it would be desirable to carry out
on the pages. Therefore, it is essential to be able to
retain (at least for some time) local copies of the
Thus, the preferred approach is to have two separate
phases: capture, when copies of the target web
pages/sites are stored locally, followed by
bobby (and/or other
tools) are used to process the pages.
The tool selected for the capture phase is called
pavuk; it is an open source package, released at no charge under
the GNU Public License. It provides for
extensive control and customisation of the capture
The capture phase for the complete survey is implemented
by a custom
perl script. This invokes
pavuk once for each site.
controlled by a customisation file. This is dynamically
tailored, as necessary for each site, but most options are
copied from a single template. This template is presented
for reference in Appendix C. For our purposes, the key configuration
settings in this template are:
- Only HTML pages are captured. This is implemented via
three distinct and complementary configuration settings:
DisableHTMLTag: A variety of HTML elements can specify hyperlinks. However, for some of these the link targets will not be HTML pages--therefore, such links can be discarded without further inspection.
DisallowedSuffixes: If a link URL ends with a suffix that normally indicates a non-HTML media type, then the URL is not processed further. Thus, for example, if the URL ends in
".jpg"this usually indicates that it refers to an image (in the JPEG graphics format). 27 separate suffixes are listed in this way. This provides an efficient, but generally unreliable, level of filtering for HTML content.
AllowedMIMETypes: The MIME type of a web resource is a technical description of its media format. The MIME type is automatically signalled by a web server when a client (browser) requests to download the resource. The MIME type
"text/html"identifies HTML format web pages. If any other MIME type is detected, the page download is aborted. This provides a less efficient, but highly reliable, secondary filtering.
DisallowedSuffixesmechanism is more efficient because the remote web server is not even queried for the resource. It has the disadvantage that as URL suffixes do not indicate media type in general, some URLs which do refer to HTML resources may be wrongly discarded. Moreover, where the URL design for a particular site does not use systematic suffixes at all, this mechanism is simply not applicable. The
AllowedMIMETypesmechanism, on the other hand, is somewhat less efficient (as a transfer from the remote server must be at least initiated for each relevant URL) but, provided the server functions correctly according to the relevant standards, it is guaranteed to reliably restrict the capture to HTML format resources.
TransferQuota: The maximum quantity of data captured per site is set to 200 KBytes.
MaxLevel: The maximum number of links followed (the recursion "depth") is set to 3.
SleepBetween: This forces a delay (of 1s) between successive page downloads. This is a courtesy to limit the impact of the capture activity both on the loading of the remote server and other users of shared bandwidth.
EnableJS: This controls processing of client side scripting.
pavukdoes not include a full script interpreter; but it is capable of attempting heuristic, surface level, scanning of scripting code for embedded hyperlinks. In our case this is set to
falseto disable all script processing. This is a deliberate choice, because WCAG checkpoints 1.1 and 6.3 require that significant functionality provided via client side scripting should also be available with scripting turned off.
The configuration settings tailored for each site are:
URL: This specifies the top level, or root URL for the site. The root URLs for all sites included in this study are listed in Appendix A.
URLMatchPattern: This specifies a URL "pattern" which is used to classify pages as belonging to the same site. As discussed in section 2.4, this is used to specify that only URLs with the same host name (and, indeed, the same protocol or scheme, namely
http:) will be included.
For the purposes of summarising the accessibility
diagnostics generated by
bobby, and especially
for comparing these (between individual pages, sites, whole
surveys etc.), it is desirable to formulate specific
At the most summary level, the approach adopted here is based on the notion of WCAG conformance levels. A given site is classified as non-conformant, at a given level, if it fails one or more checkpoints at that level. The benchmark measures for the complete survey are then defined as the proportions of sites which can be definitively classed as failing to achieve WCAG conformance levels A, AA and AAA.
In considering the significance or severity of individual
bobby diagnostics, clearly all Priority 1
defects take precedence over all Priority 2 defects; and
Priority 2 defects in turn take precedence over all Priority
3 defects. However, within a priority level, one can
reasonably ask whether it is possible to rank the
significance of the different defect types. We elect to do
this by again measuring the proportions of sites classed as
failing (at least once) the individual
tests. These provide at least a rough ranking--in the sense
that comprehensive repair of the highest ranking defects
would affect the greatest proportion of sites.
However: it must be borne in mind that such a ranking is
naturally limited to the specific defect types included in
the study--i.e., those which can be fully and automatically
bobby. Thus, there may be other
defect types, not considered here, which would be of more
significance to the practical accessibility of specific
Note also that these measures do not attempt to probe the relative significance of different defect types within individual sites. The latter sort of analysis will normally need to consider the relative densities of the different defect types in the site--as, presumably, repair effort may depend on how pervasive the defect is. However, for the purposes of the current study, which is focussed on policy, particularly in regulation, education and training, such more detailed profiles of individual sites are not at issue.
The project naturally involves a large number of prototype or developmental runs to test, refine, and debug the overall system functionality. However, for consistency, all detailed data reported here is drawn from just one "definitive" run, and all specific discussion is based on this. The site capture phase took place on 9th April 2002, starting at 16:11:49 IST (UT+01) and ending at 17:59:19 IST (UT+01). Thus, all results relate only to the state of the target sites at that point in time.
This section reviews the computing and communications resources required for a single survey run. This is primarily to inform subsequent discussion of the prospects and issues in extending the work described here: particularly in terms of repeating the survey (to implement comparative longitudinal studies) and increasing the scale (in terms of number of sites, sampling coverage per site etc.).
During the capture phase,
various information regarding each URL encountered, and each
attempted page capture, in a number of plain text log files.
While these files are all preserved for reference, selected
information from them is also inserted into the
The data capture phase had a total duration of 01:47:26, i.e., rather less than two hours. The total data captured was 32,402,205 bytes (31 MBytes). The total number of individual pages captured was 3270; thus the average page size was 9.7 KBytes.
The average data capture rate was
4.9 KByte/s.7 At first sight
this may appear low, given that the work was done using a
broadband Internet connection via the Irish National
Education and Research Network HEAnet; however the effective bandwidth
utilisation was deliberately throttled down by configuring
pavuk, as already mentioned, with a 1s delay
between successive page downloads. This was done to limit
impact both on the host sites and on other users of shared
bandwidth. Clearly, if it was decided to increase the scope
of future studies, this delay could be reduced or
eliminated. Thus, the total capture time need not increase
in direct proportion to either the number of sites or the
data transfer quota per site.
There were 10 sites for which no data was captured; these
are listed in Appendix D. For a further 18 sites, only the initial,
"home" page was captured, indicating that
failed to successfully follow any links from that page;
these are listed in Appendix E. The remaining 186
sites, for which
pavuk achieved some more or
less significant site traversal, are listed in
Some (limited) analysis has been done of the reasons for these problems with site capture; factors which have been identified include:
- Simple communications or connection failures. These probably indicate transient problems either with the target site, or communications pathways to it. In any study on a significant scale it is inevitable that some site captures will fail in this way. However, it does suggest, particularly if scaling up, that it would be beneficial to allow for further capture attempts on suspect sites, perhaps 24 or 48 hours after the original attempt.
- A site may rely on client side scripting to establish
hyperlinks within the site. Such links will not be
pavukhas been configured to ignore all client side scripting (see section 2.6.2). Note that, to the extent that capture of a site is significantly limited by failure to process client side scripting, then this already indicates violation of the WCAG guidelines. For the purposes of the study here, it would be preferable if this behaviour could be more explicitly, and automatically, detected (as opposed to merely being implicit in a small site capture profile, and thus potentially confounded with other effects).
- A site may rely on other non-HTML client side
mechanisms, such as
flashanimated splash screens etc., for initial navigation from the home page. Exactly similar considerations then apply as in the case of client side scripting.
- Redirection failures. It is possible for a web server
to indicate that the client should "redirect" a given
request to a different URL. In normal circumstances, such
redirections will be automatically followed; however, in
certain cases they can cause problems. In particular:
- The redirection may rely on client side scripting.
This is, of course, a more specialised case of the
previous issue; however, it deserves special mention
as the use of client side scripting simply to achieve
redirection is usually gratuitous and indicates poor
server side design. In particular, the HTTP protocol
intrinsically provides mechanisms to implement
redirection in a way which raises no client side
accessibility issues; and indeed, the latter form of
redirection is normally processed perfectly correctly
- Five cases were reported by
pavukas involving circular redirections: i.e., where a redirected URL encounters one or more further redirections which point back to an earlier URL in the redirection chain. Such failures indicate definite server site configuration errors.
- If the redirect is to a URL which lies outside the target site (according to the pragmatic criteria of section 2.4), then it will, of course, not be followed. However, if this happens from the root URL for the site then, in effect, the site will not be sampled at all. This may indicate that the root URL has actually been superseded, and should be replaced in our original target list.
- A redirect may be used to translate a certain
"document" URL into a (bare) "folder" URL instead
(essentially by adding a
/character at the end). This can trigger a subtle problem with
pavukas it may already have created a local file to capture the expected document into; the existence of this file then effectively blocks the use of the same local name as a folder instead. This might be argued to be a bug (or at least, a deficiency) in
pavuk. It could possibly be overcome by some reconfiguration of
pavuk. On the other hand, it will only cause a problem if the site itself contains embedded URLs which have to be redirected in this way: which would generally indicate poor design (or a bug) at the server side anyway.
- The redirection may rely on client side scripting. This is, of course, a more specialised case of the previous issue; however, it deserves special mention as the use of client side scripting simply to achieve redirection is usually gratuitous and indicates poor server side design. In particular, the HTTP protocol intrinsically provides mechanisms to implement redirection in a way which raises no client side accessibility issues; and indeed, the latter form of redirection is normally processed perfectly correctly by
- Finally, for a small number of pages,
pavukrecorded no explicit failure, but the captured page size was nonetheless zero. This may again reflect transient communication problems--but may also suggest some malfunction in
pavuk. Such cases might bear more detailed examination.
In any case, these capture difficulties are only incidentally related to the direct objective of this study, namely assessing accessibility. Accordingly, the 28 sites, for which capture failed either completely or substantially, are simply eliminated from further consideration: the following analysis is then restricted to the remaining 186 sites, for which at least some significant sample of the site was successfully captured.
On this basis, the average number of pages captured per site was 17, and the average data captured per site was 169 KBytes. However, note that these are now primarily determined by the site sampling strategy: specifically the maximum hyperlink depth (3) and the maximum transfer quota (200 KBytes per site); see section 2.5. They should not be interpreted as indications of average site sizes in any absolute or objective sense.
The analysis phase involves executing
over the set of data files captured for each site. For each
bobby generates an XML formatted report
file which identifies for each file which (if any) of the
bobby Full diagnostics are triggered
on that page, and (if appropriate) at exactly what point(s)
in the page. This XML file is automatically parsed, and the
core data is inserted into the database. This is expressed
in terms of
bobby detectable accessibility
defects. In particular, for each page processed, the
database will contain a count of how many instances of each
such defect were detected for that page. Subsequent database
processing then allows summarising per defect type, per site
and/or across the whole set of sites.
bobby did not function properly for
a number of sites. Discounting the 28 sites for which the
capture phase had failed, an independent
failure was encountered on a further 27 sites. A complete
listing of these is given in Appendix G. These failures were
manifested in two different ways:
- For 12 sites the
bobbyclprogramme terminated with an abnormal status code. The reason was this failure could be identified in only a subset of these, where the problem was triggered by invalid HTML coding on the server--in particular the use of invalid characters, such as spaces, in URLs. In the other cases the failures may also be related to invalid HTML, but it is not possible to be definitive based on the limited failure information emitted by
bobbycl; it is also possible that some of these actually indicate defects in
- For the other 15 sites, the
bobbyclprogramme appeared to "lock up": that is, execution was continuing for a much longer period that normal and appeared as if it would continue indefinitely. Again, these failures may be symptomatic of either invalid HTML or defects in
To pragmatically deal with the cases where
bobbycl appears to lock up, the controlling
script is programmed to forcibly abort the
bobbycl process if it does not complete within
a specified time. The appropriate timeout value generally
depends on the typical capture size of each site, the
processing speed of the platform, and the background
platform load. Thus it can only be estimated pragmatically
and experimentally. For the particular conditions of the
current run, in those cases where
terminated normally, the maximum processing time
per site was 37s (the average was just 7s); a reasonable
value for the timeout was therefore judged to be 120s (2
In any case, for the purposes of further analysis here,
it is necessary to eliminate all sites where
bobby processing failed for whatever reason;
that now leaves a total of 159 admissible sites. This is
still 74% of the original target set, which is judged to be
sufficient to yield informative data on accessibility
measures. These are listed in Appendix H.
Across these sites, the total data processed by
bobbycl was 27,892,509 bytes (27 MByte),
comprising 2620 individual pages, in a time of 18:59
(1137s); thus the average processing rate was approximately
24 KBytes/s. This might be used as a very rough basis
bobby processing times on scaled
up studies (on a similar processing platform, or adjusted to
reflect relative platform processing benchmarks). However,
there was very significant variability in processing rate
across different sites; and, of course, such estimates will
be significantly confounded anyway by the unpredictable
incidence of cases where
bobbycl locks up.
Thus, it is likely that processing demands for larger scale
studies can only be effectively determined through empirical
Storage (disk space) requirements are largely determined by the decisions on the number of target sites and the data capture quota per site. These yielded the basic storage usage of just over 30MBytes. However, this has to be augmented by space for:
- the various log and report files generated by
- the normal storage overhead imposed by the allocation block size in the file system;
- the separate storage requirements of the database system.
In our case, the primary data is archived (uncompressed) on CD-ROM; in that form it requires c. 60MBytes. On the live disk (with a larger allocation block size), during capture and processing, the requirement was for c. 90MBytes. The live database storage requirement is c. 5 MBytes; a compressed database dump requires less than 200 KBytes.
This suggests that the overall scale of the survey could quite easily be increased by a factor of, say, 10, without posing very significant raw storage demands (e.g., the data could still be easily archived within the capacity of a single CD-ROM). Of course, scaling also depends on processing and communications requirements which are more difficult to estimate, as already discussed.
This section presents the core benchmark results from the study: a set of automated indicators of accessibility across the Irish web space.
bobby accessibility defect data,
generated as described in section 3.2, was analysed as
- For each of the 25 distinct
bobbyFull diagnostic tests, a total count of such defects was calculated for each site.
- By using the mapping of
bobbydiagnostics to WCAG checkpoints (Appendix B), a total defect count for each applicable WCAG checkpoint was calculated for each site.
- Similarly, by using the mapping of WCAG checkpoints to WCAG priorities (W3C, 1999), a total defect count for each priority was calculated for each site.
- Each site was classified for (non-)conformance level. That is, a site with one or more priority 1 defects fails level A conformance; otherwise if it has one or more priority 2 defects it fails level AA conformance; and finally, if it has one or more priority 2 defects it fails level AAA conformance. Obviously, failure at level A implies failure at AA and AAA.
- The key benchmark measures are then calculated: namely the proportions of sites which can be definitively classed as failing to achieve WCAG conformance levels A, AA and AAA.
The key benchmark conformance measures are:
|WCAG-A Conformance Failure Rate:||149/159||93.7%|
|WCAG-AA Conformance Failure Rate:||159/159||100.0%|
|WCAG-AAA Conformance Failure Rate:||159/159||100.0%|
A number of observations may be made on these results:
- It is worth emphasising again, as noted in section 2.1 that it is intrinsic to the methodology of this study that only negative results, in the form demonstrated above, can be derived. That is, while this study reliably establishes that at least 149 (93.7%) of the (eligible) sample sites do not achieve WCAG-A conformance, it does not follow that the remaining 10 sites (6.3%) do conform: detailed human evaluation of each site would be required to evaluate this.
- Recall that WCAG-A represents a minimum of best practice to achieve accessibility: "Otherwise, one or more groups will find it impossible to access information..." (W3C, 1999, Section 4). Even a crudely indicated non-conformance rate of (at least) 93.7% to this minimum standard must be a matter of significant concern.
- The interpretation of these results is at least somewhat confounded by the subjective elements in the sampling process (sections 2.3, 2.4 and 2.5), and the further erosion of the sample due to capture and analysis failures (sections 3.1 and 3.2). Both repeated and larger scale studies would be necessary to probe these limitations.
- It should still be recognised that WCAG Guidelines are simply that--guidelines. An extensive technical failure to conform to these guidelines, as indicated in this study, does suggest that many users with disabilities will experience avoidable difficulties in accessing Irish web sites; but it does not follow that such sites would be completely or even largely unusable. Rather there will be a continuum of degree of difficulty or obstruction that will vary among sites and among users. These crude measures should certainly not be taken as a discouragement to individual users with disabilities to go online--particularly if they can be directed to individually usable and valuable web sites or services (Irish or otherwise).
- For the purposes of the benchmarks shown above, all
bobbyFull defects, at a given WCAG priority level, are ranked equally. However, in practice, these defect types may have significantly different implications for accessibility (varying further according to the purposes of the site and the particular requirements of each user). The specific defect types reported are therefore examined more closely in section 5.
In this section we review and discuss the specific accessibility defects which the study has identified. We focus on those with the highest WCAG priority and affecting the highest proportions of sites across the survey. The objective is to try to identify repair or remediation activities which, if undertaken by site operators, would effectively reduce accessibility defect levels on the Irish web.
However, again, the methodological limitations of purely automated accessibility testing, especially when aggregated across a wide variety of sites, must be remembered here. While the results discussed in this section should provide helpful generic context for web operators, effective remediation would require much more careful and detailed evaluation on an individual site basis. In particular, for any given site, there may well be significant accessibility problems which are not detectable at all with the techniques employed here, and therefore not mentioned (at any ranking) in this section; but which have a much more immediate and practical effect for many users with disabilities, and should therefore be a high priority for repair.
The following table summarises all WCAG Priority 1 defects, across all sites, identified in this survey, ranked by the proportion of sites affected (i.e., the proportion of sites for which the defect was detected at least once):
|Diagnostic ID||WCAG Checkpoint||Incidence (sites %)|
It is notable that 5 out of the 7 defect types here all refer to a single WCAG Checkpoint, 1.1: "Provide a text equivalent for every non-text element" (W3C, 1999). Given the dominant impact of this category of defect on the overall results of the survey, it is worth discussing this is a little more depth. The motivation for its use is elaborated as follows (ibid.):
The power of text equivalents lies in their capacity to be rendered in ways that are accessible to people from various disability groups using a variety of technologies. Text can be readily output to speech synthesizers and braille displays, and can be presented visually (in a variety of sizes) on computer displays and paper. Synthesized speech is critical for individuals who are blind and for many people with the reading difficulties that often accompany cognitive disabilities, learning disabilities, and deafness. Braille is essential for individuals who are both deaf and blind, as well as many individuals whose only sensory disability is blindness. Text displayed visually benefits users who are deaf as well as the majority of Web users.
Text equivalents thus provide a generic mechanism for addressing a wide variety of both web functions and user disabilities. The general notion of a text equivalent was already mentioned in section 1, in the context of providing alternatives to embedded images; but the scope of this checkpoint is significantly broader than that. Text equivalents can be an effective accessibility technique for all of the following web design situations:
- Images (including those used as list bullets, spacers, graphical buttons etc.)
- Image map regions
- Graphical representations of text (including symbols)
- Animations (e.g., animated GIFs)
- Scripts and Applets
- Audio and Video content etc.
Each of the
bobby priority 1 defect
types identified in table 5.1 are considered in more detail in
g9: Provide alternative text for all images.
Given that this one defect is encountered one or more times on over 90% of sites, an improvement in this could have a very direct effect on the key WCAG-A Conformance Failure benchmark.
A basic requirement for addressing this issue is, of course, the use of web publishing tools that technically support the insertion of text alternatives. Thus, an obvious recommendation is that site operators should ensure that all their approved tools have this capability; and--equally, or even more importantly--that all relevant developers or content authors have the technical knowledge to access it.
However, it is essential that this be coupled with training in the appropriate formulation of text alternatives. While this is not generally complex, neither is it trivial or intuitive: it requires essential knowledge and understanding of the diverse purposes of equivalent texts, and the diverse needs and capabilities of users with disabilities.
Having said that, in some cases, the formulation of alternative text for images is straightforward. In particular, many images embedded in web pages are purely to provide visual decoration. In such cases the appropriate text alternative is an (explicitly) empty string, coded as:
<img alt="" src=... >
Although it may seem counter-intuitive, this is quite distinct from simply omitting alternative text completely. It makes explicit to the browser that there is no significance to the image; therefore a user who can't (or chooses not to) perceive the image need not even be alerted to its presence. By contrast, if the alternative text is omitted, as opposed to being blank, a browser must generally try to signal to the user that some sort of image is present--just in case it may be significant. In the case of a blind user using an auditory browser for example, this would mean that the browser would insert some sort of auditory indication of the presence of each such image, which may interfere quite significantly, and completely unnecessarily, with the user's effectiveness (not to say enjoyment!).
Of course, apart from this special case, text alternatives for images will usually have to be generated on a case by case basis9; but provided that appropriate tools are in place, and that developers and authors are properly qualified, the universal insertion of effective alternative text imposes essentially negligible overhead or additional effort on site operators. Given the very significant accessibility benefits which it can offer, operators should be strongly encouraged to take these steps.
g39: Give each frame a title.
Frames are a legacy HTML technology allowing for more or less arbitrary sets of distinct web resources to be dynamically combined for display purposes at the client side--i.e., by the user's browser. Historically, the frame concept was an ad hoc invention which was very poorly matched to the underlying technical principles of the web (Engelfriet, 1997b). Frames continue to give rise to a wide variety of problems both for general usability and for accessibility by users with disabilities in particular. The functionality offered by frames can now be achieved by alternative technologies that have been properly engineered into HTML, and are well supported in browsers. Accordingly, frames should be deprecated in the design of new sites; and should preferably be phased out of existing designs. Similarly, web publishing tools should be configured to avoid the use of frames; or if this is not supported, alternative tools should be identified. These recommendations can be made on general engineering and usability grounds, quite aside from accessibility considerations.
However, as long as frames are still in use, then it is essential that the special accessibility issues which they raise are adequately addressed (Engelfriet, 1997a). In this regard, it is disappointing not so much that at least 34% of the surveyed sites are (still) using frames, but that they are doing so in such a way as to erect additional barriers for certain disability groups.
HTML frames technology was conceived as a primarily visual mechanism for merging distinct web resources in a single browser window. As such, the use of frames can cause particular problems for users with a variety of visual disabilities. In general, such users may require explicit help with orientation to the frame-based organisation: i.e., they need to be able to access descriptive information about what frames are present, what their intended purposes are, and to access their individual content.
To facilitate such users, each HTML
element should have an associated, textual,
title attribute; this serves to provide
suitable orientating information about the frame content. As
will be discussed further in section 5.1.3, care is required in formulating this
text to ensure that it is not appropriate only to the
initial content of a frame but rather describes the
general role or function of the frame. In this way it should
remain appropriate even as the specific frame content is
In any case, if a
frame element has no
title attribute at all--which is what is
g39--then a user is necessarily denied what may
be critical orientation information. As a result, this
represents a potentially major and pervasive obstacle to
access. However: the titling of frames is generally
controlled in a small number of locations on the server
(perhaps just a single page), and thus the repair effort
required to correct it may be very small.
Of course, the phasing out of the use of frames technology will also ultimately be an effective remedy.
g38: Each FRAME must reference an HTML file.
bobby defect type
refers to accessibility of HTML frames technology, but to a
very precise technical aspect of frame usage.
frame element specifies, inter
alia, a corresponding web resource to be initially rendered
by the browser. However: frames are a technology to support
so called "dynamic content" in the browser. Specifically, it
is possible for user interaction via one frame to
dynamically alter the content of another frame. This sort of
dynamic content poses special accessibility issues, codified
in WCAG Checkpoint 6.2 (W3C, 1999):
Ensure that equivalents for dynamic content are updated when the dynamic content changes.
In the current case this means that if the content of a frame requires accessibility support--such as a text equivalent--then that equivalent must, of course, change appropriately (and automatically) whenever the frame content changes. Otherwise a user relying on such an equivalent is likely to become hopelessly disoriented whenever frame content changes.
The problem is that insofar as the
element itself (which is effectively just a screen oriented
"place-holder") specifies an equivalent, this can only
directly relate to the initial content. Thus, in
general, frame content must (dynamically) define its own
equivalents, rather than rely on the
element to do so. But this is only possible if the content
is of a media type that can contain embedded or linked
accessibility equivalents. Images formats, for example,
cannot generally do this. So if an image resource is
directly specified as frame content, then it will generally
be impossible to provide a correct equivalent if the
contained content is dynamically altered.
A general (though not comprehensive) way to address this
is to require that the content specified in a
frame element must have the HTML media type.
bobby reports a
this is therefore supposed to indicate a
element which is specifying a non-HTML resource as (initial)
However: in the detailed analysis during the current
study, some concerns have been identified about the
reliability of this particular
Firstly, it is not clear in principle how
bobby might reliably implement it. The problem
is that, in general, media type cannot be established simply
by examining a URL; rather it requires a
protocol transaction with a web server.10 But when
bobby is only
examining locally stored files (as in the current study)
there are no
http protocol transactions to
exploit for this purpose.
bobby might attempt
to examine the content of the target file to
determine whether it is HTML format; but in the general
case, the target file may not even have been captured
locally, and so would not be available for checking.
Secondly, manual spot checking of a number of cases where
bobby had flagged
could not verify them: i.e., inspection of the
frame elements at the code locations identified
bobby found only HTML format content
Given this uncertainly, it might legitimately be argued that this defect type should be removed from the data reported here completely--particularly in the calculation of the key accessibility benchmarks in section 4. Against that, of course, the status of this diagnostic is currently only uncertain: while some apparent "false positives" have been identified, this does not mean that all reported instances of this defect are mistaken.
Nonetheless, for completeness, the key benchmarks have
been recalculated on the basis of excluding defect type
g38; the only resultant change is that the
number of sites classified as definitely failing WCAG-A
conformance would fall from 149 to 147; i.e., the WCAG-A
minimum failure rate would change from 93.7% to 92.5%. The
WCAG-AA and WCAG-AAA failure rates remain at 100%. Thus,
such a change would not significantly alter any of the
overall conclusions of the study.
Further detailed investigation will be necessary to
definitively test the reliability of
g38. The fact that
bobby is not open source software (in contrast to the other
packages used here--
postgresql) complicates this, as the
implementation cannot be examined directly in the
bobby source code. In any case, there is little
purpose in reviewing the (potential) impact of this defect
type further at this point.
g240: Provide alternative text for all image map hot-spots (AREAs).
From the visual web user's point of view, image maps are images in which different regions are, in effect, hyperlinked. These image regions can then be "clicked on" in the normal way to follow the relevant link. When well designed, such image maps can often provide an intuitive, user-friendly, and visually appealing way of presenting certain collections of links. A very typical use is for graphical site navigation bars and menus.
However, because of the nature of the user interactions involved with image maps they can cause substantial barriers for users with a wide variety of disabilities. Obviously, a person who is blind cannot see the image at all; but even a person with moderate visual handicap may be unable to adequately distinguish the distinct regions of the image. A person with a motor disability may be unable to manipulate a mouse at all; or someone with poor sensory-motor co-ordination may find it difficult to position a visual pointer with adequate accuracy.
Because image maps may play an essential role in site
navigation, obstacles to their use may drastically limit
accessibility for users with disabilities to a significantly
greater extent than "bare" images. Thus, although the raw
site frequency of this defect is substantially smaller than
g9 this may not reflect their relative
significance. As usual, only detailed evaluation (and,
preferably, user testing) could clarify this for individual
The techniques for rendering image maps accessible are again not particularly complex. In essence they require that the links associated with an image map also have individual text equivalents associated with them: then the user's browser can be configured to convey these, and allow selection between them, in whatever way is appropriate to that individual's capabilities and preferred modes of interaction. And again, provided that suitable web development tools are used, and developers are appropriately qualified, the effort or overhead involved in doing this will be negligible.
Indeed, in the specific case of site navigation bars, these are typically only coded in a small number of page templates on the server side and then automatically incorporated into all relevant pages. In such cases, a single repair, taking perhaps an hour or less for a qualified developer, could bring about a dramatic and pervasive improvement in accessibility across an entire site.
g10: Provide alternative text for all image-type buttons in forms.
HTML forms provide for web users to enter information into a web page which can then be transmitted to the web server. Such forms are widely used in web sites, for functions such as searching the site, providing feedback, setting preferences, voting in "polls" etc. For most e-commerce sites, the use of forms is essential--to support user registration, product ordering, entering payment instructions etc.
HTML forms can contain a variety of components, such as menus, checkboxes, text entry fields etc. Virtually all forms will have one or more "buttons" which can be activated by the user to carry out some action on the form (most commonly, to "submit" the form--i.e., transmit the form data to the server for processing). A web site designer may design the HTML code for the form so as to specify that such buttons be visually displayed as images. This allows a wide variety of graphical effects, which in turn can significantly enhance the usability of the form--for those users who can perceive such effects. However, for users who cannot, or choose not to, perceive images, then these effects will be lost. In such cases, in the absence of text equivalents for the button images, the users may be left with no indication whatever as to the intended function of the buttons. This will typically render the form difficult, if not impossible, to use.
As with image maps, a lack of equivalent text for image-type buttons in forms may thus erect severe accessibility barriers. Again, the remediation requirement is typically modest. In e-commerce sites in particular, forms are generally dynamically generated by the web application software. Relatively simple and localised modifications to this software may thus provide textual equivalents for image-type buttons across the entire site.
g21: Provide alternative text for each APPLET.
"Applets" are programmes to be run in the context of a web browser. That is, a web page can specify, as part of its content, a programme that should executed on the client or user's Internet access device (i.e., their computer, or TV Internet box etc.). Applets thus allow a wide variety of effects to be achieved, that are not possible just by using "plain" HTML. These might include, for example, displaying movies or animations, or providing user controlled simulations (video games, educational tools etc.). The applet mechanism is very flexible, open-ended, and powerful.
However, precisely because the mechanism is so powerful,
it can also raise a wide variety of accessibility issues. It
is difficult to be precise here: applets would typically
need to be evaluated individually for their accessibility
implications. However, the WCAG guidelines do stipulate a
generic requirement that whatever functionality or
information is made available via applets should also be
available without using applets--which is to say,
via "conventional" (non-programmable) facilities of a web
browser. There are a variety of techniques to achieve this,
which may typically be used in combination; however, a
minimal requirement is that every applet should have a
textual equivalent. It is the absence of such equivalents
that is being flagged by defect type
While the notion of text equivalents for applets is conceptual similar to that for images, it is also a good deal more complex--simply because applets are, in principle, so general in their potential design and function. Thus, effective remediation of this defect will generally require careful analysis of the function of each applet.
g20: Provide alternative content for each OBJECT.
Conceptually, this defect type is very similar to
g21, already discussed. The
element is essentially a more recent addition to the
specification of HTML, which is even more general purpose
applet--and, indeed, is now recommended by
W3C as a replacement for
The low reported incidence of
g20 (in fact,
just a single site in the current survey) is probably thus
primarily a reflection of limited use of the
object element in general, rather than an
object is being used more
accessibly than is
In any case, where the
is used, there is, again, a minimum requirement to
provide an equivalent which will be accessible to users who,
for whatever reason, cannot access the specified
object type. As with
remediation (i.e., design of appropriate and effective
equivalents) will normally require case-by-case
The following table summarises the frequencies of all WCAG Priority 2 defects, across all sites, identified in this survey, ranked by the proportion of sites affected (i.e., the proportion of sites for which the defect was detected at least once)::
|Diagnostic ID||WCAG Checkpoint||Incidence (sites %)|
Note again that, in the WCAG scheme, while priority 2 defects are not as significant as priority 1, nonetheless they still potentially represent significant barriers to web access for a variety of users with disabilities. Site operators engaged in remediation should certainly concentrate on priority 1 defects in the first instance; but once these are corrected (i.e., WCAG-A conformance is achieved) they should then go on to seriously address all priority 2 defects. Design of new sites, and maintenance of existing sites, should aim at WCAG-AA conformance (at least)--i.e., zero priority 1 or 2 defects.
Thus, all the defect types in the table above should be considered as significant. Nonetheless, it is clear that these currently occur at significantly different levels. Accordingly, the more detailed discussion below will focus on just the top five in this ranking--even the lowest of which was still detected on almost 70% of sites surveyed.
g104: Use relative sizing and positioning (% values) rather than absolute (pixels).
The visual position and size of various elements can be specified in HTML pages--for example, font size for text, widths of tables, or individual table cells, etc. In general, HTML allows such positions and sizes to be specified in either "relative" or "absolute" units. Relative units mean that the numbers specified in the HTML should be scaled according to some norm which the user's browser already has; absolute units mean that the numbers are not scalable. The effect of using relative units is that the browser can very flexibly adjust the visual presentation according to the available visual space on the user's device, and the user's preferences and capabilities. For example, it can be ensured that even if the device, or viewing window, is relatively small, or if the user needs to use comparatively large text (for example, due to visual disability), then the text can still "flow" (so that horizontal scrolling is not required).
This defect type is significant for a number of reasons:
- The frequency at which it was detected in this survey is particularly large--indeed, it had the highest incidence (affected the greatest proportion of sites) of any single defect type.
- This defect type is particularly relevant to users with intermediate visual disability: they are not blind, but require some facilitation, particularly the use of larger font sizes. This is already a large category of disability; furthermore, as its incidence is significantly age related, and the relative population of seniors is growing, its importance is, if anything, increasing (European Commission, 2001).
- This defect type illustrates a very general concept in accessibility work: that of universal design. This is the fact that designing to include the widest possible variety of users often results in products and services that are significantly more usable for all users, whether disabled or not. In the current case, by designing a web site that uses only relative positioning and sizes, it can automatically adapt to changing user technologies and needs--such as the growing use of Internet enabled TVs, Personal Digital Assistants (PDAs) etc, which have a much wider variety of visual display capabilities than standard computer terminals. In particular, it is still, unfortunately, common to see web sites with messages such as: "This site best viewed with 800x600 screen resolution." Quite aside from the fact that many perfectly capable users will have no idea what "screen resolution" means, it surely defies common sense to actually turn away potential users (or customers, in the e-commerce case) simply because they choose not to use a particular, restricted, form of access device.
Because of the potentially pervasive effect of this defect type, remediation may appear difficult. However, in many cases, the problem arises simply because the particular web authoring tool in use is either not configured to use relative sizes and units, or (worse) is not capable of doing so. In the former case, reconfiguration and republishing of content can quickly be accomplished; in the latter case, of course, an alternative publishing tool should be sourced, but subsequently the repair effort can be small. Once appropriate tools, properly configured, are standardised upon, then this issue should not recur.
The more difficult case is where absolute positions and sizes have been "hand-coded" widely across a site to achieve particular visual effects. Typically this arises where web design has been contracted out. Remediation may then require some significant redesign effort; but, in such cases, organisations may well also want to reconsider their choice of contractor.
g271: Use a public text identifier in a DOCTYPE statement.
Properly formatted HTML pages must conform to a set of strict technical specifications. Conformance to such specifications is quite generally important to ensuring compatibility between web sites and web browsers (Zeldman, 2001); but it is absolutely crucial to ensuring compatibility with the wide variety of special purpose web browsers and assistive technologies that are necessary to address the diverse needs of users with disabilities.
Unfortunately, many web sites continue to deliver HTML pages which do not conform to these technical specifications. This situation has been able to develop historically because a very small number of browsers have accounted for the vast majority of users. These browsers have been able to process HTML which is "broken" in a variety of ways (relative to technical standards) and still render such pages in some more or less usable way. For many site operators, as long as "most" users appeared to be able to access the site, they did not concern themselves with whether it was operating in conformance with technical standards (or, in many cases, may not even have been properly aware of the technical issues).
But this is a very unsatisfactory, and indeed, unsustainable, situation. The diversity of browser technologies in active use is now growing significantly. Firstly, there is the continuing evolution of browser software: successive versions of even the "same" browser can differ significantly in their handling of various kinds of HTML "bugs"; but users are becoming progressively reluctant to continuously upgrade their browser software. So among the user population, there is a growing diversity of legacy browser versions in active use. Secondly, there is the diversification of access devices--Internet TVs, games consoles, PDAs etc. This is an unstable situation because this growing diversity makes it progressively more difficult for faulty HTML to function consistently across user populations; the only effective long term solution is to enforce strict technical standards on the server side.
Furthermore, the need to try to deal with non-conforming HTML has resulted in very significant software overhead in recent browser designs: on the one hand this has meant that "mainstream" browser software has tended to become progressively more bloated, and thus require significantly more powerful computers to run on; on the other hand, it has meant that browsers targeted at more lightweight devices (PDAs, etc.) simply cannot be designed to accommodate such technical deficiencies on web servers.
So this is another case where universal design will benefit all users, regardless of ability: but the benefit to users with disabilities will be disproportionate because of their very reliance on diverse specialist technologies.
bobby is not technically an HTML
validator--that is, a general purpose tool to
validate HTML code conformance to technical standards.
Future studies may complement
with the use of such a validator. However, for the purposes
of the current study,
bobby does give some
basic indications of HTML standards conformance. In
particular, a properly conforming HTML page minimally
contains a so-called DOCTYPE statement which
identifies which particular HTML standard the page
satisfies. If this is missing, then it is not even possible
in principle to validate the page for standards conformance;
bobby raises defect code
On this basis, the current study shows that 89.9% of
sites examined have at least some non-conformant HTML.
Furthermore, given that this
is only a preliminary test of HTML standards
conformance, the statistic above must be interpreted as a
minimum level: actual levels would require more
research to establish, but may well be significantly
g265: Do not use the same link phrase more than once when the links point to different URLs.
A "link phrase" is the (usually short) fragment of text in a web page that is hypertext linked to another web resource. For users of visual browsers link phrases are normally visually highlighted in some way (perhaps by colour, underlining etc.); and "clicking" within the link phrase causes the browser to load the linked resource. Such users can generally scan web pages visually very quickly to pick out such link phrases; and can easily read the surrounding text if they need more context to understand a particular link phrase.
For users of non-visual browsers (say using computer synthesised speech, or a braille output device) "scanning" a web page is generally a slower and more cumbersome process. One common technique to aid scanning in such cases is to simply skip from link to link; in this circumstance, only the link phrases are directly rendered to the user, and access to surrounding text (for additional context) will be relatively slow (i.e., it will undermine the very utility of this form of scanning).
This being the case, access for such users can be significantly improved if a little care is taken in the selection of link phrases; and, conversely, poor selection of link phrases can create a significant, and generally quite unnecessary, obstacle to users. More specifically, if the same link phrase is used multiple times, in the same page, but linking to different resources, this will be completely hidden from a user who is scanning only such link phrases. The most hackneyed form of this is a repetition of stock phrases like "click here", or "more", which are meaningless in isolation.
In some cases, repetitive link phrases are generated by a particular authoring tool, or perhaps by customised, site specific, software. Then, the usual advice of reconfiguring or replacing the tool, or redesigning the relevant software would apply. However, for link texts which are manually generally by individual content authors, the most important step is education and training. In most situations, once the author understands the potential variety of usage situations for link phrases, more accessible authoring then comes quite naturally and does not require any significant extra effort or resource; although, of course, some dedicated effort may be required to repair legacy content.
g41: Explicitly associate form controls and their labels with the LABEL element.
In the visual presentation of a web page there can often be important relationships between different components of the page which are expressed only implicitly by their juxtaposition in the display. A common example arises in the case of HTML based forms. As already mentioned in section 5.1.5, HTML forms are web pages where the user can fill in or select responses, and then select a "submit button" to send this information to the web server. A form thus generally consists of information explaining to the user what has to be filled in, interspersed with "form controls"--text entry boxes, radio buttons, drop down lists etc.--which the user can interact with. Typically, the relative positions in the visual display make it reasonably easy for a visual user to identify which text is associated with which control.
However: for users who are unable to use a visual display effectively (blind, partially sighted etc.) it is not possible to directly perceive these implicit, but critical, relationships. To address this, HTML provides facilities whereby a particular form control can be explicitly marked as associated with a particular text (the corresponding "label"). This coding can then be used by a suitably configured browser to help a user with a disability to recognise the correct relationships. Furthermore, coding these explicit relationships can improve general form usability; for example, the browser can associate clicks on a label as intending to activate a form control, thus providing a larger target for selection with a pointer. This may be particularly helpful to users with motor impairment which limits fine pointer manipulation; but will generally be of benefit to all users. Thus, this again illustrates the applicability of universal design.
As with a number of the defects already discussed, remediation will depend on the authoring or development tools in use; but if a given tool does not support making forms accessible in this way, then an alternative tool should be sought. And again, effective accessible design will generally require that developers and authors have received appropriate training.
g269: Make sure event handlers do not require use of a mouse.
Various HTML coding techniques (normally involving the use of client side scripting) rely on certain kinds of interaction with the user. However, depending on their individual capabilities and preferences, users may adopt a wide variety of interaction devices. In particular, the use of a conventional mouse, or even of some adapted form of screen pointing device, may be difficult or impossible for some users. Thus, if a page is coded in such a way that certain functionality or features can be accessed only by using a particular form of interface device--such as a mouse in the current case--then that functionality will be unavailable to many users with disabilities; worse still, such users may not even be aware that such functionalities exist.
There are a variety of well established techniques for "universal design" of client side interactions (including accessible client side scripting techniques, complemented with appropriate alternatives for users unable or unwilling to process client side scripting). The scale of the remediation effort will depend on the particular site, and the particular authoring and development tools in use. If the problematic interactions are being automatically embedded by certain tools then, as usual, the tool should be configured to avoid this; or if that is not possible, an alternative tool should be selected. Where the problem is a result of "hand-coding" then remediation may require some significant redesign effort; but the starting point then must be with ensuring that development staff (or external contractors, as appropriate) are properly qualified: in the general case, web site development is an engineering activity that must be subject to normal standards of professional practice.
The following table summarises the frequencies of all WCAG Priority 3 defects, across all sites, identified in this survey:
|Diagnostic ID||WCAG Checkpoint||Incidence (sites %)|
Priority 3 checkpoints are described as issues which web site developers "may" wish to address, to ensure maximum accessibility for all users (W3C, 1999). Thus, one would expect that particular organisations--depending on their particular objectives and remit--may or may not choose to conform with these checkpoints.
In the context of the results presented here, it is clear that there are very significant issues to be dealt with at priority levels 1 and 2 in the first instance. Accordingly, the specific priority 3 checkpoints are not discussed further at this point.
This section provides a critical review of the results of the study, and of the tools and methodology adopted. It also considers the implications for future work.
First and foremost, the project has demonstrated that this sort of largely automated survey of selected accessibility indicators is technically feasible; and that once the appropriate tools have been developed and integrated, the technical resources to carry out such a survey are comparatively modest.
However, the project has also identified a number of limitations and/or open issues, many of which will require further research.
- A fundamental issue is the effect of limiting the study to those accessibility indicators that can be measured by purely automatic means. On the one hand, this made the surveying of a comparatively large number of sites feasible; and, indeed, will support easy repetition, or scaling up to larger studies. On the other hand, it runs a real risk of focussing remediation action solely on these automatically detectable defects, to the exclusion of action on other--potentially much more significant--accessibility barriers. This argues strongly for the large scale work described here to be complemented with smaller scale studies which focus on those WCAG checkpoints which require user testing and expert evaluation.
- A related point which bears re-iteration is that the presented benchmark measures of accessibility must be interpreted very carefully. In particular, the fact that zero priority 1 defects were detected on a (small) number of sites does not mean that these sites are, in fact, WCAG-A conformant: it means only that they would qualify for closer evaluation.
- The question of longitudinal studies has been mentioned several times. The work reported here is based on a snapshot taken at one point in time. Much of the real value of the work would be in exploiting the same machinery to automatically repeat the survey at regular intervals, and track any trends or significant changes. Ideally, this should take the form of a dynamic web based facility which would automatically make the data--both raw and processed--available to other researchers and to the wider public. Site specific reports could be automatically relayed to individual site operators. The system already constructed for this project provides both a proof-of-principle and an excellent foundation for this; but would still require significant further research and development.
- An open question is the relationship between WCAG conformance and the actual experience of real users. By its nature, that is not amenable to automated testing. However, the web itself could be exploited to facilitate this sort of study. In particular, it should be possible to establish a web based community system that would permit real users, with a variety of abilities and disabilities, to voluntarily pool accessibility experiences, in a systematic way. Ideally, this could be directly integrated with the ongoing automated survey system described in the previous point--so that automated and user submitted evaluations for given sites could be easily correlated with each other. This could also contribute to more systematic identification of sites to include in automated surveying (this issue will be returned to below).
- It has been noted that the sampling techniques used in the study are intrinsically qualitative; the sites selected are not "statistically representative", and the results cannot be extrapolated by statistical means. This is a non-trivial problem: it is not apparent that it is even possible in principle to carry out statistically representative sampling for these particular purposes. The sample of sites used does certainly span a very wide range of the Irish web: it would be surprising if the results proved to be highly biased. Nonetheless, one immediate way of probing this question would be to implement a further study with a significantly larger sample size: given that the tools are now in place, this should be a very feasible way forward.
- A quite separate issue is the strategy for delimiting and sampling individual sites. We conjecture that the key results presented here are not strongly sensitive to this internal site sampling strategy (above some minimal sample size: realised in the current case by requiring that more than one page be successfully retrieved from any given site). However, it would feasible and desirable to carry out a further study in which a substantially larger sample was taken (or at least attempted) of each site in order to explicitly test this conjecture.
- The site capture mechanism used in this study suffers
from significant limitations with respect to sites which
require user "registration". While
pavukitself is capable of implementing certain common authentication protocols, which would allow access to some such sites in principle, this would still rely on having (manually) created user accounts on each site for this purpose. More generally, automated evaluation of "interactive" sites is fundamentally problematic. Again, while
pavukhas the capability of being programmed to automatically submit specified form data in certain situations, the appropriate data would have to be (manually) formulated and recorded in advance for each situation. It would be helpful to research this limitation further by attempting to design effective capture tools individually customised for a small number of examplar interactive sites. This is particularly important given the growing number and diverse roles of such sites.
- In the course of the study, there were indications that some sites may be making claims of accessibility (in terms of WCAG or otherwise) that are not well grounded; in part this may be due to lack of appropriate training or qualification; in part it may reflect claims that are the result of rather subjective "self-certification"--on the part of a site operator, or, worse, a subcontractor. Given that WCAG conformance testing does require expert evaluation, a reasonable question is whether there is a need for independent auditing or certification services. Note, in particular, that such certification cannot be a once-off activity, but needs to be repeated at intervals to verify continuing conformance.
- It is troubling that, of the originally identified sample of sites, no automated assessment results at all were generated for 26% of them. Certainly there must be a suspicion that the factors which caused at least some of these assessment failures actually represent significant accessibility barriers--perhaps a good deal more significant than the barriers explicitly identified for sites that could be assessed. A more detailed and systematic review of these failures should be carried out to test this conjecture.
- The finding that the vast majority of sites surveyed
(90%) had at least some pages which do not conform to
minimal technical standards for interoperability is
striking--but not necessarily surprising. The development
of the Web, and the evolution of these very technical
standards, has been so rapid that there has been a ongoing
shortage of professionally trained and qualified
developers. Moreover, in many cases, standards violations
are not the fault of individual developers, but of the
development tools which are being used, which
have also been poorly engineered, and/or released
prematurely. However, for the long term viability of Irish
web services, for all users, it is essential that
this situation be improved. This will require, among other
things, a greater emphasis on professional development and
technical qualification. Given that
bobbyonly provides very crude information on this issue, it would also be desirable to carry out specific surveys on technical web standards compliance, that could provide much more detailed information on the kinds of defects that are present.
- The scope of this study was, deliberately, limited to the Irish web space. However, to properly inform public policy, it would be highly desirable to have comparative data for other jurisdictions; this is particularly relevant in the context of EU initiatives (European Commission, 2000,2001). While the tools developed in the current project could clearly be deployed for such comparative surveys, this would be dependent on first identifying satisfactory site samples for each relevant jurisdiction--which, as discussed, is not a simply automated process.
- Almost all the packaged software employed in this
study was available at no charge under open source licensing. This
linuxoperating system, the
postgresqldatabase system, the
perlscripting language, and the
pavukweb capture robot. The one substantial non-open source package employed was
bobby. It is notable that this package was also generally found to be the least stable. It exhibited a diversity of failures, most seriously when it would appear to lock up indefinitely. Because it is not available in open source form, it was generally not possible to investigate these failures in more detail. This suggests that there may be a need to consider the development of an open source alternative to
bobby: but such an initiative was clearly beyond the scope of this immediate project.
- A point noted in the detailed discussion of a number of specific accessibility defects is the impact of web development or authoring tools on accessibility. In some cases, resolving this may simply be a question of using or configuring the tool in an appropriate way; however, in other cases it may be that certain tools are intrinsically flawed and should be deprecated. It would be worthwhile to attempt further research on the tools apparently in use, and their accessibility characteristics; it may be possible to harvest some data for this purpose automatically.
The core objectives of this study were:
- To gather objective data, across a broad variety of Irish web sites, on the state of web accessibility to users with disabilities.
- To formulate summary benchmark indicators, based on this data.
- To develop tools which would support the effective ongoing--preferably automatic--monitoring of these indicators.
- To identify specific remediation activities that could significantly improve web accessibility in Ireland.
These objectives have been achieved, as documented in the body of the report. But of course, these objectives arise in a particular context, and with particular purposes in mind. The results presented here are intended specifically to inform policy and initiatives in the development of the Irish web.
In this sense, the primary outcome is, in effect, a "wake-up call"--for government, for public agencies, for private companies, organisations and individuals. It signals that, despite Ireland's justifiable pride in its economic and technological development, despite very laudable goals in documents such as the E-Europe Action Plan (European Commission, 2000,2001), the current commitment to accessibility of the Irish web for users with disabilities is, at best, aspirational--and, at worst, cynically inadequate.
This is doubly unfortunate. It is not just that web technology is not being applied--as it could be--to positively improved opportunities and capabilities for users with disabilities; but on the contrary, as web services become more pervasive and essential, to the extent that they remain inaccessible this will actually impose progressively more disadvantage and exclusion on groups with disabilities, in our society.
On the other hand, this study also indicates that significant progress could be made quickly, and with comparatively little effort--if decision makers recognise the significance of the issue and take action accordingly. It is certainly not an all or nothing situation: if web site developers and operators focussed on just a few of the most pressing accessibility deficiencies in each particular site, significant improvement could be readily achieved.
There is of course a need for leadership; for public policy initiatives, particularly in education and training; for concrete incentives and supports to organisations wishing to improve services for users with disabilities. Finally, there must surely also be a role for legislation and regulation to guarantee and vindicate the rights of all citizens to equal treatment in a digital democracy.
- Engelfriet, A. (1997a),
- `Using Frames and Accessible Web Sites', Web Design Group. (Accessed: 2 August 2002)
- Engelfriet, A. (1997b),
- `What's Wrong With Frames?', Web Design Group. (Accessed: 2 August 2002)
- European Commission (2000),
- `eEurope 2002: Action Plan'. (Accessed: 17 October 2002)
- European Commission (2001),
- `eEurope 2002: Accessibility of Public Web Sites and their Content: Communication from the Commission to the Council, the European Parliament, the Economic and Social Committee, and the Committee of Regions', Commission of the European Communities. Microsoft Word Format. (Accessed: 17 October 2002)
- Flavell, A. J. (2002),
- `Use of ALT Texts in IMGs'. (Accessed: 20 April 2002)
- Irish Information Society Commission (2000)
- , `IT Access for All'. (Accessed: 23 October 2002)
- Irish National Disability Authority (2002),
- `IT Accessibility Guidelines'. (Accessed: 17 October 2002)
- McMullin, B. (2002),
- `Stretching the Web', DCU University View . DCU Alumni Magazine. Also printed in AIB Futures, Vol 12, December 2001 (AIB Internal Magazine). (Accessed: 17 October 2002)
- W3C (1997),
- `World Wide Web Consortium Launches International Program Office for Web Accessibility Initiative', World Wide Web Consortium (W3C). Press Release, October 22, 1997. (Accessed: 22 June 2002)
- W3C (1999),
- `Web Content Accessibility Guidelines (WCAG)', World Wide Web Consortium (W3C). (Accessed: 20 April 2002)
- W3C (2001),
- `How People with Disabilities Use the Web', World Wide Web Consortium (W3C). (Working Draft). (Accessed: 20 April 2002)
- Zeldman, J. (2001),
- `To Hell With Bad Browsers!', A List Apart (99). (Accessed: 17 October 2002)
- Zhangxu & Aldis, J. (2001),
- `No Disability in Digitalized Community', International Center for Disability Resources on the Internet (ICDRI). (Accessed: 26 September 2002)
The work described here could not have come about without generous financial support provided by AIB PLC. I am especially indebted to John Kelly, Head of Business Banking at AIB. He demonstrated an enduring faith and commitment to the the project, and the ultimate value that it could have in setting the agenda for Ireland's emerging information society.
Detailed research and development for the project was carried out by my two research students, Esmond Walshe and Carmen Marincu.
The work was carried out in the Research Institute for Networks and Communications Engineering (RINCE), established at DCU under the HEA Programme for Research in Third Level Institutions (PRTLI).
- A. Complete listing of initial target sites
bobbyFull Support Diagnostics vs. WCAG Checkpoints
- C. Template
- D. Failed
- F. Summary
- G. Failed
- H. Summary
|Site ID||Title||Root URL|
|1||Irish Farmers' Association||
|2||Society for the Prevention of Cruelty to Animals||
|3||Zoological Society of Ireland||
|4||Arts Council of Ireland||
|5||Irish Museum of Modern Art||
|6||National Concert Hall||
|7||Royal Irish Academy of Music||
|10||Advertising Standards Authority for Ireland||
|11||Chambers of Commerce of Ireland||
|12||Consumer's Association of Ireland||
|13||Director of Consumer Affairs||
|16||Irish Business and Employers Confederation||
|17||Irish Small and Medium Enterprises||
|18||Small Firms Association||
|22||National Council for the Blind||
|23||Irish Guide Dogs for the Blind||
|24||Association for Higher Education Access and Disability||
|25||Irish Defence Forces||
|26||Central Applications Office||
|27||Dublin City University||
|28||Dun Laoghaire Institute of Art Design and Technology||
|29||Dublin Institute of Technology||
|30||Cork Institute of Technology||
|31||Higher Education Authority||
|32||Institute of Public Administration||
|33||National Adult Literacy Agency||
|34||University College Dublin||
|35||University of Dublin Trinity College||
|36||National University of Ireland Galway||
|37||National University of Ireland Cork||
|38||University of Limerick||
|39||Letterkenny Institute of Technology||
|40||Union of Students in Ireland||
|41||Association of Secondary Teachers Ireland||
|42||Royal Irish Academy||
|46||Irish National Teachers Organisation||
|47||Irish Times (portal)||
|49||Electricity Supply Board||
|53||Allied Irish Banks PLC||
|54||Bank of Ireland||
|56||EBS Building Society||
|57||National Irish Bank||
|58||Bord Iascaigh Mhara||
|59||National Library of Ireland||
|61||Department of Agriculture Food and Rural Development||
|62||Department of Arts Heritage Gaeltacht and The Islands||
|63||Office of the Attorney General||
|64||Department of Defence||
|65||Department of Education and Science||
|66||Department of Enterprise Trade and Employment||
|67||Department of Environment and Local Government||
|68||Department of Finance||
|69||Department of Foreign Affairs||
|70||Department of Health and Children||
|71||Department of Justice Equality and Law Reform||
|72||Department of Marine and Natural Resources||
|73||Department of Public Enterprise||
|74||Office of the Revenue Commissioners||
|75||Department of Social Community and Family Affairs||
|76||Department of the Taoiseach||
|77||Department of Tourism Sport and Recreation||
|78||Irish Statute Book||
|79||Automobile Association (AA)||
|80||Alzheimer Society of Ireland||
|81||Arthritis Foundation of Ireland||
|83||Central Remedial Clinic||
|84||Eastern Regional Health Authority||
|85||Health Research Board||
|86||North Western Health Board||
|88||Voluntary Health Insurance (VHI)||
|90||Simon Community of Ireland||
|91||Amnesty International Irish Section||
|92||Irish Council for Civil Liberties||
|93||Central Statistics Office||
|96||Irish Courts Service||
|97||Law Society of Ireland||
|98||The Institute of Chartered Accountants in Ireland||
|99||Cork County Council||
|100||Dublin City Council||
|101||Dun Laoghaire/Rathdown County Council||
|102||Fingal Country Council||
|103||Kerry County Council||
|107||Independent News and Media PLC||
|118||The Labour Party||
|120||Green Party of Ireland||
|123||Football Association of Ireland||
|125||Golfing Union of Ireland||
|127||Irish Sports Council||
|128||Paddy Power Bookmakers||
|140||Federation of Irish Scout Associations||
|141||Irish Youth Hostel Association||
|142||National Youth Council||
|143||National Youth Orchestra||
|144||National Centre for Technology in Education||
|145||2003 Special Olympics - World Summer Games||
|146||Research Institute in Networks and Communications Engineering (RINCE)||
|147||Science Foundation Ireland||
|161||Horizon Technology Group PLC||
|162||IAWS Group PLC||
|164||Jurys Doyle Hotels||
|165||Irish Life and Permanent PLC||
|166||Sherry Fitzgerald Group||
|167||Hamilton Osborne King||
|172||Sun Microsystems Ireland||
|175||Electronic Business Solutions Ltd.||
|176||Prestige Systems Ltd.||
|180||ContentPAL/ERS Solutions Ltd.||
|186||Club Travel Ltd.||
|187||Business Software Alliance Ireland||
|188||Buy and Sell||
|192||Irish Internet Association||
|194||IE Domain Registry Limited||
|201||Cedar Tree Ireland||
|206||The Communications Interactive Agency Ltd.||
|213||Mediahouse Internet Services||
|214||Cork Web Studio||
The table below shows the complete list of 25
bobby diagnostics, with WCAG mappings and
priorities, included in the current survey.
|ID||Description||WCAG Priority||WCAG Checkpoint|
|g9||Provide alternative text for all images.||1||1.1|
|g21||Provide alternative text for each APPLET.||1||1.1|
|g20||Provide alternative content for each OBJECT.||1||1.1|
|g10||Provide alternative text for all image-type buttons in forms.||1||1.1|
|g240||Provide alternative text for all image map hot-spots (AREAs).||1||1.1|
|g14||Client-side image map contains a link not presented elsewhere on the page.||3||1.5|
|g271||Use a public text identifier in a DOCTYPE statement.||2||3.2|
|g104||Use relative sizing and positioning (% values) rather than absolute (pixels).||2||3.4|
|g2||Nest headings properly.||2||3.5|
|g125||Identify the language of the text.||3||4.3|
|g31||Provide a summary for tables.||3||5.5|
|g38||Each FRAME must reference an HTML file.||1||6.2|
|g37||Provide a NOFRAMES section when using FRAMEs.||2||6.5|
|g4||Avoid blinking text created with the BLINK element.||2||7.2|
|g5||Avoid scrolling text created with the MARQUEE element.||2||7.3|
|g33||Do not cause a page to refresh automatically.||2||7.4|
|g254||Do not cause a page to redirect to a new URL.||2||7.5|
|g269||Make sure event handlers do not require use of a mouse.||2||9.3|
|g109||Include default, place-holding characters in edit boxes and text areas.||3||10.4|
|g35||Separate adjacent links with more than whitespace.||3||10.5|
|g39||Give each frame a title.||1||12.1|
|g41||Explicitly associate form controls and their labels with the LABEL element.||2||12.4|
|g34||Create link phrases that make sense when read out of context.||2||13.1|
|g265||Do not use the same link phrase more than once when the links point to different URLs.||2||13.1|
|g273||Include a document TITLE.||2||13.2|
No data was successfully captured for the sites identified below (sample date: 9th April 2002). Thus, these sites were discarded from further consideration.
|Site ID||Root URL|
Only one page (corresponding to the "root" or "home" URL) was successfully captured for the sites identified below (sample date: 9th April 2002). Thus, these sites were discarded from further consideration.
This shows a summary of the capture data for each site. The site capture phase took place on 9th April 2002, starting at 16:11:49 IST (UT+01) and ending at 17:59:19 IST (UT+01). In general, the content of web sites is dynamic. Thus, the data captured and analysed in this study represents only a "snapshot" of each site taken at that particular point in time.
bobby failed to analyse the following sites.
A status value of 1 indicates that
terminated abnormally (a
java exception was
reported); a status value of 3 indicates that
bobby was forcibly terminated because the
allowed time was exceeded.
|Site ID||Root URL||Status|
The table below lists the 159 sites which were
successfully captured by
pavuk, and for which
bobby successfully generated accessibility
defect reports. All the results reported are based on this
- ... Accessibility?1
- Much of the text in this section is derived from a previous short article (McMullin, 2002).
- ... Web2
- Technically, the web is just one specific service hosted on an underlying communications network, which is the Internet. However, given that the web is by far the most familiar Internet service, and often now provides the primary user interface to other services, I will generally not distinguish between them here.
- ... users.3
- However, to be properly effective, this technique does rely on the author/designer of the web site having a clear understanding of the need for, and the appropriate use of, such alternative text (Flavell, 2002).
- ... site.4
- Note that this is, in effect, 200 KBytes of text--as opposed to graphics or other content. It corresponds to roughly 70 pages of print, per site, and is therefore quite substantial.
- ... form5
- ASP: Application Service Provider. See also: http://whatis.techtarget.com/definition/0,289893,sid9_gci213801,00.html
- ... checkpoint.6
- There are an additional 3
bobbydiagnostics which do not relate to the WCAG guidelines, but only to the requirements of Section 508 of the US Rehabilitation Act; since this act does not apply in the Irish jurisdiction, these diagnostics were excluded from the current study, and will not be discussed further.
- ... 4.9 KByte/s.7
- This is slightly different from the average data transfer rate because some additional data was "transferred" but not "captured"; this can arise where a transfer is initiated, but subsequently aborted for some reason--for example, if the site transfer quota is exceeded.
- ... itself.8
- It should be noted that
bobbyis not distributed in source form; this makes further investigation in cases such as this problematic.
- ... basis9
- Flavell (2002) provides a more extended discussion and tutorial.
- ... server.10
- This issue is also discussed, in a somewhat different context, in section 2.6.2.