Guide to the Census of Population, 2016
Chapter 10 – Data quality assessmentGuide to the Census of Population, 2016
Chapter 10 – Data quality assessment

Introduction

Data quality assessment provides an evaluation of the overall quality of census data. The results of this assessment are used to inform users of the reliability of the data, to make improvements for the next census, to adjust census data for non-response and, for two coverage studies (reverse record check and the Census Overcoverage Study), to produce official population estimates. Quality assessment activities take place throughout the census process, beginning prior to data collection and ending after dissemination.

Sources of error

However well a census is designed, the data collected will inevitably contain errors. Errors can occur at virtually every stage of the census process, from material preparation to creation of the list of dwellings, data collection and processing. Census data users should be aware of the types of errors that can occur, so they can assess the usefulness of the data for their own purposes.

Main types of errors:

Coverage errors occur when dwellings and/or persons are missed, incorrectly enumerated or counted more than once.

Non-response errors occur when some or all information about individuals, households or dwellings is not provided.

Response errors occur when a question is misunderstood or a characteristic is misreported by the respondent, the census enumerator or the Census Help Line operator.

Processing errors can occur at any stage of processing. Processing errors include errors that can be made at data capture during coding operations, when written responses are converted into numerical codes, and during imputation, when valid (but not necessarily accurate) values are inserted into a record to replace missing or invalid data.

Sampling errors apply only when answers to questions are obtained from a sample. This type of error applies only to the 2016 Census long-form questionnaire.

Measuring data quality

Many data quality studies have been conducted for recent censuses to allow data users to assess the impact of errors and improve their own understanding of how errors occur. For the 2016 Census, special studies examine errors in coverage and data quality, i.e., non-response, response and processing.

Three studies are conducted to measure coverage errors:

1. Dwelling Classification Survey – One of the sources of coverage error in the census is the misclassification of dwellings on Census Day. This error can occur when an occupied dwelling is classified as unoccupied, or when an unoccupied dwelling is classified as occupied. The purpose of the Dwelling Classification Survey is to study these types of classification errors and adjust counts, if necessary. A sample of dwellings for which no census questionnaire was returned is contacted, information is collected on the occupancy status and, if occupied, on the number of usual residents.
2. This information is used to adjust the census data for dwellings, households and persons. This is done by correcting the classification errors and adjusting household size distribution through imputation for dwellings that did not return the questionnaire. It is done in time for the initial population count release.

2. Reverse Record Check – This study provides estimates of persons missed by the census (after accounting for the adjustments described in the Dwelling Classification Survey above). Estimates are developed for each province and territory and for various population subgroups (e.g., age-sex groups and marital status).

• For the provinces, this study comprises two steps:
  • Step 1: Selecting a sample of persons who should have been enumerated in the census, using sources such as the previous census, birth registrations, immigration and non-permanent residents' records, and the sample of persons missed in the Reverse Record Check from the previous census.
  • Step 2: Linking persons selected in Step 1 to the Census Response Database (CRD) to determine whether these persons were enumerated. The survey is then used to trace and interview persons who could not be linked with certainty to the CRD in order to collect additional information. Persons who have died or who emigrated prior to Census Day are identified using administrative records, such as the death register, or during tracing or the interviews.
• For the territories, Step 1 consists in linking the persons on health insurance records to the Census Response Database to identify persons who were enumerated in the census. The Reverse Record Check sample is then selected among the unmatched persons.
• The results of the Reverse Record Check are the most important source of information about persons missed in the census. However, unlike the Dwelling Classification Survey, the estimates are not used to adjust census data before the initial population count release.

3. Census Overcoverage Study – In the 2011 and 2016 censuses, double-counting of persons is determined by searching for linked records that have a high degree of matching on sex, date of birth and name. Linked records are sampled and checked manually, and results are used to estimate the census overcoverage (or the number of duplicate persons).

• When combined with the results of the Reverse Record Check, the results of the Census Overcoverage Study provide estimates of net coverage error in census data. This net error is used to calculate the official population estimates.

Certification

Certification consists of several activities to rigorously assess the quality of census data at specific levels of geography in order to ensure that the quality standards for public release are met. This evaluation includes the certification of population and dwelling counts, and variables related to dwelling and population characteristics.

During certification, response rates, invalid responses, edit failure rates, and a comparison of data before and after imputation are among the data quality measures used. Tabulations for the 2016 Census are produced and compared with corresponding data from past censuses, other surveys and administrative sources. Detailed cross-tabulations are also checked for consistency and accuracy.

Depending on the certification results, census data can be released in one of three ways:

• First, the data may be released unconditionally, meaning that the data are of suitable quality.
• Second, the data may be released conditionally or with restrictions. In this case, the data will be released with a special note alerting users to possible limitations, or the data may be specially processed, for example, by combining reporting categories to address quality or confidentiality concerns.
• Finally, the data may be suppressed for quality reasons.

For more information on the quality indicators and certification results, see the reference guides for the various domains of interest.

Response rate for the 2016 Census of Population

One of the key data quality measures used for the Census of Population is the response rate. Table 10.1 shows the response rates for the 2016 Census of Population both nationally and for each province and territory. The rates are provided for all occupied private dwellings for which a short form or long form was to be received and for the subset of occupied private dwellings for which a long form was to be received. For the long form, the unweighted response rate and the weighted response rate are provided.

The rates in Table 10.1 were calculated following data processing and data quality assessment. Response rates are calculated as follows: the number of private dwellings for which a questionnaire was filled out divided by the number of private dwellings classified as occupied according to the census database. The final classification of dwelling occupancy status is based on the data analysis collected by field staff, the data provided by respondents and the results of a quality study on the occupancy status of a sample of dwellings. The rates in Table 10.1 differ from the collection response rates previously disseminated because they take into account data processing and verification of the dwelling occupancy status and thus are considered final. With respect to weighted response rates, they are based on the long form's final sampling weights. The weighted response rates are therefore calculated as follows: the number of sampled weighted private dwellings for which a questionnaire was filled out divided by the number of weighted sampled private dwellings classified as occupied.

Quality of long-form questionnaire estimates

Estimates produced from the long-form census questionnaire are obtained by means of a sample survey. Such surveys have two types of errors: sampling error and non-sampling error. The former occurs when a characteristic is estimated by measuring only part of the population instead of the entire population. The latter includes all errors that are not related to sampling. The latter type of error also occurs in census counts, i.e., counts produced from questions found in both the short-form questionnaire and the long-form questionnaire.

Sampling error

Sampling error is the difference that would be observed between an estimate from the long-form questionnaire and the actual value for the population if there were no non-sampling errors, i.e., if there were no coverage error, response error, processing error or non-response. It is inevitable when conducting a sample survey such as that conducted using the long-form census questionnaire.

Several factors influence sampling error. The sampling error is smaller when the sampling fraction is larger and when the sample size is larger. Ultimately, if the sampling fraction is 100%, as for the short-form census questionnaire, then the sampling error will be nil. It will also be small if the variability of the variable of interest in the population is low. This error also depends on the effectiveness of the sample plan. For example, it will be smaller if the populations in the strata of the sample plan are fairly homogenous, or, in the case of a characteristic measured at the level of the person, if the individuals in the households are fairly heterogenous. Finally, sampling error depends on the estimation methods used, such as the weighting method, as some are more effective than others. For example, when the weight of a survey is adjusted so that a weighted total is equal to the census total, the sampling error for that weighted total is nil. However, it must be noted that it is impossible to adopt a weighting method that would eliminate sampling error from all possible estimates drawn from the long-form questionnaire.

Sampling error cannot be measured directly. Indeed, to do so, the actual value of the variable of interest in the population would need to be known to subtract it from the estimate drawn from the long-form questionnaire, an estimate that should not include any non-sampling errors. However, its extent can be estimated using variability measures such as standard error and the coefficient of variation.

Appendix 1.9 presents a measurement of sampling error (standard error).

Non-sampling error

Apart from sampling, several factors can lead to errors in survey results. These non-sampling errors can be of several types. Dwellings or individuals may have been incorrectly enumerated—this is coverage error. Respondents may not understand the questions and answer them incorrectly—this is response error. Responses may be entered incorrectly during data capture, or the coding of the responses may be incorrect. This is processing error.

Non-response error is also a non-sampling error. A distinction is made between partial non-response (no response to one or some questions) and total non-response (no response to the survey because the household could not be reached or refused to participate).

Non-sampling error is likely to bias estimates. Attempts were made to minimize it during each stage of collection and processing to reduce its impact. For example, as described in Chapter 9, an attempt was made to correct non-response error and coverage error by means of imputation or by adjusting weights. However, a residual error remained following this process, except when the imputation of the missing value proved to be exact.

Moreover, non-sampling error is not as easy to measure as sampling error. Nonetheless, measures of variability produced from the long-form questionnaire estimate both sampling variability and variability caused by total non-response error, under the assumption that the model used in weighting to correct this non-response is accurate. In fact, total non-response variability is measured because it may be significant in a survey with a large sampling fraction.

Comparability of estimates from the 2016 Census long-form questionnaire with estimates from the 2011 National Household Survey

Users must be careful when comparing estimates from two surveys, as they can differ significantly in methodology, quality and target population.

The estimates from the 2016 Census long-form questionnaire were derived from a mandatory survey that had a high response rate, while the estimates from the 2011 National Household Survey (NHS) were derived from a voluntary survey. The response rate for the 2016 Census long-form questionnaire was 96.9%, while the 2011 NHS had a response rate of 68.6%. The definition of the target population of the 2016 Census long-form questionnaire and that of the 2011 NHS were exactly the same.

Non-response bias occurs when a survey's non-respondents are different from its respondents. The higher a survey's non-response, the greater the risk of non-response bias. The quality of estimates can thus be affected if such a bias is present. The risk of non-response bias was taken into account for the NHS. In fact, Statistics Canada conducted several studies on the 2011 NHS, as well as various simulations, before and after collection, to assess the potential risk of bias and its extent. A number of measures were taken to mitigate its effects. Non-response error may be greater for the estimates from the 2011 NHS than for the estimates derived from the 2016 Census long-form questionnaire, particularly for smaller domains of interest.

In addition, the response rate for the 2011 NHS varies significantly from one community to another, particularly small ones. The quality of the estimates and the risk of bias can thus vary significantly between different communities. For the 2016 Census, the response rates to the long-form questionnaire vary less between communities. There is therefore less variation in the quality of the estimates, and the risk of bias is very negligible. The estimates from the 2011 NHS can contain inaccuracies because of a lower response rate than the 2016 Census. Comparisons of estimates from the 2011 NHS and from the 2016 Census long-form questionnaire for a given geographic area must take into account the differences in observed response rates.

Unanswered questions are identified in returned questionnaires. Imputation replaces missing, invalid or inconsistent elements with plausible values. When carried out properly, imputation can improve data quality by replacing non-responses with plausible responses similar to those that the respondents would have given if they had answered the questions. It also has the benefit of producing a full dataset. Imputation for partial non-response (i.e., the unanswered questions in returned questionnaires) was greater for the 2011 NHS than for the 2016 Census long-form questionnaire. These differences are greater for questions in the second half of the NHS questionnaire (about work, the workplace, the mode of transportation, languages of work and housing). The 2011 and 2016 reference guides present imputation rates for each question at the provincial, territorial and national levels. Comparisons of estimates from the 2011 NHS and from the 2016 Census long-form questionnaire must take into account the differences in imputation rates.

Table 10.2 presents national-level imputation rates for variables from the 2011 NHS and the 2016 Census long-form questionnaire. The imputation rates for questions 2 to 9 were calculated differently in 2011 and 2016. In 2011, the imputation rate excluded imputation for household non-response, while this was included in 2016. For the other questions, the calculation method for the imputation rate can differ slightly between 2011 and 2016 for some variables.

The quality assessment of the NHS estimates deemed their quality to be acceptable overall for Canada, the provinces and territories, and census metropolitan areas. The NHS estimates were comparable with those from other data sources at the same geographic levels. At a more detailed geographic level, estimates from the NHS could not be compared with those from other data sources.

Users are encouraged to use the main quality indicator provided, the global non-response rate (GNR), to judge the quality of estimates from the 2011 NHS and the 2016 Census when assessing the reliability of comparisons. The GNR is an important measure of the quality of estimates from the NHS and the long-form questionnaire. It combines household non-response and partial non-response. For the NHS and the long-form questionnaire in particular, the GNR is weighted to account for the sampling. The GNR is a potential indicator of non-response bias. For each region for which data are published, the GNR is available for both the 2011 NHS and the 2016 Census long-form questionnaire. Nationally, the GNR was 26.1% for the 2011 NHS and 5.1% for the 2016 Census long-form questionnaire (see Chapter 11 for more information). The GNR for the 2011 NHS and the GNR for the 2016 long-form questionnaire differ more for smaller geographic areas. In cases of greater discrepancy between the GNR for the 2011 NHS and the 2016 long-form questionnaire, users should take care in making comparisons. Users are also encouraged to read any quality notes that may be included with dissemination products.

Comparability of estimates from the 2016 Census long-form questionnaire with estimates from the 2006 Census long-form questionnaire

Estimates from the 2006 Census long-form questionnaire were derived from a mandatory survey. The response rate for the 2006 Census long-form questionnaire was 93.8%. The risk of non-response error was very low, and this risk is similar for estimates from the 2016 Census and 2006 Census long-form questionnaires.

The definition of the target population for the 2016 Census long-form questionnaire differs from the definition for the 2006 Census long-form questionnaire. The 2016 Census long-form questionnaire targeted the total population usually living in Canada in private dwellings, in the provinces and territories. This target population includes persons who live on Indian reserves and in other Indian settlements; permanent residents; and non-permanent residents such as refugee claimants, holders of work or study permits, and members of their families living with them.

Foreign residents are not enumerated in the 2016 Census long-form questionnaire. These include representatives of a foreign government assigned to an embassy, high commission or other diplomatic mission in Canada; members of the armed forces of another country stationed in Canada; and residents of another country who are visiting Canada temporarily.

The 2016 Census long-form questionnaire also excludes persons living in institutional collective dwellings such as hospitals, nursing homes and penitentiaries; Canadian citizens living in other countries; and full-time members of the Canadian Armed Forces stationed abroad. Finally, the 2016 Census long-form questionnaire excludes persons living in non-institutional collective dwellings such as work camps, hotels and motels, and student residences.

The 2006 Census long-form questionnaire did not target exactly the same population. Compared with the 2016 Census long-form questionnaire, the 2006 questionnaire included persons living in non institutional collective dwellings such as work camps, hotels and motels, and student residences. It also targeted foreign residents such as representatives of a foreign government assigned to an embassy, high commission or other diplomatic mission in Canada. These differences between the target populations of the 2016 and 2006 long-form questionnaires are minor and relate to only a very small percentage of the total population. Users must nonetheless take these differences into consideration when comparing estimates from 2016 and 2006.

Comparability of the variability of estimates from the 2016 Census long-form questionnaire with that of estimates from the 2011 NHS and the 2006 Census long-form questionnaire

As mentioned in the previous sections, estimates produced using data from a sample survey, such as those from the 2016 Census long-form questionnaire, include sampling error, i.e., an error stemming from the fact that only a sample of the population was observed. Sampling error is determined using variability measures such as standard error or the coefficient of variation (CV). In Appendix 1.9, standard error is used to compare the variability of estimates from the 2016 Census long-form questionnaire with that of estimates from the 2011 NHS and the 2006 Census long-form questionnaire.

Moreover, the purpose of the 2016 and 2006 Census long-form questionnaires and of the 2011 NHS was to produce estimates for a series of questions for a variety of geographic areas, ranging from very large areas (such as provinces and census metropolitan areas) to very small areas (such as neighbourhoods and municipalities), and for various population groups, such as Aboriginal peoples and immigrants. These groups also vary in size, especially when cross-classified by geographic area. Such groupings are generally referred to as "domains of interest." The purpose of this section and of Appendix 1.9 is to compare the variability of estimates from 2016, 2011 and 2006, not to compare the estimates. However, sampling variability should be taken into account when comparing estimates from these surveys, particularly for small "domains of interest," as the observed differences can be the result of sampling variability rather than an actual difference in the population.

Description of standard error

The "standard error" of an estimate is a numerical measurement of the random component of its error. Standard error can be interpreted as follows. If the sampling, collection and processing for the long-form questionnaire could be repeated many times and if an estimate for a given characteristic were calculated every time, the estimates produced in about 68% of cases would be within one standard error of the census value (i.e., the value that would have been obtained if a census had been conducted instead of a sample survey). Furthermore, the estimates produced in about 95% of cases would be within two standard errors of the census value. This means that, in general, the lower the standard error, the more accurate the estimate. The standard error is a key element in deriving other measures of variability, such as the CV; in constructing confidence intervals; and in making statistical inferences (e.g., determining whether an estimate is significantly different from a given value or another estimate). Estimates of standard error for the 2016 Census long-form questionnaire will be published in early 2018 in a supplement to the profile of aggregate dissemination areas (ADAs). They will include estimates of standard error for ADAs, census divisions, provinces and territories, and Canada.

Derivation of the coefficient of variation (CV)

The CV of an estimate is the ratio of the estimate of standard error and the estimate itself, expressed as a percentage. Like standard error, the lower the CV, the more accurate the estimate. The CV is an interesting measure of variability, as it does not depend on the estimate's unit of measure. This makes it possible to compare the accuracy of estimates that have different units of measure. However, as the CV is a ratio, it tends to have a very large value when the quantity in the denominator (i.e., the estimate of interest) is very small. Thus, care is needed when interpreting the CV of a very small proportion.

Distinction between standard error, response rate and global non-response rate

Standard error does not measure bias, such as non-response bias. It is important not to confuse standard error, the non-response rate and the global non-response rate. In fact, the non-response rate indicates the risk associated with household non-response error, and the global non-response rate indicates the risk associated with household non-response error and partial non-response. However, standard errors calculated based on the 2016 Census long-form questionnaire, the 2011 NHS and the 2006 Census long-form questionnaire include total household non-response variability, to a certain degree and in addition to sampling variability.

Conceptual and methodological differences between standard errors for the 2016 Census long-form questionnaire, the 2011 NHS and the 2006 Census long-form questionnaire

Several factors influence the values of standard errors and can explain the differences between cycles. First, the target population, sampling methods and estimation methods differ from one cycle to another. Furthermore, the variability measured is not exactly the same for all cycles: sampling variability is estimated in all cases, but household non-response variability is not measured in the same way in all cycles. In fact, in 2006, household non-response variability was not measured in Indian reserve collection units and canvasser enumeration collection units, while it was measured in 2011 and 2016.

Factors that contribute to reducing sampling variability are a larger sample, a larger sampling fraction, lower variability of the characteristic in the population being studied, a more effective sample plan and more effective estimation methods. The extent of non-response, the differences and similarities in the characteristics of respondents and non-respondents, and the estimation methods are the main factors that influence non-response variability.

The following subsections describe these factors and some conceptual and methodological differences between the sample for the 2016 Census long-form questionnaire, the 2011 NHS and the 2006 Census long-form questionnaire.

Target population

The target population for each of the three surveys is different. The 2016 Census long-form questionnaire targeted the Canadian population as of May 10, 2016. The 2011 NHS targeted the Canadian population as of May 10, 2011, while the 2006 Census long-form questionnaire targeted the Canadian population as of May 16, 2006. Furthermore, the target population in 2011 and 2016 included only persons living in private dwellings, while the target population in 2006 also included persons living in non-institutional collective dwellings (approximately 1% of the population).

Sample plan

The sample plan for the census long-form questionnaire consists of only one sampling phase. The plan for the 2011 NHS is more complex and includes two sampling phases. The sampling fractions also differ from one cycle to the next. In 2016, one in four dwellings was sampled. In 2011, an average of one in three dwellings was sampled at first. After several weeks of collection, i.e., as of July 14, 2011, the initial sample was reduced. Only the respondents were kept (approximately two-thirds of the initial sample), plus a follow-up subsample of approximately one in three dwellings drawn from among the remaining non-respondents. In 2006, the sampling fraction was one in five.

Generally, the standard error should be lower for larger sampling fractions and for larger sample sizes. It should also be lower for the simplest and most "efficient" sample plans, i.e., the one-phase sample plans in 2006 and 2016.

Extent of household non-response

Household non-response reduces the number of responses observed, which increases the variability of estimates. The final non-response rate for the 2016 Census long-form questionnaire was 3.1%. In contrast, the unweighted non-response rate for the 2011 NHS was 31.4%, and the non-response rate for the 2006 Census long-form questionnaire was 6.1%. The estimates from the 2011 NHS are therefore more affected by the extent of household non-response than those from the 2016 and 2006 long-form questionnaires.

Differences in characteristics between respondents and non-respondents

In general, the characteristics of respondents and non-respondents in the sample should be as similar as possible. In fact, if they were perfectly comparable (e.g., if responding were independent of the characteristics of interest), there would be no non-response bias. Furthermore, since there would be no non-response bias, the measurement of non-response variability would measure the total non-response error.

On the other hand, if the characteristics of respondents and non-respondents were very different, then the non-response bias would be significant. This could present a problem, as it is not considered in the measurement of variability. Ultimately, it would be possible to have a non-response variability estimate of nil, but a significant bias. The higher the non-response rate, the greater the risk of such a situation occurring.

The standard errors for the 2011 NHS could be pushed downward more than those from the 2006 and 2016 long-form questionnaires, because of the differences in the characteristics of respondents and non-respondents. In fact, given the higher household non-response rate for the 2011 NHS, the characteristics of its respondents tend to be more homogenous.

Estimation methods

One way to minimize the impact of non-response is to use estimation methods, including weighting methods, that make good use of the available information. As the extent of non-response was greater in 2011, more information was used to reduce the non-response error than in 2006. In fact, in 2006, only geographic information and household size were used to adjust for non-response, while all census variables and some administrative data were used in 2011. In 2016, more information continued to be used to correct non-response, despite the higher response rate than in 2011. The main goal of using this information is to reduce the non-response bias. However, doing so can result in increased variability of estimates, for example if the variability of weights adjusted for non-response is greater. The methods for correcting non-response in 2011 and 2016 should therefore increase the variability of the final estimates more than those of 2006.

Calibration was carried out in the last stage of weighting for each cycle to produce the estimates (counts, proportions, averages, etc.). The calibration process ensured that certain estimates of survey counts corresponded to known counts. The calibration was done using counts from the census long-form questionnaire or administrative data matched to census records. In 2006, counts were based on demographic and geographic variables. In 2011, counts were added for family and language variables. In 2016, counts were also added based on administrative data matched to census records (i.e., data on income, on immigration and from the Indian Register). The effect of calibration, apart from allowing for concordance with census counts, is to reduce the variability of estimates produced from variables related to control counts. It would thus be possible to see less variability for variables related to the calibration subjects used for a given cycle.

Moreover, although the number of calibration subjects has increased over time, the overall number of constraints used in the calibration has been reduced from one cycle to the next. This is because simulations used for developing estimation methods for 2011 and 2016 revealed that tighter calibration can lead to higher variance estimates for the variables that were least related to the calibration subjects. This could therefore explain in part the differences between the standard errors of the three cycles. Appendix 1.9 presents the measures of sampling error, as standard errors, for estimates from the 2006 Census long-form questionnaire, the 2016 Census long-form questionnaire and the 2011 NHS.

• Previous
• Next