OeNB Report 2024/14: OeNB Barometer: methodological notes 2023

Valentin Voith and Maximilian Zieser ¹

This document describes the current methodological approach of the OeNB Barometer survey, which has been redesigned to reduce different forms of bias through fully random sampling procedures and a combination of design, non-response and post-stratification weights. Thereby, it serves as a methodological reference for the survey waves of the OeNB Barometer conducted from late 2023 onward. While sample size and composition of survey waves are generally similar by design, concrete statistics reported in this document only apply to the survey wave in late 2023, where the approach was first implemented. This document may be updated for future waves if necessary.

The OeNB Barometer is conducted semiannually by the Oesterreichische Nationalbank (OeNB). Apart from a set of questions kept constant over time, the content of the survey regularly changes to focus on topics of strategic interest or on specific research questions. Interviews in the OeNB Barometer are commissioned by the OeNB and conducted by IFES (Institut für empirische Sozialforschung GmbH) using two modes, CAPI (computer assisted personal interviewing, i.e., face-to-face interviews) and CAWI (computer assisted web interviewing, i.e., online questionnaires). The OeNB Barometer aims at a total net sample of about 1,400 respondents per wave. With a target CAWI share of 10–30% in the net sample and response rates of approximately 40% for CAPI and 20% for CAWI, the OeNB Barometer typically requires a total gross sample of between 3,800 and 4,600 addresses.

Table 1 gives an overview of methodological approaches of the OeNB Barometer over time. The OeNB Barometer underwent a range of methodological changes, in part due to the COVID-19 pandemic. Until 2019, the OeNB Barometer relied on CAPI only using a stratified clustered random sample. Necessitated by lockdowns during the COVID-19 pandemic, telephone interviews with the option to participate online were used in early 2020. In 2021, additional online participants were recruited using access panels, i.e., from a participant pool registered with IFES. Starting in 2022, a combination of CAPI and CAWI was used. At this stage, however, CAWI respondents were almost exclusively recruited via access panels and could thus not be considered a random sample. Moreover, up to and including 2022, only design and post-stratification weights were used.

Starting with the survey wave conducted in late 2023, the methodological approach was redesigned. The survey now relies on a mixed mode approach of CAPI and CAWI, with a share of 10% to a maximum of 30% CAWI. Since recruitment for CAWI now uses push-to-web invitations by letter to randomly drawn addresses instead of an online access panel, the sampling procedure for both modes now relies on random selection of participants. Moreover, an additional weighting step has been introduced in the form of non-response weights based on geographic and interviewer characteristics.

The mixed-mode approach and recruitment techniques in their current form have several advantages. First, random sampling must be considered far superior to access panels or convenience samples in any interview mode, resulting in considerably less bias. Second, on the operative level, CAWI can act as a buffer in case of unexpected response rates or other obstacles for CAPI, as its implementation is quicker and more flexible than CAPI. Third, CAWI may be able to reach parts of the population less likely to participate in CAPI and may thus reduce overall non-response bias. Also, different modes may introduce different forms of response and non-response bias that cannot be fully accounted for by weighting (see section 3). Nevertheless, having both modes represented in the data allows monitoring these biases. Moreover, different modes require considerable care in designing the questionnaire so that it is suitable for online and personal interviews.

The remainder of this document is structured as follows: Section 1 describes the sampling design for CAPI and CAWI in detail. Section 2 gives an overview of the fieldwork and interview procedure. Section 3 details the weighting method, presenting design weights, non-response weights, post-stratification weights and final weights. Section 4 concludes with a short summary and outlook.

1 Sampling design

This section describes the current sampling approach for the OeNB Barometer and is thus valid for waves from 2023 onward. The CAPI sampling procedure, however, was already employed in 2019 and before, and remained largely unchanged. The CAWI recruiting procedure was first employed in the OeNB Barometer on a large scale in 2023. Before that, CAWI relied on an access panel or on push-to-web from computer assisted telephone interviewing (CATI). Concerning data sources, addresses are obtained from Austrian Post (Österreichische Post AG), geographic information is obtained from Statistics Austria.

1.1 Sampling design for CAPI

Following the design of the European Social Survey (ESS ERIC, 2024), the sampling procedure for CAPI is a stratified three-stage cluster sampling design. The sampling frame are all Austrian private addresses provided by Austrian Post. It provides a near-complete coverage of the Austrian residential population, with the exception of individuals without a physical address and those living in institutions such as prisons or nursing homes. From this sampling frame, a stratified and clustered sample is drawn using 1) strata based on NUTS-3 regions and municipality size, from which 2) primary sampling units (PSUs) are drawn. 3) Within PSUs, random household samples are drawn, resulting in address lists with 8 (Vienna) or 12 (all other regions) households used by interviewers. 4) Upon contact, individuals are randomly selected within households. The sampling process is described in further detail below.

First, to guarantee a balanced distribution of the gross sample across Austrian regions and municipality sizes, strata are created as combinations of NUTS-3 regions and four categories of municipality size in all federal provinces except Vienna (see table 2). As is evident in table 2, not every NUTS-3 region contains municipalities in all population size categories. For Vienna, each of the 23 districts is assigned its own stratum. In total, this results in 124 strata.

As illustrated in figure 1, the resulting strata are populated with the smallest geographic counting units (so-called “enumeration districts”) available from Statistics Austria. If, based on the population size, the number of addresses to be selected within a stratum becomes too small, strata are combined by merging municipality size categories within NUTS-3 regions to ensure the selection of at least one enumeration district per stratum.

These enumeration districts represent the PSUs which are drawn from the strata in the next step. Using enumeration districts instead of households as PSUs achieves a geographic clustering of addresses within a stratum, which is necessary when conducting on-site personal interviews to keep distances that interviewers need to travel small and thus cost effective.

Second, the PSUs (i.e., enumeration districts) are drawn from the strata. The number of PSUs to be drawn from each stratum is determined by the size of the gross sample necessary to reach the target sample size, the number of households to be drawn per PSU as well as the population size of the respective stratum.

For the OeNB Barometer, the overall target sample size is set at 1,400 completed interviews, of which at least 70% (i.e., a total number of 980 in case of the late-2023 wave) are to be conducted via CAPI. The CAPI target sample is divided between the 124 strata according to their share of the total population of Austria. The gross sample required to reach the targeted number of completed interviews is calculated based on past experiences, assuming an average response rate of about 37% for Vienna and about 41% for the rest of Austria. In all provinces except Vienna, 12 households are selected per PSU, while in Vienna, due to the particularly short distances between PSUs, only eight households are selected per PSU. On this basis, the number of PSUs to be drawn from each stratum can be calculated. To illustrate how the number of PSUs to be drawn is identified, table 3 shows the relevant summary metrics for the late-2023 wave of the OeNB Barometer at the level of Austrian provinces.

Having determined how many PSUs are to be drawn per stratum, the PSUs are randomly drawn with replacement from the strata. The probability of drawing a PSU is proportional to the population size in the PSU, i.e., a PSU with a higher number of residents has a higher chance of being selected.

Third, from the selected PSUs, random samples of households, which form the secondary sampling units, are drawn without replacement. As mentioned above, in Vienna, 8 households per PSU are drawn into one address list used by an interviewer. In all other municipalities, 12 households are drawn into a list. As a PSU can be selected more than once, there can be multiple lists of 8 (Vienna) or 12 (rest of Austria) for a single PSU.

Fourth, an interviewee is drawn from individuals, i.e., the tertiary sampling units, present at the address at the time of contact. Quasi-random selection is achieved by asking for participation of the household member aged 16 or older with the closest upcoming birthday. However, if respondents from groups that are typically difficult to reach are available, guidelines for interviewers recommend deviating from a random selection. In late 2023, the CAPI gross sample consisted of 2,452 addresses, resulting in 987 completed personal interviews in the net sample with an average response rate of 40.3%.

1.2 Sampling design for CAWI

Similar to CAPI, sampling is based on address data from Austrian Post. Except for the geographical clustering of households within strata, which is unnecessary as no interviewer travel time needs to be considered, the same sampling procedure as described for CAPI (see above) applies. Consequently, the target sample is again allocated to the 124 strata according to population size. The gross sample is calculated based on the inverse of the expected participation rate for push-to-web surveys, which is assumed at 17.5%. Having identified how many households are to be selected per stratum, the corresponding number of postal addresses is drawn. ² As in CAPI, quasi-random sampling within households is achieved by asking for participation of the household member aged 16 or older with the closest upcoming birthday. The CAWI gross sample in late 2023 consisted of 2,320 addresses receiving an invitation letter, resulting in 442 completed online interviews with an average response rate of 19.1%.

2 Fieldwork

For CAPI, participants are contacted by letter announcing the selection into the survey sample and the visit by an interviewer. Interviews are conducted verbally in person at the interviewees’ place of residence. Some questions are supplemented by cards which show questions, items or answer options in print. No incentives are offered for participation in personal interviews. Interviewers all receive basic training by IFES. Moreover, the majority of interviewers have already had experience with working for the OeNB Barometer.

For CAWI, participants are contacted by letter, which includes a QR code with a unique URL allowing only one completion of the questionnaire per invitation. The online questionnaire in web interviews is accessible both on computer and mobile devices, with responsive web design ensuring legibility and usability on all common screen sizes.

To increase usability and avoid scrolling particularly on mobile devices, item batteries, i.e., items with identical answer options that conventionally would be presented in matrix view, employ an “auto jump” function. Items are thus shown separately and, upon selection of the response, automatically move to the next item until the battery is completed. If needed, participants are able to close the questionnaire and continue at a later time with their progress saved. Contact information in case of technical difficulties is provided. Participation in CAWI is incentivized unconditionally with five euro included in the invitation letter. Moreover, upon completion of the questionnaire, participants receive a ten-euro voucher as a conditional incentive.

In general, differences in question phrasing between CAPI and CAWI are kept to a minimum. However, some differences exist concerning answer options, particularly in “don’t know” and similar response options. In CAPI, “don't know” is not read aloud as an option but is marked as a spontaneous response if given. In CAWI, this option is visually separated but explicitly presented and may thus be chosen by a larger proportion of respondents. While alternative approaches for presenting “don’t know” response options in CAPI exist, these can also lead to deviations from results obtained via CAWI (see de Leeuw et al., 2016; Porritt and Marx, 2011). In both modes, missing responses are coded accordingly, e.g., as “don’t know” or “refused”. No values are imputed, as these types of item non-response may present relevant information, particularly for knowledge and attitude questions, which are frequently asked in the OeNB Barometer.

Interviewers and the programming of questionnaires generally assure a certain level of data quality. Moreover, interview data are checked for plausibility and completeness. For example, due to anomalies, specifically a proportion of >33% missing responses overall, 15 CAWI cases were excluded from the data of the late-2023 wave. Excluded cases are not counted toward the net sample and thus factor into the non-response weights described in section 3.2.

3 Weighting

3.1 Design weights

Design weights are necessary to compensate for individuals’ different probabilities of being included in the gross sample. In line with the sampling strategy, design weights follow the general approach of the European Social Survey (ESS ERIC, 2024). In general, design weights are calculated as the reciprocal of the sampling probabilities. For CAPI, weights are calculated as \(\frac{1}{(p_{1} \times p_{3} \times p_{4})}\). For CAWI, weights are calculated as \(\frac{1}{(p_{2} \times p_{4})}\). Sampling probabilities \(p_{1}\) to \(p_{4}\) are computed as follows, with lower case letters representing sampled frequencies, upper case letters total frequencies, and indices the clusters from which samples are drawn:

1. For CAPI, \(p_{1}\) represents the probability of a PSU (“enumeration district”, see subsection 1.1) to be drawn from its stratum, computed as the number of drawn PSUs \(u\) in the stratum \(s\) (\(u_{s}\)) multiplied by the total number of households \(H\) in the PSU \(u\) within the stratum \(s\) (\(H_{us}\)) over the total number of households \(H\) in the stratum \(s\) (\(H_{s}\)), i.e., as \(p_{1} = u_{s} \times \frac{H_{us}}{H_{s}}\).

2. For CAWI, \(p_{2}\) represents the probability of selecting a household within the stratum, calculated as the number of sampled households \(h\) in the stratum \(s\) (\(h_{s}\)) over the total number of households \(H\) in the stratum \(s\) (\(H_{s}\)), i.e., as \(p_{2} = \frac{h_{s}}{H_{s}}\).

3. For CAPI, \(p_{3}\) represents the probability that a household is selected into an address list from a PSU. It is calculated as the list length 8 (for Vienna) or 12 (for other provinces) over the total number of households \(H\) in the respective PSU \(u\) (\(H_{u}\) ) , i.e., as \(p_{3(Vienna)} = \frac{8}{H_{u}}\) or as \(p_{3(other\ provinces)} = \frac{12}{H_{u}}\).

4. On the household level, \(p_{4}\) reflects the probability of an individual being selected from a household. It is calculated as the reciprocal of the number of individuals aged 16 or older \(I\) in the respective household \(h\) (\(I_{h}\)), i.e., as \(p_{4} = \frac{1}{I_{h}}\).

3.2 Non-response weights

Non-response weights aim to compensate for different participation probabilities in the gross sample. While design weights address the probability of being included in the gross sample, non-response weights address the probability that an interview is actually completed. To this end, we follow the general approach of the Household Finance and Consumption Survey (Albacete et al., 2023), using available data on the municipality level as well as interviewer characteristics in case of CAPI. However, due to data limitations, e.g., unavailable survey paradata, our approach cannot include individual or household-level characteristics of non-completed interviews and thus only relies on the most currently available municipality statistics and interviewer characteristics. ³

On the municipality level, the following variables are used based on the most current data from the Statistics Austria microcensus survey:

1. Average annual gross income (as a continuous variable)

2. Share of university educated population (as a continuous variable)

3. Share of foreign citizens (as a continuous variable)

4. Average age of population (as a continuous variable)

5. Unemployment rate (as a continuous variable)

6. Municipality size, with levels

   1. < 2,000 inhabitants

   2. 2,000–5,000 inhabitants

   3. 5,001–20,000 inhabitants

   4. 20,001–50,000 inhabitants

   5. 50,001–300,000 inhabitants

   6. > 300,000 inhabitants

For CAPI only, the following interviewer characteristics are used:

1. Gender

2. Age groups from 30 to 84 in steps of five years (as a continuous variable)

3. Education (upper secondary, school leaving certificate)

4. Length of employment at IFES, with levels

   1. 0–9 years

   2. 10–19 years

   3. 20+ years

Non-response weights are calculated entirely separately for the samples of each survey mode (CAWI and CAPI) to prevent inflating non-response weights by correcting for entirely mode-specific effects on non-response. ⁴ This approach is appropriate as in both modes independent samples are drawn from the same sampling frame. To estimate the likelihood of participation, we use logistic regressions separately for both the CAPI and CAWI subsamples, predicting the probability of participation for each individual in the sample. Table 4 presents the corresponding regression table.

To avoid excessive variance and minimize the influence of outliers with extreme weights, the probabilities of participation resulting from the logistic regressions are (separately for the CAPI and CAWI samples) divided into five quintiles (see table 5, column 1). The reciprocals of the means of these quintiles yield the non-response weights (table 5, column 2). These raw weights are then normalized separately relative to the size of the respective net sample of each survey mode, (table 5, column 3), ensuring that the sum of the weights corresponds to the number of participants. Normalizing the raw weights separately for the CAPI and CAWI subsamples means that the resulting non-response weights account only for structural differences in response rates between the modes that can be attributed to the covariates outlined above but not for non-response that is solely attributable to the main effect of the survey mode. This also explains why the normalized weights are of similar size despite the very different response propensities for CAPI and CAWI.

3.3 Post-stratification

Post-stratification aims to weight cases in a way that the distributions of certain characteristics or combinations of characteristics match known distributions in the population. Post-stratification is used in addition to design and non-response weights aiming to further reduce bias. The following combinations of characteristics are used for post-stratification:

1. Gender crossed with age (in groups 16–29, 30–59, 60+)

2. Education (with or without school leaving certificate) crossed with residence in Vienna

3. Federal province

4. Internet usage (below once a week or at least once a week) crossed with age (in groups 16–44, 45–64, 65+)

As target distributions, data from the most current microcensus surveys by Statistics Austria are used. Data on internet use are taken from an extra survey on ICT usage in households at the end of the microcensus survey for a stratified random subsample via CAWI and CATI, which is mandated by EU law and conducted on an annual basis. Although the target population of the European survey comprehends adults from 16 up to 74 years old, for Austria age groups were extended up to 84 years. Even though the sample size is about four times as large as the one of the OeNB Barometer, target distributions relying on this data may be less accurate compared to the actual microcensus data, as only a subsample is asked and participation in the extra survey is nonobligatory.

Including internet usage (crossed with age) into post-stratification aims to capture some of the remaining heterogeneity in the compositions between CAPI and CAWI subsamples, in particular concerning (a lack of) online affinity, which may not only be correlated with the propensity to participate in CAPI or CAWI surveys, but also with responses to questions of interest, particularly on the utilization of digital financial products or payment methods or digital financial literacy.

Post-stratification weights were computed using a raking algorithm (also iterative proportional fitting ) that aligns the distributions of characteristics in the sample to the target distribution by iteratively adjusting sample weights. As opposed to design and non-response weights, post-stratification was conducted jointly for the CAPI and CAWI subsamples. Weights were computed on the basis of design and non-response weights (as a product) using the R package anesrake (Pasek, 2018). No limits are set on deviations of samples from target distributions, i.e., all variables were included in the raking procedure.

3.4 Final weights

Final weights are computed by using the product of design and non-response weights as the basis for the post-stratification algorithm (see above). As mentioned, design and non-response weights are calculated separately for the two survey modes, whereas post-stratification weights are computed for the full sample regardless of the survey mode. Moreover, by calculating separate post-stratification weights, additional final weights are provided exclusively for CAPI when only the CAPI subsample is intended for use in analyses.

In general, weighting reduces bias but increases variance in comparison to unweighted data. In some instances, however, due to the combination of weighting methods, weights may also have a net positive effect on variance. For the late 2023 wave of the OeNB Barometer, we find that design and post-stratification weights alone result in an unequal weighting effect ⁵ ( UWE ) of 1.59, which only marginally increases to 1.63 if non-response weights are introduced. Initially, no upper limit was set for the final weights, which resulted in few weights above 5 and none above 8. To avoid giving excessive weight to individuals and ensure a uniform approach over the semiannual survey waves, we limit final weights to the maximum value of 5. As shown in chart 1, the distribution of final survey weights is right skewed and exhibits few observations with weights beyond a value of 3.

Finally, table 6 compares selected weighted and unweighted mean values and proportions of variables taken from the late-2023 wave of the OeNB Barometer. Looking at the average age for the unweighted and weighted samples, we find that the young are underrepresented in the survey likely due to lower response rates. Moreover, the weights successfully corrected for the accidental oversampling of Lower Austria and Styria. That weighting is not just an academic exercise becomes evident when examining people’s expectations for the economy: For the weighted sample, the weighted proportion of those having a positive outlook on the economy is 2.5 percentage points higher than the unweighted one.

4 Concluding remarks

The document serves as reference for OeNB Barometer waves in 2023 and later. It aims to give users of OeNB Barometer data detailed insights into the survey design and enable direct reference of the methodological approaches. Future adaptations to the OeNB Barometer will be incorporated into updated versions of this document. Potential improvements include the integration of data necessary for accurate variance estimation and analyses of the effects of survey mode on response rates and response bias.

The current approach represents a significant step toward the “gold standard” of survey methodology exemplified by the Survey of Consumer Finances (Aladangady et al., 2023) by the US Federal Reserve with its comprehensive use of non-response weights, replicate weights and multiple imputations of missing responses. In Europe, the Austrian contribution to the Household Finance and Consumption Survey (Albacete et al., 2023) closely approximates this benchmark.

As a major development, the OeNB Barometer now uses a mixed-mode approach, with both personal and online interviews relying on a stratified multistage random sampling strategy. Moreover, non-response weights have been introduced in addition to design and post-stratification weights. While the OeNB Barometer cannot completely replicate the comprehensive methodology of more elaborate surveys, its development over time reflects considerable efforts to achieve representative data on the Austrian population of high quality while maintaining cost-effectiveness.

References

Aladangady, A., J. Bricker, A. C. Chang, S. Goodman, J. Krimmel, K. B. Moore, S. Reber, A. H. Volz and R. A. Windle. 2023. Changes in U.S. Family Finances from 2019 to 2022: Evidence from the Survey of Consumer Finances. Board of Governors of the Federal Reserve System. https://www.federalreserve.gov/publications/files/scf23.pdf

Albacete, N., P. Lindner and K. Wagner. 2023. Eurosystem Household Finance and Consumption Survey 2021: Methodological notes for Austria. OeNB Report 2023/1. Oesterreichische Nationalbank. https://www.hfcs.at/dam/jcr:e09e4bd0-066d-41bf-8f85-d2b161f037ef/HFCS-Austria-2021_Addendum_Methodological-Notes.pdf

de Leeuw, E. D., J. J. Hox and A. Boevé. 2016. Handling Do-Not-Know Answers: Exploring New Approaches in Online and Mixed-Mode Surveys. In: Social Science Computer Review 34(1). 116–132.

European Social Survey European Research Infrastructure Consortium (ESS ERIC). 2024. ESS Round 12 Survey Specification for Member, Observer and Guest Countries. https://europeansocialsurvey.org/sites/default/files/2024-04/ESS012_projection_specification_v2.pdf

Kish, L. 1995. Survey sampling. Hoboken: Wiley.

Pasek, J. 2018. Anesrake: ANES raking implementation (R package version 0.80). https://cran.rproject.org/web/packages/anesrake/index.html

Porritt, D. and B. R. Marx. 2011. Handling “don’t know” responses in parallel CATI and online surveys. In: Australasian Journal of Market & Social Research 19(2). 27–33.