Reproducibility and Validity of Dietary Intake Measurements

Reproducibility, Repeatability, and Reliability

'Reproducibility' is the extent to which a tool is capable of producing the same result when used repeatedly in the same circumstances. The terms 'repeatability' and 'reliability' are often used synonymously with 'reproducibility.' All of these terms are equivalent to the word 'precision,' often used by biochemists to describe the variation of a measurement based on repetition of a particular assay using a single piece of equipment.

A measurement may have good reproducibility and yet have poor validity, but a measurement that has good validity cannot have poor reproducibility.

In reality, 'the same circumstances' can never exist in relation to dietary measurements because diet (whether of individuals, households, or countries) varies over time, be it on a daily, weekly, seasonal, or annual basis. In epidemiological studies, the aim is usually to assess 'usual' intake. Part of the variation in any dietary measurement will thus relate to genuine variability of diet. The remaining variation will relate to biases associated with the method. Due consideration must be given to these time-related factors when evaluating the reproducibility and validity of dietary measures, and a well-designed validation study will separate the variability associated with reproducibility (the error in the method) from that associated with genuine biological variation over time.

Validity

Validity is an expression of the degree to which a measurement is a true and accurate measure of what it purports to measure. Establishing validity requires an external reference measure against which the 'test' measurement (the one being used in the main survey or research activity) can be compared. In nutrition, there is no absolute reference measure of the truth. Every measurement of dietary intake includes some element of bias. The best that can be managed is to assess relative or congruent validity of measurements, comparing results obtained with the test instrument with what are believed to be more accurate measures of food or nutrient intake obtained, for example, using a biological marker.

There are two main problems arising from inaccurate measurements:

1. Incorrect positioning of a country, household, or person in relation to an external reference measure (e.g., dietary reference values)

2. incorrect ranking of countries, households, or people in relation to one another

The first type of error can result in inappropriate investigations or actions being taken to remedy an apparent dietary deficit or excess that does not really exist. Alternatively, no action may be taken when some is needed (e.g., a true deficit is not detected because diet is overestimated). The second type of error tends to undermine the ability to assess relationships between diet and health. For example, if someone who properly belongs in the top quarter of the distribution of saturated fat intake (associated with an increased risk of heart disease) is classified in the bottom quarter because he or she underre-ports his or her true fat intake, the relationship between fat intake and outcome (risk of heart disease) will not be shown. Again, this can lead either to inappropriate recommendations for improving health in the population or, more often, to a failure to take action because the true relationship between diet and health is obscured by measurement error.

Table 1 lists some of the sources of measurement error, their principal effects, some ways of taking errors into account in analysis, and ideas for dealing with them in practice. At the national level, errors in reporting of food production, imports, exports, food moving in and out of stocks, and nonfood uses (e.g., sugar used in the brewing industry) tend to produce overreporting of food and nutrient availability in economically developed countries and underreporting in developing countries. The consequences for between-country comparisons will therefore vary nutrient by nutrient. Where true values are higher in developed countries and lower in developing countries (e.g., energy), a tendency in wealthier countries to overstate consumption and in poorer countries to understate consumption will exaggerate the true differences between countries (Figure 1). Where true intakes tend to be higher in developing countries and lower in developed countries (e.g., nonstarch polysaccharides), such biases in measurement will minimize any apparent differences between countries. These types of errors do not lend themselves easily to correction or adjustment. The best solution is to try to improve reporting mechanisms that reduce duplication or omission, although this may be particularly problematic in developing countries, in which a high percentage of the population is engaged in subsistence agriculture.

At the household level, information can be obtained from records of food acquisitions (expenditure and/or amounts), often referred to as household budget surveys. Errors associated with the recording process (omission and misrecording) will contribute to the error of estimates of consumption. Keeping a record of purchases can lead householders to alter their usual purchasing patterns or encourage purchasing in excess of true requirements (especially easily storable items such as flour, sugar, oil, and cooking or spreading fats). Some form of crosscheck is necessary—either an internal reference measure such as a cross-check list of foods purchased or used at mealtimes or an external reference measure such as an independent assessment of food consumption by all household members. There are problems related to the estimation of waste and the amount of food given to pets. The true amount of food wasted cannot be known, and direct measures of waste are likely to introduce bias into the measurements. Adjustments for waste range from 4 to 10%, and 6% is a typical value for average waste.

In household budget surveys, the amounts of food eaten away from home (or, more correctly, derived from outside the household food supply) are sometimes measured directly by individual household members who keep records of their food and drink consumption away from home. Alternatively, the amount of food eaten from outside the household food supply can be measured indirectly by asking the main household respondent to record which meals have been consumed from the household food supply and the ages and genders of all people present at those meal, including any visitors. The menu records do not usually include information about snacks between meals. Whether based on the direct or indirect method, household surveys tend to underestimate the amount of food eaten away from home and thus to overestimate the proportion of the diet consumed from the household food supply.

The estimate of the nutrient content of the diet in household-level surveys is based on nutrient conversion factors. These differ from food composition data. Food composition data reflect the chemically determined nutrient content of foods. Nutrient conversion factors modify these values to allow for preparation and cooking losses so that the final estimate of the nutrient content of the diet relates the amount of food initially purchased to the amount of food and nutrient likely to be consumed. There are many assumptions in the determination of the nutrient conversion factors (e.g., the proportion of water-soluble vitamins lost in the cooking process). The result may be an over- or underestimate of

Table 1 Sources of error at different levels of dietary assessment, their effects, ways to adjust for measurment error in analysis, and ways to minimize error in data collection

Level of assessment

Source of error

Effect of error

Statistical adjustment

Ways to minimize

National: Food balance sheets

Household: Household budget surveys, with or without larder inventories

Individual: prospective or retrospective methods

Inaccurate values of food production, imports, exports, storage, nonfood uses Omission of food items

Misrecording of food items

Distortion due to effect of survey (e.g., selection of items believed to be 'healthy') Overpurchasing stimulated by measurement process

Estimate of amount wasted or given to pets

Estimate of amounts of food eaten away from home or consumed by visitors, based on menu records Use of nutrient conversion factors

Underreporting

Differential misclassification

Distortion of usual diet (prospective recording or retrospective recall)

Over- or underemphasizes apparent differences between countries

Underestimation of consumption, especially for certain foods (e.g., sweets) Misrepresentation of true consumption pattern Misrepresentation of true consumption pattern

Overestimation of consumption, especially for certain foods (e.g., flour and sugar)

If amount is underestimated, leads to overestimation of consumption (and vice versa) If amount is underestimated, leads to underestimate of total consumption (if overestimated, leads to overestimation) Unknown biases introduced into calculation of the nutrient content of the diet Underestimate of usual intake

The error in measurement not consistent between subgroups, making valid comparisons impossible Overestimation of intake of some foods and nutrients and underestimation of others

Not possible; error likely to vary nonsystematically between countries Inflate estimate of intake to compensate for known underestimation Not possible; likely to differ between households for different foodstuffs Not possible; likely to differ between households for different foodstuffs

Deflate estimate of intake to compensate for known overestimation

Deflate estimate of intake to compensate for known overestimation (or inflate) Inflate estimate of intake to compensate for known underestimation (or deflate)

Not possible; errors not systematic between foods

Calibrate result against valid measure of intake and adjust accordingly Identify likely error according to characteristics on which subgroup comparisons are to be based, and adjust accordingly Not possible unless nature of distortion is known for different foods and for different subgroups

Improve reporting mechanisms and estimates of waste and production losses

Cross-check food purchasing list at final interview

Cross-check food purchasing list at final interview

Cross-check food purchasing list at final interview

Validate purchasing against reference measure of actual consumption at individual level (for all household members)

Improve estimates of waste through surveys

Household members keep records of amounts eaten away from home (but see problems with records kept by individuals)

Improve estimates of nutrient content and of preparation and cooking losses

Use more than one measure of intake, including valid reference measure

Undertake appropriate validation studies within subgroups

Use biomarkers of intake to confirm dietary measures; use more than one measure of intake, including valid reference measure

Adapted from Nelson M (1998) Methods and validity of dietary assessment. In: Garrow J and James WPT (eds.) Human Nutrition and Dietetics, 10th edn. London: Churchill Livingstone.

Observed difference

True difference

Observed difference

True difference

Observed difference

True difference

Overestimate of energy in developed country (-), underestimate in developing country (— )

Observed difference

True difference

Overestimate of NSP in developed country (-), underestimate in developing country (— )

Figure 1 Effects of measurement error on between-country comparisons of nutrient availability.

the actual nutrient content of the diet consumed, depending on the nutrient concerned.

At the individual level, the high reproducibility of results using a given technique led researchers for many years to assume that the observed results were valid. Reproducibility was confused with validity. Researchers argued that the results from their chosen technique were correct, but they were unable to explain adequately why measurements of the same diets using different techniques yielded different results. It is now clear that different techniques introduce different types of errors in assessment. Moreover, one can argue that the main sources of error in the assessment of diet at the individual level (e.g., errors in perception of frequency of consumption and of food portion sizes; the desire to increase the reporting of those foods perceived as being 'healthy' or to reduce the reporting of foods perceived as being 'unhealthy'; and items forgotten, intentionally or unintentionally) are not readily amenable to correction.

The most common error in the assessment of diet at the individual level is underreporting. Thus, the mean estimate of energy intake in a group of subjects that includes people who underreport their true consumption is also likely to be an underestimate. Moreover, any comparison of subgroups that include different proportions of underreporters will yield a false picture of the relationships between subgroups. For example, when comparing energy intake based on 7-day weighed records between normal weight and overweight subjects, it is often observed that the overweight subjects (whose energy requirements are higher) will report energy intakes equal to or less than those of the normal weight subjects. This is because overweight subjects show a strong tendency to underreport their usual consumption levels, particularly of foods with a high proportion of fat. It is worth noting that there is also a small group of subjects who overreport their usual consumption, but their characteristics are less readily defined.

In many validation studies, comparisons are made between methods to establish 'concurrent validity,' often between a new instrument such as a food frequency questionnaire and a more established technique such as a weighed inventory of diet. Table 2 summarizes the problems associated with the use of specific dietary reference measures. The

Table 2 Limitations of dietary reference methods appropriate for validation of dietary assessment measures

Reference

Limitations

method

Weighed records

Underreporting

and household

Unrepresentative of 'usual' diet over

measures

sufficient number of days

Distortion of food habits due to recording

process

Diet history

Interviewer bias

Inaccuracy of portion size reporting due to

conceptualization and memory errors

Errors in reporting of frequency, especially

overreporting of related foods listed

separately (e.g., individual fruits and

vegetables)

Requires regular eating habits

Repeat 24-h

Under- or overreporting of foods due to

recalls

reporting process (e.g., alcohol

underreported and fruit overreported)

Unrepresentative of 'usual' diet over

sufficient number of days

Inaccuracy of portion size reporting due to

conceptualization and memory errors

principal error is likely to be underreporting within certain subgroups, and any validation study needs to address this. Issues regarding distortion of diet are more difficult to identify, however. Work has been carried out to identify some of the errors associated with perception, conceptualization, and memory in relation to assessment of food portion sizes. Agreement between methods is no guarantee of validity. If the two methods are biased in the same way (e.g., using a diet history to validate a short food frequency questionnaire), then the observed level of agreement is likely to overstate the validity of the new method. If the validation is based on repeated measures and the errors in the reference assessment are repeated (e.g., validating a food frequency questionnaire against repeat 24-h recalls), then the level of agreement between the two methods is likely to underestimate the true correlation between the test measure and the truth because the true variance of diet will be underestimated by the reference measure.

Some types of bias may influence reporting in a similar way in all subjects. For example, if a food frequency questionnaire does not ask for sufficient detail about tropical fruit consumption in a survey of West Indian families living in London, then all responses are likely to be biased in the same way. This will result in nondifferential misclassification: Everyone's intake of tropical fruit (and hence vitamin C and carotene intake) is likely to be similarly misclassified. If, on the other hand, the nature of the bias is likely to be related to individual characteris-tics—for example, West Indian women overreport their tropical fruit consumption compared to West Indian men—then the tropical fruit intake of men and women will be misclassified in different ways in different subgroups. This is known as differential misclassification. If the risks for men and women are being compared in a diet-disease analysis, the errors in the estimates of those risks will not be the same for both groups.

There are many factors in addition to body size or gender that may influence the ability of a dietary measure to estimate accurately the level of food consumption or nutrient intake. Table 3 lists the potential confounders of good dietary reporting that may affect validation studies. If there is concern about misclassification in one group (e.g., smokers) being different from misclassification in another group (e.g., nonsmokers), then validation must identify the extent of misclassification in the two groups independently so that it can be compensated for in any subsequent analyses. It is therefore important to measure all of the potential confounders that may be a source of misclassification (differential or nondifferential).

Table 3 Factors that may influence the reliability and validity of dietary assessments and that may need to be measured in a validation study

Related to the subject Related to the measuring process

Table 3 Factors that may influence the reliability and validity of dietary assessments and that may need to be measured in a validation study

Related to the subject Related to the measuring process

Age

Portion size

Gender

Interviewer

Height, weight, etc.

Learning effects

Region

Recency effects

Social class

Lag time

Education

Number of foods on an FFQ

Language

Nutrient database

Culture and ethnic

background

Smoking

Social approval

Social desirability

The design of a validation study must also reflect the purpose for which the dietary assessment is being carried out. Most dietary assessments measure current or recent consumption. However, if the aim is to estimate past diet related to the induction of cancer 10 years before the time of the assessment (e.g., in a case-control study), then the validation process should in theory address the relevant time period. In practice, this may be very difficult (i.e., if there is no robust reference measure of diet relating to the relevant time period). In such circumstances, the weaknesses of an unvalidated test measure should not be overlooked nor understated.

101 Everyday Tips for Losing 10 Pounds

101 Everyday Tips for Losing 10 Pounds

Studies show obesity may soon overtake tobacco as the leading cause of death in world. Who Else Could Use 101 'Everyday' Ways to Lose 10 Pounds or more and Keep it Off! You've been putting it off too long. Hey, everyone needs to lose weight from time to time. You're no different!

Get My Free Ebook


Post a comment