Food behavior checklist effectively evaluates nutrition education
Mary L. Blackburn
Marilyn S. Townsend, UC Cooperative Extension (UCCE), Alameda County
Lucia L. Kaiser, Department of Nutrition, UC Davis
Anna C. Martin, UCCE San Joaquin County
Estella A. West, UCCE Santa Clara County
Barbara J. Turner, UCCE Los Angeles County
Amy B. Joy, Department of Nutrition, UC Davis
California Agriculture 60(1):20-24. DOI: 10.3733/ca.v060n01p20.
We developed a short food behavior checklist (FBC) to evaluate the impact of nutrition education on fruit and vegetable intake among ethnically diverse women in the Food Stamp Nutrition Education Program (FSNEP) and the Expanded Food and Nutrition Education Program (EFNEP). To validate the FBC, interviewers collected three 24-hour dietary recalls as well as responses to 11 FBC behavioral questions about fruits and vegetables from 100 English-speaking, low-income women at baseline. A randomly selected subgroup (n = 59) provided a blood sample for analysis of total serum carotenoids at baseline and follow-up. After 6 hours of nutrition education, the treatment group reported significant improvements in three of the seven FBC questions related to fruit and vegetable intake, while no significant changes occurred in the control group. All seven FBC questions were significantly correlated with total serum carotenoids. This short, culturally neutral FBC is a valid and reliable indicator of fruit and vegetable consumption. Compared with the 24-hour dietary recall, it is also less time-consuming to administer, code and analyze, with a reduced respondent burden.
Valid and reliable evaluation tools are essential to justify federally funded nutrition-education programs, such as the Expanded Food and Nutrition Education Program (EFNEP) and the Food Stamp Nutrition Education Program (FSNEP). UC Cooperative Extension specialists need dependable and consistent evaluation data to develop curriculum and staff training materials. Evaluation tools can also demonstrate the effectiveness of such programs, and can help nutrition educators tailor them to the specific needs of diverse populations. For example, if evaluation data indicates that participants are consistently practicing certain desirable food behaviors, the educator may adapt the intervention and devote more teaching time to other behaviors that need improvement. In addition, good evaluation data at the county and state levels can help program directors determine how to allocate scarce resources and leverage additional funds to enhance or expand successful programs. Data documenting the successful outcomes of the EFNEP and FSNEP programs has been useful in generating new funding for nutrition and wellness education, as well as research activities.
UCCE nutrition educator Lan Nguyen addresses a class at Fremont Adult School. In California two such programs, EFNEP and FSNEP, together evaluate the nutrition education received by 21,000 low-income families each year.
The U.S. Department of Agriculture (USDA) funds EFNEP and FSNEP to provide community nutrition education to families with limited incomes (up to 180% of the poverty level). EFNEP has served California families with minor children since 1969 and currently conducts adult education programs targeting head of households in 16 counties. The target group is preschool children, but families with older children are not denied services. About 10,000 families graduate (with certificates) and are evaluated annually after receiving a minimum of 6 to 8 or more hours of instruction in nutrition, money management and food safety. Also, 10 of the 16 counties have youth programs that provide nutrition education training to teachers in elementary schools with large percentages of children qualifying for free and reduced-priced meals. The FSNEP program, instituted in California in 1994, targets food stamp recipients and applicants in 42 counties, and evaluates the nutrition education received by about 11,000 families annually. Both EFNEP and FSNEP hire and train paraprofessional staff as nutrition education assistants (now called nutrition educators). EFNEP and FSNEP staff collaborate with agencies and groups to teach low-income adults how to plan, purchase and prepare low-cost foods for a nutritious diet.
EFNEP and FSNEP nutrition educators (including, above, Michele Brown) teach low-income adults how to plan, purchase and prepare economical, nutritious food. In California, a new food behavior checklist provides a simpler and more user-friendly method for evaluating the programs' effectiveness.
EFNEP and FSNEP are required by USDA to conduct pre- and post-education evaluations of all adult graduates to determine impacts on dietary practices. Both programs use the most recent Dietary Guidelines for Americans (USDA 2005) as a standard to measure improvement among their graduates. Since 1969, EFNEP has measured change in dietary intake with nutrition indicators derived from one pre- and one post-24-hour dietary recall. The participant is asked to recall and record all the food and drink they consumed during the 24-hour period just prior to the recall. If the recall is conducted one-on-one, the educator asks the person to recall the food items and the time of day or night they were eaten and then records the responses. In a group setting the educator explains how the recall is done and answers questions, and the participant records his or her own responses.
In a randomized, controlled study of more than 600 EFNEP graduates in California, the 24-hour dietary recall method evaluated the impact of 6 to 8 hours of nutrition education using the one-on-one teaching model. This study found significant improvement in fruit and vegetable (P ≤ 0.001) and milk consumption, food preparation, and shopping skills (Del Tredici et al. 1988). Similarly, results from annual evaluations of FSNEP graduates in California showed positive benefits for nutrition education delivered in small group settings (Joy et al. 2001).
Other states have documented that the positive impact of the EFNEP program can be long lasting (Burney and Haughton 2002; Torisky et al. 1989). A Michigan study found that the positive improvements after the EFNEP intervention were maintained for more than 5 years (Nierman 1986); plus, the families saved more money on food than the federal cost of administering the EFNEP program (Burney and Haughton 2002).
Over the last decade, the adult EFNEP program in California moved from a one-on-one teaching model to a group delivery method with between 6 and 30 participants in a class. Administering the traditional 24-hour dietary recall in a group setting is cumbersome. Data collected from these self-administered dietary recalls may not be complete, because an accurate recall requires specific probing by an interviewer that is not feasible in a group setting. Moreover, since underreporting is common with the dietary recall method, particularly among overweight women, use of this method may be especially problematic in low-income groups with a high prevalence of overweight and obesity (McClelland et al. 2001). With today's emphasis on group education, EFNEP teaching staff views the 24-hour dietary recall as an impediment and have stressed the need for a new evaluation tool.
An alternative to the 24-hour recall, food frequency questionnaires (FFQs), are also difficult and time-consuming to administer in a group (Kristal et al. 1998). FFQs include a list of 100 or more foods, a rough indication of portion size, and a set of frequency options (how often consumed) to choose from. Although brief FFQs are available (15 to 60 questions) these shortened versions may not contain enough food items or details to be useful among the ethnically diverse populations served in California (McClelland et al. 2001). With longer evaluation tools such as the full FFQ, respondent burden is also a major concern among low-literacy populations. Moreover, neither the dietary recall nor the FFQ assesses behavioral changes in food-related practices such as reading nutrition labels, removing skin from chicken, serving vegetables at dinner, eating fruits and vegetables for snacks, managing money or handling food safely. However, these are key behaviors targeted by EFNEP and FSNEP.
We recognized the need for a simple and less time-consuming evaluation tool. At USDA's request, we began a study in 1997 to validate a food behavior checklist (FBC). We developed the California FBC and tested it for validity and reliability to measure dietary quality related to chronic disease. Eleven of the 41 FBC questions were related to fruit and vegetable consumption. Our study focuses on whether the new FBC may be used as an alternative to the 24-hour food recall to assess fruit and vegetable intake among EFNEP and FSNEP participants.
Convergent validity: Agreement between two instruments or indicators measuring the same construct (i.e., food behavior checklist and 24-hour dietary recall).
Criterion validity: Agreement between an instrument or indicator and another measure considered to be a “gold standard,” which tap into the same construct (i.e., the food behavior checklist and a biochemical indicator).
Internal consistency: An alternative estimate of reliability, which measures agreement among the questions on the fruit and vegetable FBC.
Readability: Ease of understanding or comprehension due to vocabulary, sentence length, writing style and other factors on the FBC.
Reliability: Measures the stability of individual questions on the FBC over time and uses a scale to measure the agreement between similar questions.
Sensitivity to change: Ability of the FBC to detect changes in fruit and vegetable behaviors occurring as a result of the intervention.
In addition to being valid and reliable, a good evaluation tool should be responsive to change, easy to administer and analyze, and culturally relevant with a low respondent burden (). A tool is considered valid or accurate if it yields an estimate that correlates well with an accepted “gold standard” (criterion validity). The validity of a new tool may also be established by comparing it to a known tool that measures a similar concept or behavior (convergent validity). A tool that is lengthy or difficult to administer could detract from the delivery of nutrition education, as well as limit the accuracy of the evaluation data.
A reliable evaluation tool must yield similar results consistently when it is administered to the same people on at least two separate occasions in the absence of an intervention. Another test of reliability, referred to as internal consistency, is a measure of correlations among a group of related questions (within the same tool) on the same scale.
A good tool should also be sensitive enough to detect changes in client behaviors exhibited after completing the educational program. The limited-literacy and culturally diverse groups in the EFNEP and FSNEP programs are other key considerations, particularly if the evaluation tool is self-administered in a group setting.
A detailed description of the research methods and findings related to developing the comprehensive FBC appears in earlier publications (Murphy et al. 2001; Townsend et al. 2003). This paper gives an overview of the steps taken to develop a short fruit and vegetable FBC, then test it with a group of low-income women in FSNEP.
The questions included in the fruit and vegetable FBC were based on the subject matter taught in the EFNEP and FSNEP programs and the desired behavior changes to be evaluated (see box, this page).
Baseline data from the original 1997 study of low-income women (n = 100) receiving 6 hours of nutrition education was used to validate the fruit and vegetable FBC questions (Townsend et al. 2003).
We used three validation methods. First, we correlated fruit and vegetable questions with a biomarker, total serum carotenoids (criterion validity). Carotenoids found mainly in fruits and vegetables are the chief source of vitamin A in North American and European diets. These fat-soluble compounds can be stored and converted to vitamin A inside the body (Kagan 1953). They are absorbed in the small intestine, enter the blood via the lymph system and reach peak concentration in the blood within 3 to 5 hours of ingestion. The concentration of human serum (blood) carotenoids depends largely upon the content of the carotenoids in the diet (Woh1 and Goodhart 1968).
Second, we compared fruit and vegetable intakes, as determined by the FBC and three 24-hour dietary recalls (convergent validity). Finally, we compared nutrient intakes such as vitamins A, C and folate (part of the vitamin B complex), as determined by the FBC and the three dietary recalls (convergent validity).
Cronbach's alpha test was used to determine if the results were internally consistent or reliable (Nunnally and Bernstein 1994). As another means to measure reliability, the FBC questions were administered to a separate group of 44 low-income women on two different occasions 3 weeks apart, with no nutrition education in between. The fruit and vegetable FBC was also evaluated for clarity, literacy level and appropriateness for diverse audiences (non-Latino white, non-Latino black and English-speaking Latino adults).
All items were positively correlated to both serum carotenoids and dietary recall variables.
Randomized, controlled FBC trial. Performance of the short fruit and vegetable FBC was assessed among FSNEP participants in eight California counties: Alameda, Fresno, Los Angeles, Monterey, San Francisco, San Joaquin, San Mateo and Santa Clara. FSNEP staff recruited participants from schools and other community organizations.
At baseline, trained interviewers collected demographic data, three 24-hour dietary recalls and the FBC from 100 English-speaking, low-income women who were eligible for food stamps and had at least one child less than 19 years old living at home. These women were randomly assigned to the intervention group that received 6 hours of FSNEP classes or to a delayed control group that received the classes after the study was completed. The content of the nutrition education classes included the Food Guide Pyramid, food labeling, unit pricing, food selection, meal planning, food-related money management and shopping. (The USDA Food Guide was significantly revised in mid-2005, several years after our study, to stress variety rather than just promoting the consumption of five half-cup servings of fruits and vegetables per day.)
In this study, a seven-item portion of the food behavior checklist was validated as a reliable measure of changes in fruit and vegetable consumption. Alternative versions for non-English-speaking and low-literacy audiences are now being evaluated. Top, UCCE nutrition educator Nelly Camacho administers the checklist to program participants in Hayward.
The intervention and control groups were not statistically different for a number of variables: education, monthly income, household size, race/ ethnicity and participation in food assistance programs. However, the women in the intervention group were older than those in the control group, 33.2 ± 8.2 years versus 25.8 ± 4.5 years (P ≤ 0.0001) (table 1).
Complete follow-up data, including three additional 24-hour dietary recalls and a second fruit and vegetable FBC, was gathered for 73 women in the intervention group and 14 women in the control group. A randomly selected intervention subgroup (n = 59) provided a second blood sample for analysis of total serum carotenoids at completion of the educational experience.
Paraprofessional staff reported that the fruit and vegetable FBC was easier to administer and required less time than the 24-hour dietary recall. The short FBC initially contained 11 questions related to fruit and vegetable intake. Of these, seven questions were significantly and positively related to both serum carotenoid levels (the “gold standard”) and dietary recall variables () (Murphy et al. 2001). Questions related to serum carotenoid levels included: Do you eat more than one kind of fruit daily? and Do you eat more than one kind of vegetable a day?
These seven questions were significantly correlated with two or more nutrient or food group intakes (such as vitamins A and C) as estimated by the dietary recall method. The dietary recalls were analyzed using the USDA Nutrient Database for Individual Surveys (USDA/Food Survey Research Group, 1995 release, updated to Release 8). Daily servings of fruits and vegetables assessed by the 24-hour dietary recalls were correlated to total serum carotenoids (r = + 0.35, P ≤ 0.01). However, a fruit and vegetable score based on the sum of responses for the seven FBC questions had a higher correlation with serum carotenoids (r = + 0.44, P ≤ 0.001) (Townsend et al. 2003).
TABLE 1 Study sample (intervention n = 73, control n = 14) at baseline
The internal consistency of these seven fruit and vegetable questions was also acceptable based on the Cronbach alpha of 0.80. Reliability coefficients for the seven individual questions ranged from 0.35 (P ≤ 0.05) to 0.65 (P ≤ 0.0001). No ethnic or cultural differences were found among the responses to these questions at baseline (Townsend et al. 2003).
Once the validity of the fruit and vegetable FBC was established, we used it with a group of low-income women participating in nutrition education programs. Before the education experience, most of these women did not meet the Food Guide Pyramid recommendations. Only 27% consumed at least two servings of fruit and about 37% consumed three or more servings of vegetables a day. The fruit and vegetable consumption patterns of these low-income women (a mean of 4.5 servings per day) were very similar to the general U.S. population (4.4 servings, median) (Thompson et al. 2005). These fruit and vegetable figures may seem high, but note that all forms of potatoes — including french fries — were counted as vegetables. In addition, 95% of the women studied consumed less than the recommended 25 grams of fiber per day (Institute of Medicine 2002). Their fat intakes ranged from 29% to 42% of total calories, compared with the recommended range of 25% to 35%.
From baseline to follow-up, the intervention group reported significant improvements in the three fruit questions on the FBC (Townsend et al. 2003). The control group did not report any significant changes for fruits and vegetables (data not shown). However, no significant changes occurred for either group in the number of daily servings of fruits and vegetables, as estimated by the dietary recall method. Since the biochemical indicator (serum carotenoids) tended to increase over time (intervention: + 5.2 ?g/dl; controls: + 4.1 ?g/dl), the dietary recall method, as administered in this setting, may not have been as sensitive to change as the FBC.
For both the intervention and control groups, the energy intake determined by the dietary recall method was actually lower at follow-up, compared to baseline (intervention: -217 kcal; controls: -456 kcal). This drop over time may have been due to subject fatigue in providing multiple dietary recalls.
Due to limited funds, the number of women remaining throughout the study in the control group was quite small (n = 14), which probably limited the study's power to detect change in fruit or vegetable intakes assessed by the FBC or dietary recall. Use of the valid FBC in an intervention study needs to be replicated with a larger sample. This endeavor will be relatively easy now that the FBC's validity and reliability have been established, and since it is no longer necessary to collect blood serum samples and multiple 24-hour dietary recalls.
Based on the correlations with total serum carotenoids, the fruit and vegetable FBC appears to be valid and reliable for the assessment of fruit and vegetable intake. In addition, we found that the fruit and vegetable FBC has a number of advantages over the group-administered dietary recall for evaluating the impact of nutrition education. Overall, the FBC is more user-friendly and less cumbersome for both the clients and nutrition educators. The entire FBC takes less time to administer (10 minutes vs. about 30 minutes), and is less complex and easier to score than the dietary recall. The FBC was simple enough to be self-administered with minimal difficulties.
The FBC questions were reflective of the subject matter taught in EFNEP and FSNEP, and the results were responsive to the program goals and outcome objectives. Preliminary results indicate that the seven-item fruit and vegetable FBC is culturally and ethnically neutral with no differences detected in response patterns among English-speaking white, Latino and black women.
EFNEP and FSNEP would benefit from additional research to determine if the FBC is also a valid and reliable tool for non-English-speaking audiences. Research is currently under way to determine if pictorial versions of the FBC would be useful in evaluating nutrition education impact among limited-English and low-literacy populations.