past Lilly Fellow of Engineering

---

The Use of Active Learning to Address ABET Course Learning Objectives in a Large Undergraduate Environmental Engineering Class

Alison M. Cupples, Weimin Sun, Susan Masten

---

Abstract

The overall aim was to determine if in class activities based on active learning improve the achievement of Accreditation Board for Engineering & Technology (ABET) course learning objectives (CLOs) and overall student learning. The study was performed (Fall semester of 2011) in a large (~80 student) undergraduate introductory environmental engineering course (CE 280, Principles of Environmental Engineering).  The study was initiated because previous ABET data collected from CE 280 (Fall 2009) indicated a low level of CLO achievement. The approach was simple and involved between 2 and 4 in class exercises or short questions in every lecture session. The students responded to the questions using a response pad (resulting in 5% of their final grade). The questions were all multiple choice and were either qualitative questions or short quantitative questions. The questions were typically at the beginning, then after 15, 35 minutes and 50 minutes. Three sets of data were collected to determine the efficiency of this approach. Of these, two data sets were compared between 2009 and 2011. The first assessment method concerned a comparison of Course Learning Objective Evaluation (or CLOSE) forms. The second data set involved pre- and post- surveys in 2011. The third set involved a comparison of student performance in questions relating to each CLO. The CLOSE forms illustrated that the students believed they had a high level of understanding of each CLO in 2009 at the end of the semester and this did not change in 2011. The pre- and post- surveys indicated the students recognized the value of in class assessment as well as the use of response pads. The post- survey showed a statistically significant more positive response to considering the CLOs when studying. The final assessment method concerned an analysis of student performance on questions relating to each CLO. In 2009 over five CLOs were not achieved, however in 2011, only 1 CLO was not achieved. These data indicate the introduced pedagogical approach was highly successful at improving student learning of CLO content.

In Fall 2009, ABET data were collected to determine if the course learning objectives (CLOs) in a large, required, undergraduate class were being achieved. The class, Principles of Environmental Engineering and Science (or CE 280), meets three times per week for 50 minutes and typically contains ~80 students. From the data collected, it was clear that the students were having difficulty with a significant number of the topics included in this class. To address these shortcomings, a study was designed to include in-class, active learning and assessment activities to improve the teaching and learning of CE 280 content. In 2009, CE 280 contained 19 CLOs, however, this number was reduced to 15 in 2011 (Table 1). The CLOs that were similar between 2009 and 2011 were compared in this study (CLOs 1-14 in table 1). Dr. Cupples was the instructor in both 2009 and 2011.

Table 1.  A list of the 2011 CLOs for CE 280

Number	CLO
1.	Define and describe the primary environmental regulations, and discuss how regulations affect engineering practice.
2.	Use material balances to determine the concentration of water contaminants.
3.	Perform elementary chemical calculations.
4.	Describe the four step process of risk assessment.
5.	Solve hydrologic mass balance problems.
6.	Give examples of biological and chemical substances that cause water pollution.
7.	Provide examples of the human health effects of pollutants.
8.	Perform simple BOD calculations.
9.	Calculate dissolved oxygen concentrations in a river.
10.	Describe the basic engineering approaches to drinking water treatment.
11.	Describe the basic engineering approaches to wastewater treatment.
12.	List the regulated air pollutants and their sources.
13.	Describe the impact of regulated air pollutants on the environment.
14.	Define the key characteristics of a hazardous waste and provide examples of hazardous waste treatment technologies.
15*.	Define key aspects of sustainability relating to environmental engineering and science

* CLO 15 was not included in the study because it was not used in 2009.

Starting in Fall 2011, between 2 and 4 in class exercises were included in every class session. The students were asked a question and were required to answer the question using their response pad (classroom performance system or CPS by eInstruction^TM). The first homework assignment was to buy and activate their response pad. The answers they provided using their response pad contributed a small percentage (5%) to their final grade. The questions posed to the students were both qualitative questions (to highlight important concepts) as well as short, quantitative questions. The questions were both designed to target material relating to the CLOs as well as additional concepts/problems discussed in class. The questions were all in the multiple-choice format and required only 1-4 minutes to complete the answer. When longer quantitative problems were targeted (e.g. mass balance examples) only a portion of the question was given to the student, and the instructor completed the rest. Questions were often included at the beginning and end of a lecture period for review purposes. Also, in general, questions were typically inserted after 15 and 35 minutes into the lecture period. For some questions, the students performed collaborative learning by worked in pairs and for others they worked alone. The instructor provided the students with the correct answer following the submission of their answers. The majority of the questions were set a level that most students could answer relatively easily.

The effectiveness of the method introduced in 2011 was assessed in three ways (shown in Table 2). Of these, two (approaches 1 and 3, table 2) were compared between 2009 and 2011. Additional information regarding each approach is provided below.

Table 2. The three approaches used to test the success of the introduced pedagogical approach.

Assessment Approaches	Information Gained
1. Course Learning ObjectiveS Evaluation (CLOSE) forms	Student opinion
2. Surveys at the beginning and end of the semester	Changes in student opinion
3. Analysis of student performance relating to the CLOs	Actual student achievement

 

The CLOSE forms are designed to obtain information on student opinions relating to each CLO.  The CLOSE forms were distributed and completed by the students in the last lecture period. For each CLO, the students provide a rating for four questions. Question 1 relates to their opinion before they took the course and questions 2, 3 and 4 relate to their opinions at the end of the course. The four questions were as follows:

• Indicate your level of understanding of this CLO before you took this course
• Indicate you level of understanding of this CLO at the conclusion of this course
• Indicate how confident you would feel explaining what you have learned to others
• Indicate how confident  you would feel using information based on this CLO in a job

The student rating system for the CLOSE forms changed from 2009 to 2011 from a 4 point scale to a 5 point scale. The higher the score, the higher the confidence level (e.g. 0 = not confident to 5 = very confident, in 2011).

To account for the change between years and to enable a comparison, the data from both years were normalized.

The second assessment method involved a short survey, given to the students at the beginning and the end of the class (only in 2011). The survey included questions to obtain student opinions on three areas, as follows 1) in class assessment, 2) use of response pads, and 3) CLOs. The surveys were collected and stored until grades were submitted.

The third assessment method involved an analysis of student performance on homework and exam questions relating to each CLO. The Civil and Environmental Engineering (CEE) department uses one guideline to relate student performance to each CLO. Each CLO is considered to be achieved if more than 75% of the students obtain more than 75% of the available points. Data collected in 2009 and 2011 were compared to determine if the introduced pedagogical approach improved student performance in CLOs achievement. Between 1 and 5 exam or homework questions were used to generate these data for each CLO.

The 2009 and 2011 CLOSE data were summarized and are illustrated below (Figure 1 A and B). In 2009, for the majority of CLOs a clear increase can be seen between the question 1 (understanding before the course) and questions 2, 3 and 4 (understanding after the course) (Figure 1A). The same trend can be seen for 2011(Figure 1B). The only CLO showing limited improvement for before and after the course is CLO 3 (to perform elementary chemical calculations). In 2009, there was only a slight increase and in 2011 there was no increase. Interestingly, when the data are compared between years, no change is seen, indicating the new pedagogical approach had no effect on student opinion of their own achievement of each CLO. This trend could potentially be explained by the already high rating of questions 2, 3 and 4 in 2009, resulting in limited room for improvement (ratings were almost saturated) in 2011.

The pre- and post- surveys provided additional data on student opinions. In 2011, CE 280 students completed a survey at the beginning and at the end of the semester in 2011. From 77 students, 39 students completed both surveys and also signed a consent form. As discussed above, the surveys involved questions relating to three areas. The results are summarized in Figure 2.   

Both surveys indicated positive opinions concerning the use of in class assessment for improved student learning. As expected, the students indicated a strong disagreement with only taking notes and not practicing examples. When the data were compared using two tailed t-tests, no significant difference was found between the two surveys for the section focusing on in class assessment. Both surveys indicated the students were not concerned about using clickers in class (almost all had used clickers in a previous class). Also, the surveys provided data suggesting that the students believed clickers provided an educational benefit. In addition, both surveys indicated the students enjoyed and were comfortable using clickers, and that the devices helped in learning the material. The final section of the surveys provided data on student opinions on CLOs. These data indicated the students were already aware of the importance of CLOs. 

The pre- and post- survey data were compared (two-tailed t-test). For the three sections, there was no significant difference (α = 0.05) in student answers between the two surveys, except for one question.  Indicating that, in general, many opinions did not change as a result of the pedagogical approach used. A significant difference was found for the question “when I study, I consider the CLOs” (p value = 6 X 10^-6, α = 0.05). As would be expected, the responses in 2011 were higher than those in 2009. Overall, the surveys provided valuable data on student opinions on in class assessment, the use of clickers and the importance of CLOs. Perhaps the most important message to be derived from both surveys is the positive student opinion on in class assessment and the use of clickers

The final assessment approach to test the efficiency of the introduced pedagogical method in CE 280 was an examination of student performance relating to material targeted by the CLOs. As discussed above, the CLO was considered to achieved (successful teaching and learning of that CLO), if 75% (or more) of the students achieved > 75% of the available points on questions targeting each CLO. The data collected from 2009 and 2011 for each CLO were compared and are shown in Figure 3. For each CLO data from between 1 and 4 questions (exam or homeworks) were included.

Using the 75% threshold, several CLOs were not achieved in 2009 (CLOs 1, 3, 4, 8, 9, 10, 11, 14). In contrast to this, in 2011, only 1 (CLO 14) was not achieved. These data indicate the pedagogical approach introduced in 2011 was successful.

The percentage of students achieving more than 75% of the available points for each question was higher in 2011 for all CLOs except for one (CLO 13). Unfortunately, many CLOs involved only one data point, limiting any statistical analysis. Standard deviations were calculated and t-tests were performed on CLOs with enough data points. Only one (CLO 4) showed a significant difference between years (p value = 0.045, α = 0.05, two-tail).

These data on student performance provide compelling evidence that the approach was successful in improving student learning relating to the defined CLOs.

Conclusion

In 2011, in- class exercises were designed and included in CE 280 to improve student learning. The exercises were targeted to ABET CLOs as well as additional material covered in this class. The comparison of the 2009 and 2011 CLOSE forms indicated limited increase in student opinion of CLO achievement in 2011 (however, student ratings were already high in 2009). Survey data at the beginning and end of the semester provided evidence of strong positive opinions on using clickers and on using in class assessment. The pre- and post- survey data suggested student opinions did not change a result of taking the class. However, opinions did change over the semester on one issue, that being they will be more likely to consider CLOs when studying for exams. Finally, and most importantly, a comparison of 2009 and 2011 ABET evidence data illustrated a higher percentage of the students achieved each CLO in 2011. In other words, student learning improved as a result of the pedagogical approach introduced in 2011. The instructor intends to continue to use this approach.

Acknowledgements

The authors thank participating CE 280 students, Dr. DeZure, Cindi Leverich (the Office of Faculty and Organizational Development), Dr. Neeraj Buch (instruction on clickers, support letter, agreement on release time), Dr. Ron Harichandran and Dr. Thomas Voice (support letters) and the 2011-2012 Lilly Teaching Fellow cohort. This project was funded by a grant awarded to Dr. Cupples as part of the Lilly Teaching Fellowship Program at Michigan State University.