Since ChatGPT brought a new wave of technology, the Center for Teaching & Learning Innovation has been surveying our university students each semester about their use of generative AI. To understand how students' use of generative AI is changing, a survey was conducted over seven days from June 3rd to June 10th, 2024. The first survey on generative AI usage was conducted in June 2023 with 219 students responding. The second survey in November 2023 had 415 students participate. The third survey, in June, had 423 student responses. In this Teaching Tip, we will look at how students' experiences with generative AI have changed based on the results of three surveys and understand the implications.

1. Were the results provided by generative AI satisfactory?

  Satisfaction with the responses of generative AI not only shows students' preference for generative AI but also indicates whether they will continue to use it. Our university surveyed students' satisfaction with generative AI responses three times, and overall satisfaction has increased. Specifically, the positive responses ("satisfied" and "very satisfied") increased from 59.3% in the first semester of 2023 to 69.2% in the second semester of 2023, an increase of about 10 percentage points. Furthermore, in the first semester of 2024, the combined positive responses increased to 74.9%, a 5.7 percentage point increase compared to the second semester of 2023. Meanwhile, the combined negative responses decreased over time: 13.7% in the first semester of 2023, 10.1% in the second semester of 2023, and 7.1% in the first semester of 2024. The most significant difference in the three surveys was in the "satisfied" responses, which increased from 43.8% of students in the first semester of 2023 to 60.5% in the first semester of 2024. This result suggests that as generative AI technology has improved, the quality of responses provided to students has also increased. Additionally, it can be interpreted that students' skills in crafting prompts to elicit satisfactory answers from generative AI have also improved.

2. Has the Efficiency of Learning Improved with the Use of Generative AI?

  When students were asked if their learning efficiency improved by using generative AI, all three surveys showed that students positively perceived an increase in learning efficiency. The question format differed between the surveys: the first survey used "yes," "no," and "other (write-in)" responses, while the second and third surveys used a 5-point scale. The left image below categorizes the positive responses on the 5-point scale as "yes," negative responses as "no," and neutral responses as "other" to compare the results at a glance.

  Meanwhile, the image below compares the results of the second and third surveys, both using the same 5-point scale. In both surveys, the smallest percentage of students answered "not at all" (2.4% and 0.5% respectively), while the highest percentage answered "yes" (42.7% and 46.6% respectively). Compared to the second survey, fewer students in the third survey felt that learning efficiency decreased, and more students felt that it increased. Over time, more students believed that using generative AI improved their learning efficiency.

3. How Much and How Did Students Use the Results Provided by Generative AI?

  In the second survey  November of last year and the third survey June of this year, students were asked how much they used generative AI's responses and whether they fact-checked those responses. This question was not included in the first survey. Survey results showed that over time, students increasingly used the results provided by generative AI. In both surveys, the highest percentage of students responded that they "used the results a little" (49.2% in the second survey, 54.4% in the third survey). Only a small percentage "hardly used the results" (2.9% and 1.4% respectively). The overall use of generative AI's results increased from 59.1% in the second semester of 2023 to 69.8% in the first semester of 2024, a rise of 10.7%. Additionally, the percentage of students who "mostly used the results" increased from 9.9% to 15.4%, showing a significant increase in the use and perceived value of generative AI responses.

  When asked about the extent to which they fact-checked generative AI's responses, about two-thirds of respondents in both the second and third surveys said they did. Specifically, 66.2% in the second semester of 2023 and 63.8% in the first semester of 2024 positively responded that they fact-checked AI's answers. The percentage of students who fact-checked "most of the necessary information" decreased from 36.1% to 31.4%, while the percentage of those who performed lower levels of fact-checking increased.

4. Are You Worried That Using Generative AI Responses Violates Academic Ethics?

  We examined the changes in responses to the question, "Are you worried that using generative AI responses violates academic ethics?" The response format varied across the first, second, and third surveys. To compare the results, we grouped the 5-point scale responses from the second and third surveys into positive ("yes" and "very much so") and negative ("not at all" and "no") responses. The proportion of students worried that using generative AI responses violates academic ethics showed a gradual increase: 35.2% in the first semester of 2023, 37.6% in the second semester of 2023, and 43.5% in the first semester of 2024. Meanwhile, the proportion of students who were not worried that using generative AI responses violates academic ethics showed a gradual decrease: 60.7% in the first semester of 2023, 37.9% in the second semester of 2023, and 36.2% in the first semester of 2024.

  When looking at the results of the second semester of 2023 and the first semester of 2024 separately, the proportion of "yes" responses showed the biggest difference. In the second semester of 2023, 27.5% of students answered "yes," while in the first semester of 2024, this increased to 33.3%, a rise of 5.8%. Additionally, the proportion of students who answered "neutral" decreased by 4.3%, and those who answered "almost no" decreased by 2.4%. These results suggest that as students continue to use generative AI, their awareness of academic ethics improves.

5.What Do Our Students Expect the University to Guide Them About Regarding Generative AI?

  When comparing the survey results from November 2023 and June 2024, the responses about what students expect the university to guide them on regarding generative AI were similar in terms of ranking and proportion. The most desired information was "how to effectively use generative AI for learning" (about 70%), followed by "practical uses of generative AI in various fields (learning, research, coding, competitions, etc.)" (about 65%). "Ethics of using generative AI" decreased by 7.2% compared to November 2023, but about 30% of students still seek this information.

   In this Teaching Tip, we investigated and analyzed the experiences of university students using generative AI in the first semester of 2024. We compared these findings with survey responses from June and November 2023 to observe the trends and how students' use of generative AI has changed. Overall, satisfaction with generative AI has steadily increased along with learning efficiency, leading to more positive perceptions among students. This indicates that generative AI is becoming a valuable learning tool. However, concerns about learning ethics are also growing. So, while using generative AI in learning is helpful, it is important to continue educating and discussing how to use it ethically. We hope this Teaching Tip helps professors understand students' use of generative AI and aids in planning lessons and assessments.

Sang-eun Lee, Min-young Ku, Ye-jin Kim (2024), How Are Our University's Students' Experiences with Generative AI Changing? (CTL Teaching Tips #42). Seoul: Sungkyunkwan University, Center for Teaching and Learning Innovation.