BY Katie Kim, URC Intern Fall 2023
At the URC we are interested in student and faculty perspectives on generative AI. To explore how it is affecting our faculty we conducted a study in Fall 2023. During this process, I engaged in an intriguing experiment: after analyzing the AI Focus Group meeting notes myself, I tasked generative AI with the same analysis to compare the outcomes. This experiment was designed to examine how the AI's interpretation and insights might differ from a human perspective. Let's examine various aspects of these differences.
Tagging Approach
ChatGPT created broad thematic tags, capturing general concepts such as "AI Impact on Teaching" and "Library System Enhancement with AI." This approach aimed at providing a high-level overview, capturing key themes like the impact of AI on pedagogy and research integrity. Conversely, the researcher’s approach used granular, detailed tags. For example, tags like "AI Engagement: Limited" and "Preserving Traditional Education Methods" offer a more nuanced view of individual opinions and concerns. The human review allowed for a deeper exploration of the diverse opinions and experiences shared during the meeting. By focusing on the intricate details of each discussion point, the researcher's approach provided a comprehensive understanding of the various perspectives on AI's role in academia, highlighting the complexity of faculty and student views.
Participant-Specific Insights
The researcher’s analysis stands out for its focus on individual contributions. Each faculty member's perspective is distinctly identified and tagged, allowing for a clear mapping of viewpoints and concerns. This detailed approach contrasts with the AI-generated summary, which synthesized the collective input into broader themes without specifically attributing points to individuals. This distinction is crucial for understanding the diversity of opinions and identifying specific areas of consensus or disagreement among faculty members.
Emphasis on Specific Themes
The researcher’s approach highlighted specific themes with greater clarity, such as ethical considerations, the balance between AI and traditional teaching methods, and concerns about academic integrity. Each individual tag acted as a representative element, reflecting the diverse themes and conversations of the larger meeting, allowing for a segmented and detailed understanding of these themes. The AI-generated approach, while touching upon these aspects, presented them in a more integrated and holistic manner, focusing on the interplay between these themes in the broader context of AI in education.
Conclusion
1) Advantages
AI's ability to create broad thematic tags provides a high-level overview essential in understanding general concepts such as the impact of AI on teaching and research. This capability is particularly useful in quickly synthesizing large volumes of data, offering a holistic perspective that can be invaluable in identifying overarching themes and trends in academic discussions.
2) Disadvantages
However, AI's approach has limitations, particularly in capturing the nuanced views and detailed insights that human analysis can provide. AI may struggle with understanding context and the subtleties within specific thematic elements, such as ethical considerations or the balance between AI and traditional teaching methods. This lack of depth can lead to oversimplified interpretations of complex academic discussions.
3) Considerations
Given these strengths and weaknesses, it is essential to use AI as a complementary tool alongside human analysis, especially for initial data processing and identifying broad themes. For a comprehensive and accurate analysis, careful supervision and integration of human insights are crucial. This approach ensures that the depth and context often missed by AI are adequately addressed, leading to a more balanced and nuanced understanding of the academic discourse.