Design for Equity Lab: Advancing an Adaptive Equity-Oriented Pedagogy for STEM in Higher Education

A Peacemaking Project by Judy N.

What is the injustice we are solving?

The United States faces challenges competing in the global economy due to a lack of diverse STEM graduates (U.S. Department of Education, 2017). Simultaneously, universities experience difficulties recruiting and retaining STEM students, especially underrepresented minorities (URM). According to Seymour and Hewitt (1997), 90% of students leaving STEM cited poor teaching as one of their primary concerns. The Higher Education Research Institute found that URM students who aspire to major in STEM as a freshmen have a substantially lower likelihood of completing such a degree within five years than their white and Asian American peers (Hurtado, Eagan, & Chang, 2010). First-year STEM majors are also more likely to withdraw from college than their non-STEM counterparts and are more likely to change fields of study before graduating. Recognizing that faculty teaching can have a direct impact on student engagement and learning in STEM courses (Seymour & Hewitt, 1997), I examine how an adaptive equity-oriented pedagogy (AEP) impacts student achievement, psychosocial outcomes, and retention for all students, especially underrepresented minority (URM) students (Phuong et al., 2017). AEP instructors use ungraded, weekly formative assessments to adjust their teaching to address diverse students’ learning needs (i.e., their strengths, interests, and areas for growth). Preliminary results suggest that, compared to control conditions, UCB students learning through AEP scored, on average, 12% higher on the final and outperformed control groups by a full letter grade in the course (Phuong et al., 2017). Students experiencing AEP demonstrated greater improvements in positive psychosocial outcomes (e.g., motivation, reduced stereotype threat, growth mindset, self-efficacy, inclusion). These effects endured when controlling for GPA and intersectional identities (e.g., race, gender, sexual orientation, income, disability, immigration status). Based on the successes of these studies, further graduate research will allow me to expand randomized control trials (RCTs) with larger samples that examine student outcomes in different sections of STEM courses. This research will contribute a methodology for instructors to evaluate and refine equity-oriented curricula to improve URM students’ success, which directly addresses the U.S Department of Education’s priority of improving STEM teaching and completion rates for URM in universities. With stakeholders, I aim to develop a national model for equity-oriented STEM pedagogy based on the case of UC Berkeley as a large, diverse public university. We will use evidence from rigorous RCTs to assist with faculty member buy-in, a long-standing barrier to education reform. We seek to advance a pedagogy that attracts, supports, and enables URM students to thrive and grow as STEM scholars and professionals. Goal by May 31st: In the next two months, we plan to conduct an experimental pilot study to test the effectiveness of using ipads as a part of the adaptive equity-oriented pedagogy curriculum compared to existing technologies. The study will analyze two groups of students randomly selected into two different conditions. In one condition, a teacher will lead a lesson on statistics with active learning activities that students can work on in groups using an ipad. In the other condition, the same teacher will lead the statistics lesson with active learning strategies using traditional powerpoint slides and worksheets. Researchers will observe student engagement, collaboration, and dialogue during both conditions. Pre- and post-surveys will be given to students before and after the study to assess their knowledge and interests of the content material of the lesson. The purpose of this study is to assess whether the ipad technology produces significant differences in gains for student engagement, collaboration, and learning compared to existing technologies used in the classrooms. If there are statistically significant differences with large effect sizes, then it might be useful to scale the ipad technology in future iterations studying adaptive equity-oriented pedagogy to improve equity in STEM students' learning. References Hurtado, S., Eagan, K., & Chang, M. (2010). Degrees of success: Bachelor’s degree completion rates among initial STEM majors. Higher Education Research Institute at UCLA, January. Phuong, A. E., Nguyen, J. & Marie, D. (2017b). Evaluating an adaptive equity-oriented pedagogy: A study of its impacts in higher education. The Journal of Effective Teaching. 17(2), 5-44. Seymour, E., & Hewitt, N. (1997). Talking About Leaving: Why Undergraduates Leave the Sciences. Westview Press. United States Department of Education. (2017). Science, Technology, Engineering and Math: Education for Global Leadership. U.S Department of Education. Retrieved from https://www.ed.gov/stem.

Our Compassionate Solution:

To solve this injustice of low STEM retention rates among underrepresented students in higher education
we will address the lack of support that faculty have with implementing high-impact inclusive teaching practices
by equipping and supporting instructors with applying adaptive equity-oriented pedagogies that have been shown to improve academic achievement for all students, especially URM students

Our Project Plan:

Goals

  • gain empathy by collaborating and listening to instructor and student needs
  • equip instructors with adaptive equity-oriented pedagogies
  • support instructors with applying adaptive equity-oriented pedagogies
  • test ipad in pilot studies by May 31st

We will increase my / our compassion by...

working collaboratively with instructors in higher education who are often seen as the problem. We will listen emphatically to instructors' and students' needs to identify optimal ways to help them succeed with improving teaching and learning in STEM classrooms.

How will you show courage?

Common conceptions suggest that higher education instructors are resistant to change, especially since they feel like they are the expert in terms of content knowledge and communicating that content knowledge. I am working against and within a research 1 university that prioritizes research over teaching when promoting and retaining faculty. Therefore, I demonstrate courage by seeking to overcome the organizational constraints of UC Berkeley. I will achieve this goal by researching and advancing a practical solution (i.e., the adaptive equity-oriented pedagogy) that many instructors can implement at a R1 university. Goal by May 31st: In the next two months, we plan to conduct an experimental pilot study to test the effectiveness of using ipads as a part of the adaptive equity-oriented pedagogy curriculum compared to existing technologies. The study will analyze two groups of students randomly selected into two different conditions. In one condition, a teacher will lead a lesson on statistics with active learning activities that students can work on in groups using an ipad. In the other condition, the same teacher will lead the statistics lesson with active learning strategies using traditional powerpoint slides and worksheets. Researchers will observe student engagement, collaboration, and dialogue during both conditions. Pre- and post-surveys will be given to students before and after the study to assess their knowledge and interests of the content material of the lesson. The purpose of this study is to assess whether the ipad technology produces significant differences in gains for student engagement, collaboration, and learning compared to existing technologies used in the classrooms. If there are statistically significant differences with large effect sizes, then it might be useful to scale the ipad technology in future iterations studying adaptive equity-oriented pedagogy to improve equity in STEM students' learning.

How will you collaborate with others?

I will collaborate with instructors each week by collecting student assessment, course feedback form, and observation data. I will collaborate with researchers, instructors, and students to study the effects of AEP ipads and non-ipads interventions on students' academic and psychosocial outcomes.

How will you know you are moving in the right direction? (What are specific ways you can measure your impact?)

We will know if we are moving in the right direction if we achieve our goals. To check on the progress towards our goals, we will analyze our data each week to identify if students are improving their academic achievement. We will follow-up the quantitative analyses with qualitative interviews of students. The purpose of this study is to assess whether the ipad technology produces significant differences in gains for student engagement, collaboration, and learning compared to existing technologies used in the classrooms. If there are statistically significant differences with large effect sizes, then it might be useful to scale the ipad technology in future iterations studying adaptive equity-oriented pedagogy to improve equity in STEM students' learning.

Key Steps

  • collect 1) pre-semester and post-semester interviews; 2) weekly student surveys, formative assessments, and observation notes; and 3) midterm and final exams
  • weekly research meetings to analyze data and provide reports to inform instructors’ use of adaptive equity-oriented pedagogy
  • Use Stata to run statistical analyses (e.g., correlations, t-tests, F-tests, Fisher’s exact test, regressions, effect sizes, chi-square tests, cluster analysis, etc.)
  • meet and will continue meeting each week for 6 hours to score, input, and analyze data
  • assessing bi-monthly research reports on effective teaching
  • collaborate with UC Berkeley to survey students and track their GPA, retention, and pathways in subsequent courses
  • use findings to improve campus policies, resourceful services, faculty development, and learning outcomes for over 18,000 UCB STEM students, instructors, and staff.
  • test ipad in pilot studies by May 31st

Reflection:

How did the project deepen your team's understanding of the injustice?

The injustice that my project sought to understand was inequities within undergraduate STEM courses in higher education. With the funds to introduce a new type of technology, we were able to deepen our understanding of whether access to an ipad device can facilitate or hinder underrepresented students’ learning gains within the classroom. Our previous studies have already shown that adaptive equity-oriented pedagogies and active learning strategies helps improve underrepresented students improve by a full letter grade. Our new study exploring a classroom using an ipad showed greater learning gains for students compared to classrooms using paper or laptop materials. The study assessed students’ learning gains and confidence within different technology conditions after an educational intervention about learning statistics. All study participants were randomly assigned to each condition. The conditions included using an ipad, a laptop, and paper. Participants took a pre-test at the beginning of the class and a post-test at the end of the class. This tests assessed students on interactions, an advanced topic in multivariate regression. Our results showed that the ipad gained the most at 100% from pre- to post-test. The paper condition had a learning gain of 40%, while the laptop condition had a gain of 75%. This suggests that learning gains may actually improve with technology and that lack of access to such technology is an injustice to equitable learning opportunities.

How did your community change as a result of your project?

The community of STEM undergraduate teachers and students realized the impact of technology for underrepresented students’ learning opportunities and equity. In particular, the teachers began to organize in professional teaching communities to discuss how to embed digital technologies within their classroom pedagogy to improve equity. Students also realized the benefits of using technology (e.g., easier collaboration such as google documents and software for mathematical calculations). Teachers and students are our primary stakeholders, however higher education leaders are also an important stakeholder within our community. This research is a starting point that the Division of Equity and Inclusion at UC Berkeley has considered for giving grants and programs that provide access to ipads for low-income, minority, and underrepresented students.

How many people were impacted by your project?

27

Explain how you came up with the number of people impacted by the project?

Since this was a pilot study, we wanted the number of participants to be a reasonable number that is manageable but still offers some power to interpret. We recruited 5 teachers who had a total of 22 students who consented to be a part of the study.

How did your team learn more about the people affected by the injustice?

We learned more about the people affected by the injustice by analyzing the pre- and post-data mentioned in the first question. We also had a pre- and post-survey asking about their confidence with the material. All groups answered the question below regarding confidence about interaction effects before and after the intervention. The response options included (1) not at all confident, (2) slightly confident, (3) neutral, (4) very confident, and (5) completely confident.

What did your team learn?

In the pre-test, 60.9% of students were not at all confident, 21.7% slightly confident, 8.7% neutral, and 8.7% very confident. In the post-test, 22.7% were neutral, 59.1% were very confident, and 18.2% were completely confident. We also observed students during the time they were in the study to note their interactions with peers and the technology.

What challenges did your team overcome?

There were certain challenges in the beginning of the study, such as connecting to wi-fi and using software on the ipad. For example, a group of students were using excel on the ipad but had a learning curve to see how it can efficiently be used on an ipad with a sensitive touchscreen. We had overcome these challenges by connecting to the university wi-fi network. We also adjusted the pilot study agenda by giving time for students to explore and tinker with their technologies.

How have you involved others in designing, carrying out, or expanding this project?

For our pilot study, we engaged in a process of design based teacher action research in which the teachers of the study played an active role in the design, implementation, and analysis. We co-created the study design with the teachers, since they knew their classroom pedagogy the best and how to weave in the ipad technology. The teachers also employed a form of democratic pedagogy which allowed the students’ feedback to play a role in pedagogical decisions. Teachers also helped us implement the research successfully and provide their perspective in analyzing the data results.

What advice would you give to someone starting a peacemaking project?

My advice would be to listen to the needs of the community and make sure your project incorporates community voice. Starting a peacemaking project may seem like a giant and ambitious task, which it is, but it is also not your task alone. Identify all relevant stakeholders and get them involved within your project, so that there is not a singular vision of peace.

Comments:

Adrian B.
28 June 2018 16:57

Hi Judy--this reflection feedback is from Dalton, a member of the Peace First team. Due to confusion with the new platform, it was posted on the old version of your project--I wanted to be sure you got to see it!

I also wanted to add--there are a few questions you may not have seen on the reflection form. Could you fill out those questions so your reflection is complete? Thanks so much!

Best,
Adrian

Hi team!

Congratulations on finishing your peacemaking project! I’m on the Peace First team and wanted to say thank you for your amazing work and for taking time to share about your work.

Below you will find some feedback based on your Reflection, which we hope will help you to celebrate your incredible accomplishment and reflect on how to grow and develop your project in the future:

Wow! I'm so impressed with your tracking and evaluation process. You all have set yourselves up for success by doing such a good and thorough job of work and that's going to really benefit you as you progress your work. Great job!

I also really love how you've been able to connect a lack of technology to a much larger injustices of inequality - very compelling.

Moving forward, I'd love to hear more about how you really collaborate with the people you are helping - are they helping develop some of these programs? What kind of input do they have in this process?

We hope you will stay in touch and keep us up to date as you continue your work to create change!

Best,
Dalton

Andrew P.
28 February 2018 0:06

I am so excited about this!

Judy N.
28 February 2018 0:14

Me too! Thanks so much for your enthusiasm and support!

RaulPF C.
7 March 2018 8:30

Hi Judy,
Thanks so much for sharing your project. We'll be giving you feedback on your project insight soon, and then you can submit a mini-grant application!

In the meantime, check out some other projects that are working on similar issues:
https://www.peacefirst.org/projects/girls-empowerment-stem-initiative
https://www.peacefirst.org/projects/stem-seminars-underprivileged-schools

We encourage folks to connect with people working on similar topics, as networks only strengthen our work--drop by some of these links if you have time and comment on what others are doing, ask questions, and generally get to know people!

Thanks so much for doing this important work. Let us know if you have any questions, and if there are other ways we can support you!

Best,

Raul

Tom B.
8 March 2018 17:07

Hi Judy,

Thank you for sharing your project. I am a member of the Peace First team and I’m here to give you feedback on your compassionate insight.

I am very impressed with your level of analysis for this. You have clearly put a lot of effort in to compiling a clear understanding of (and evidence for) your injustice. It is through analysis or experience that we get an understanding of issues and therefore what needs to be done to address them.

You have specified a particular group that are affected by an injustice, you have also identified a root cause of the injustice and you have proposed an approach to tackle it.

It is all very clear and logical - well done!

I’m really looking forward to seeing the outcome.

Fish S.
6 April 2018 19:00

Hi Judy! Congratulations! Your project has been selected to receive a Peace First Challenge mini-grant. We will be in touch soon with details about this payment.

The mini-grant process is also a space for you to get feedback on your peacemaking project. We hope you will use this feedback to further strengthen your project. Please see the feedback on your mini-grant below

Strengths:
Judy, I am really amazed by this project! The depth of thought and research you've invested in your work is incredible, matched only by your compassion for making sure that every student has equitable access to the education they need. Your compassion for those marginalized by the current system, your courage in working within (and maybe even against) a large institution, and your commitment to collaboration with faculty are exciting -- as is your commitment to listening and collecting feedback. Your plan is thorough and appears sound. We are excited to support this work -- please let us know how we can help!

Things to consider (Opportunities to further strengthen the project)
I am honestly at a loss to identify ways this application could by strengthened, Judy -- your work is deeply compassionate, thoughtful, and well-designed! The only thing is that I hope to hear more about your work to make sure that students of all needs and backgrounds are included as part of the design work.

Congratulations and best of luck with your project!