Saturday, April 13, 2013

Graduate Teaching Community Spring 2013 Workshop Series on Out-of-Class Assessments


Hey GTCers!

To those of you who are joining us for the first time this quarter: welcome! And a warm welcome back to all our returning members as well.

During our first meeting this past Thursday, we discussed both the main ideas of this quarter’s theme, “Out-of Class Assessment,” as well as some of the topics we think would be fruitful areas to further explore as we move through the spring.

Some of the first questions we addressed were the most obvious: What is out-of-class assesment? What are some of the things we need to consider when constructing or grading these kinds of assignments?

To help us explore this topic, we looked at a book entitled Effective Grading: A Tool for Learning and Assessment. This text, written by Barbara E. Walvoord and Virginia Johnson Anderson, explores the absolute basic elements instructors must master in order to craft the most productive course assignments.

The authors cited a 1993 report by George Kuh on the effectiveness of out-of-class learning in the university setting which summarizes the research on the contributions of out-of-class experiences to valued outcomes of postsecondary education, including:

(a) cognitive complexity (e.g., critical thinking, intellectual flexibility, reflective judgment);
(b) knowledge acquisition and application;
(c) humanitarianism (e.g., interest in the welfare of others);
(d) interpersonal and intrapersonal competence (e.g., self-confidence, identity, ability to relate to others); and
(e) practical competence (e.g., decision making, vocational preparation)


Out-of-class learning, which we (as teachers) structure through assignments, relates to the larger and more complex whole of grading. Grading is a content-dependent process that serves multiple roles:

Evaluation. The grading process should produce a valid, fair, and trust-worthy judgment about the quality of each student’s work

Communication. The grade itself is a communication to the student, as well as to employers, graduate schools ,and others. The grading process also spurs communication between faculty and students, among faculty colleagues, and between institutions and their constituents.

Motivation. Grading affects how students study, what they focus on, how much time they spend, and how involved they become in the course. Thus, it is a powerful part of the motivational structure of the course.

Organization. A grade on a test or assignment helps to mark transitions, bring closure, and focus effort for both students nad teachers.

Faculty and student reflection. The grading process can yield rich information about what students are learning collectively and can serve as the first step in systematic assessment and information-driven teaching.

Support for larger projects. Among the thousands of research grants available for use, there is a significant number whose focus includes a pedagogical aspect. Questions regarding a professor’s ability to relate his/her research to pedagogical concerns can be answered with greater authority when strong data on student performance in his/her classroom can be provided.


Once we’d determined the many considerations and consequences that arise from all form of assessment, we then started brainstorming some questions related to grading and out-of-class work:

How can I construct good assignments?

How can I foster healthy motivation around grades? How should I respond to the student who asks, ‘What do I need to do to get an ‘A’ [or a ‘C’]?’

How can I establish criteria and standards for student work? Should effort and improvement count? Should I grade on the curve? How should I handle grammar and punctuation? How can I fairly grade students who enter with a wide range of skills and preparation?

How can I guide students’ learning process in the most effective way?

How can I communicate effectively with students about their grades? Which kinds of comments and feedback are most useful? How can I help my students without doing their work for them?

How can I analyze the factors that are influencing learning outside the classroom? How might their environment impact their interaction with the work I assign?

What are the principles of good practice in managing the grading process? How can I handle the workload and make grading time-efficient?

How can what I learn through the grading process help me improve my teaching?

How can we use students’ out-of-classroom work to evaluate learning in an entire degree program or in general education?

How do we assess the highest kinds of learning, such as originality, global perspective, or ethical decision making? Will assessment force us to ‘dumb down’ what we teach?

Do assignments (and/or their subsequent assessment) have to be ‘objective’?

By the time we’d finished our brainstorming session, our meeting time had come to a close. But armed with a litany of topics to explore, as well as a list of potential guest speakers, we felt very excited about the topic at hand!

We have an exciting quarter ahead of us, and we greatly look forward to all of the insights and explorations yet to come. Keep your eyes peeled - we’ll be sending out more information on each week’s workshop topic as it takes shape. As always, thank you for your interest in the GTC - we hope to see you there!

All the best,

Sarah and Donnelly
2012-2013 GTC Coordinators

Contact Information:
Sarah Messbauer: smessbauer@ucdavis.edu
Donnelly West: donwest@ucdavis.edu



For more information on the theme for this quarter’s workshop series, check out the following:

Astin, A.W. (1993). What matters in college? four critical years revisited. San Francisco, CA: Jossey-Bass.

An economic case for high school reform (Editorial). (2007, November 1). Minneapolis Star Tribune. Available: www.startribune.com/opinion/editorials/11148976.html.

Guskey, T. R. (2000). Grading policies that work against standards … and how to fix them. NASSP Bulletin, 84(620), 20–29.

Guskey, T. R., & Bailey, J. M. (2001). Developing grading and reporting systems for student learning. Thousand Oaks, CA: Corwin.

Khu, G. D. (1993). In their own words: What students learn outside the class. American Educational Research Journal, 30, 277-304.

Khu, G. D., Douglas, K. B., Lund, J. P., & Ramin-Gyurnek, J. (1994). Student learning outside the classroom: Transcending artificial boundaries. ASHE-ERIC Higher Education Report, 8, 1-160.

Liu, S., & Parmley, K. (2005). An analysis of student out-of-class experiences and self-reported general-education learning outcomes. New Paltz, NY: SUNY New Paltz, Office of Institutional Research.

Marzano, R. J. (2000). Transforming classroom grading. Alexandria, VA: ASCD.

Marzano, R. J. (2007). The art and science of teaching: A comprehensive framework for effective instruction. Alexandria, VA: ASCD.

O'Connor, K. (2007). A repair kit for grading: 15 fixes for broken grades. Portland, OR: Educational Testing Service.

Pace, R. C. (1990). The undergraduate: A Report of their activities and process in college in the 1980s. Los Angels, CA: University of California, Center for the Study of Evaluation.

Price, L. (1993). Characteristics of early student dropouts at allegany community college and recommendations for early intervention. Cumberland, MD: Allegany Community Coll.

Reeves, D. B. (2004). The case against zero. Phi Delta Kappan, 86(4), 324–325.

Reeves, D. B. (2006). Leading to change: Preventing 1,000 failures. Educational Leadership, 64(3), 88–89.

Terenzini, P. T., & Spring, L. (1995). Influences affecting the development of students’ critical thinking skills. Research in Higher Education, 36, 23-40.


The Ultimate Contract: Syllabus Design


Hi GTCers!

How many times have you heard the phrase, “The syllabus is a contract”? For most of us, that concept runs like a mantra though our minds. But why? What is it about this seemingly innocuous document that causes so much strife and heartache for students and teachers alike?

During our eighth workshop of our “Course Design” series, we engaged with a presentation by Avi Jones entitled “The Ultimate Contract: Syllabus Design”. Syllabi, as we all know, are some of the most important and fundamental documents we create, and often set the stage for larger trends of success or failure in the classroom. Given the importance of such a document, it only makes sense for teachers considering course design to give some serious thought to what syllabi are and how they work.

Avi started off asking a basic question: what’s in a syllabus? Some of the possible information we thought up included:

            Lecture schedules
            Contact info
            Exam dates
            Course Overviews
            Expectations
            Required Materials
            Resources/Accessibility Statements
            Grade Breakdowns

We also gave some time to a discussion of course philosophies - the reasons why professors teach a class, and how they think it contributes to their students’ educational goals. On this point, Avi recommended that we all review the ideas upheld by the UCD Principles of Community statement. “It’s one of those things that nobody talks about,” Avi said, “but it’s something that everyone should know.” Understnading the POC - and taking the time to consider where your course aligns with those statements - can really give you an edge on the job market. Why? Because hiring decisions aren’t made by departments alone. Administrators are the ones who ultimately have to sign off; knowing administrative policies and being able to articulate them demonstrates an interest in the institution as a whole, something most administrators view as hugely attractive. Considering the POC, then, has benefits that stretch well beyond course design.

We also spent a few moments going over what not to list on a syllabus. This included personal phone numbers, facebook profiles, home addresses, or anything else that might lead to uncomfortable situations down the road. While most of us strive to be open an accessible to our students, maintaining professional distance is always something we need to keep in mind. Creating that atmosphere of authority is essential for classroom management, and it begins with the syllabus.

Following this discussion, we did a short activity that asked us to look at a real course syllabus and consider the following prompt: Based on your own experiences as an educator, critically assess one of the four course goals below with a partner. Are they realistic, measureable, or plausible?


This was followed immediately by a second prompt: Based on the provided syllabus, determine what you as a student in this course are responsible for.

Dissecting a syllabus with these questions in mind revealed several interesting perspectives. The first was related to course goals - not everyone includes them when designing a syllabus, but as we learned, students use syllabi to figure out what those goals might be. It often plays a major role in whether or not students who are unsure about taking the class will eventually drop. Even if we don’t explicitly include a statement of goals, then, instructors should still make sure their intentions are clear. As Avi pointed out, students have trouble questioning the wording of goals, especially early on in the quarter. But the misunderstandings and misconceptions that can result from lingering questions can lead to trouble down the road. It’s important for all of us to consider what elements of our syllabi might be unclear, why, and how to fix them. ESL teachers, for example, can address confusion in their syllabi by considering other ways of demonstrating the information. Including visuals (flow charts/timelines/etc.) might help clarify what they expect from students.

This discussion about clarity led to an often-asked question to be posed: Is it at all worthwhile to make things a touch confusing in order to force cohesion among the students? To make them think critically? To make them actively engage with their education, advocating for what they need? The response was: not very. Why? Because the responses from our students will almost always be equally bad. While it’s important to think positively about the capabilities and skills of our students, we also need to accept the fact that the skills required to overcome that kind of planned confusion - critical thinking and intellectual advocacy being two of them - are often skills that they develop during the course of class, not before.  

Another question that came up addressed the sometimes problematic issue of instructor syllabi. If the professors we teach for are ambivalent about the confusing nature of their syllabi (or their teaching style more generally), how should we as TAs address this? Should we situate the professor as a drill sergeant (the common enemy) and ourselves as the buffer between? Lots of opinions were offered on this question; some of us create our own, clearer syllabi. Others take extra time to explain the professor’s meaning during sections. Others took a more hands-off approach. But despite the differing opinions, each of us agreed that the primary goal should be to create a clear, coherent and concise syllabus at the start.

Avi’s workshop helped to demonstrate the many, many ways that the syllabus serves as an essential tool for instructors, TAs, and students. If that tool isn’t what we need it to be in order to achieve the goals of our course, the result is going to be an impossibly difficult situation. As Avi so colorfully phrased it, “We just can’t herd cats into a sprinkler with a blowtorch”. Now that’s something to think about!

For more information on syllabus design, check out the following resources:

Website on Syllabi from the Center for Research on Learning and Teaching at the University of Michigan

A huge catalog of resources related to syllabus design from the Center for Excellence in Teaching at the University of Southern California

(Posted by Sarah Messbauer)

Say What? Nonverbal Modes of Participation


Hey GTC!

In our seventh workshop of the quarter, we featured a presentation by Megan Saucke entitled, “Say What? - Nonverbal Modes of Participation”. The workshop addressed the reasons why students don’t speak in the classroom - many of which are related to issues of privilege. Understanding participation, then, requires understanding the power dynamics in the class.  With this understanding in hand, we as teachers can better see why offering non-verbal modes of participation are important.

First, we discussed why it is important to participate verbally in class. Verbal participation provides good practice articulating concepts; it engages active learning instead of passive learning; and it reinforces a practical life skill. Communicating verbally with someone is something all our students will have to do at some point, especially while on the job. Communicating well in a group, accurately synthesizing information, and creatively presenting work to an audience are all top skills that employers look for in potential employees. Participating in class discussion also provides an element of empowerment and authority that can help students build their personal self-confidence.

All of the items listed above are legitimate benefits from verbal participation in the classroom. But these same items also feed directly into many of the reasons that students don’t speak in class. A few of the reasons we brainstormed were:

            Shyness
            Classroom culture (International Students)
            Unprepared
            Processing speed
            Language barrier
            Anxiety over ‘rightness’
            Afraid their questions are dumb
            Privileges masculine modes of communication (gender issues)
            Middle class personalities (see Bordieu), class has a heavy influence
            American cultural norms
            Ability/Alternative ability
            Access - high school/college environments
Access to technology, like computers, etc., also considering availability, like comp labs that close overnight

While some of the issues listed above (like unpreparedness) are problems that cannot be completely fixed by instructors, what Megan emphasized was that classrooms generally privilege more extroverted personalities. Lots of people like to think through their answers before they speak - but some people think while they speak. This puts the first group at a distinct disadvantage, especially in high-enrollment classes where the potential for TAs or professors to get to know their students is limited.

The issues surrounding classroom culture are also some of the most significant and challenging for teachers to address. Some cultures see asking questions in the classroom as disrespectful or embarrassing. Students from these cultures don’t want to disagree with students or the professor out of the fear that they will offend them. Some students come from environments where answering questions is viewed as showing off. Others are used to more direct-recall questions than the debating style that many instructors try to foster.

Regardless of the reason, however, the simple fact is that verbal participation is a difficult obstacle to overcome for many of our students, and can often have a negative impact on their classroom experience.

But the good news is that providing opportunities for non-verbal students to increase their active participation in the classroom often requires little extra effort on the part of the instructor. The list below documents just a few of the many opportunities teachers can provide for their more introverted students:

            End-of-class papers
            Hand raising (“Raise your hand if you agree, if you disagree, etc.”)
            Forum posts
            Chat room
            Google doc collaboration

Encouraging students to take advantage of these simple techniques can have a really profound impact on the student experience.

It’s important to note that although it’s important to consider the backgrounds and experiences of our students, that doesn’t mean we need to completely exclude verbal participation from the classroom. The benefits listed above are still very real, and college is the perfect place for students unfamiliar with verbal participation to develop these skills. In the last portion of the workshop, Megan asked us to brainstorm a list of ways to gradually encourage quieter students to speak up. The ideas we listed were:

            Think-pair-share
            Write answer first
            No one can answer for 30 seconds
            Write on board, then talk about it
            Clarify expectations, or set expectations at the start
Especially in regard to time commitment outside class, how to participate, classroom ground rules, tell them how to play “the college game”
Don’t bait-and-switch
ask for the answer to one question, then ask that same student a follow-up question that they weren’t prepared to answer
            Limit how much a person can dominate a conversation
            Reinforce rules about interrupting others
            Figure out ways to reward or validate correct answers
            Learn names
            Invitations to talk one-on-one (extra credit possibilities)
            Be clear about creating a safe space where all questions are respected
            When quiet folks DO ask questions, be encouraging
“Wow, that’s an awesome question”
                        Give that question the time it deserves

Understanding the factors that influence students’ verbal or non-verbal participation, combined with the simple strategies listed above, can create a classroom that fosters the exact kind of growth and development the college experience is meant to foster.

(Posted by Sarah Messbauer)

"Test"ing Balance – Effective Evaluations for Students and Teachers


Howdy, GTCers!

Exams are just a way to assign grades right? Maybe, but effective assessment can provide instructors with valuable feedback about student learning and provide students with valuable feedback about how well they understand the class material. In the fourth workshop of the GTC’s “Course Design” series, facilitators Amanda Schrager and Matt Dumlao wanted to explore different types of testing, Bloom's Taxonomy/different types of questions, and how often to give assessments.

High stakes and low stakes testing
Everyone is familiar with the dreaded midterm, term paper, and final, but there are actually many different ways to assess learning. In addition to the traditional high stakes assessments just mentioned, that account for large percentages of a student's grade, there are many ways to test student learning with low stakes assessment. These low stakes assessments account for only a small part of the overall grade (if any at all), and serve to provide feedback for students (how well do I understand the material?) and instructors (does the content and pace make sense for this set of students?) rather than simply determine overall grades. Some of our favorite low stakes assessments include clicker questions, response questions, polls, pop quizzes, and homework assignments. Can you think of a good way to include (or improve upon) low stakes assessment in your classroom?

Bloom's taxonomy
Giving tests is more than just asking random questions about the course material. Different types of questions assess learning differently. In 1956, Benjamin Bloom first proposed a classification of learning objectives now known as Bloom's taxonomy. The ideas have been modified since their introduction, but at the core is a pyramid categorizing different types of questions.




In order to give effective assessments, think about your learning goals and what type of information you are trying to get across to your students. For example, if you are teaching a beginning Spanish class, you probably shouldn’t be asking your students to compose essays or respond to complex questions. In an introductory class, you're most likely trying to convey the basics of the subject and get your student familiar with the vocabulary of the field. Keep to the bottom of the pyramid for these classes, you want to assess knowledge and comprehension of the basics. Even your best students will likely struggle answering analysis questions. Equally, if you are teaching a specialized upper division class on plant molecular biology, your students should be coming to class with a strong base of general biology, genetics, etc, allowing you to teach more difficult concepts and ask students to think critically and address question at the application level and above. Remember, questions from the top and bottom aren't better or worse than those from the other half… write a test that aligns with your learning goals and assesses at a level appropriate for the course. Furthermore, the type of test (e.g., scantron-based multiple choice, short essay, etc.) might assess different levels of Bloom’s taxonomy more or less effectively. Multiple-choice tests are excellent for knowledge and comprehension level questions, but higher order cognitive skills may not be tested adequately.

Time. What's it worth?
Giving five midterms in a 10-week quarter is a great idea right? What about a single midterm for an 18-week semester? Timing of low and high stakes assessments can be just as important as the questions you ask. Over testing can be overwhelming your students and TAs/graders. You certainly aren't going to create a positive environment for learning if your students are constantly stressed about exams and your TAs are grumpy from all the extra grading. On the other hand, using just one or two high stakes assessments will make learning from mistakes nearly impossible for the students and quite difficult for them to figure out how much of the course content they truly understand. Finding the right balance of high and low stakes assessments can provide your students with the feedback they need and a fair determination of overall grade without causing your graders to revolt. How often do you assess your students? Do you include a mix of low and high stakes assessment?

References
In preparation for this workshop, Amanda and Matt came across several great resources they’d like to share:
1.              Wikipedia.org has a great entry on Bloom’s taxonomy. (We know, citing Wikipedia is a faux pas, but hey, it’s informative.)
2.              In addition to providing the beautiful pyramid image for the blog post, this website also has a great section on Bloom’s taxonomy.  
3.              There are also many great journal articles on teaching. Here are two of their favorites:
                        Allen, D., and Tanner, K. (2002). Approaches to Cell Biology       Teaching: Questions about Questions. Cell Biology Education 1, 63-67.
                        Crowe, A., Dirks, C., and Wenderoth, M.P. (2008). Biology in Bloom:   Implementing Bloom's Taxonomy to Enhance Student Learning in             Biology. CBE-Life Sciences Education 7, 368-381.

(Posted by Amanda Schrager and Matt Dumlao)