EDU6978: Week 03: Due 2012-07-15


During my internship year of teaching my primary of way of eliciting evidence of student learning was through discussion in class.  The format of the class was short review of topics related to the Math SAT, then a sample Math section of the SAT, then a review of the questions covered.  I could provide evidence of student learning if I saw students doing better on geometry, algebra, numeracy, or chart and graph questions that we worked in class.

With the information of students either getting problems right during sample tests or being able to describe where they went wrong, I was able to suggest other homework or exercises that students might complete.

Since there was not grade in the class, the students were all preparing to take the SAT on June 3rd.  That was the final summative assessment, if you will.  What I think I could have improved on was the gathering of concrete data that students were getting more answers correct in various subtopics of the Math SAT.  I have spoken with my mentor teacher at length about a more effective use of homework, but I think it is fair to say that homework is not something that students are motivated to do at my school, and the value of pressing the issue was unclear.

I have to admit, too, that I was way more focused on the presentation of material and of the simulating of the testing environment and getting the students to see questions, than whether the students were improving their performance on the test.  An embedded formative assessment model would enable me to tailor my lesson delivery based on student ability to complete lessons or problems.  Also I would enable students to help each other learn and push them to be in charge of their own learning.

I was extremely happy to see students engaging each other on explanations of problems they got right, i.e. to help other students who didn’t get the answers.  However I didn’t harness that information to feed back into what the instruction part of the course was doing.  Also for all the test sample questions we took and talked about, I never kept a running tally of who was getting which questions right or wrong.  That would have enabled me to tailor some instruction appropriately.


STEM Education Quality Framework (Dayton STEM Network, 2011)

  • Really liked this summary.


Framework in some detail


Links to STEM Model Projects and Programs

To be evaluated against the Dayton Regional STEM Education Quality Framework.


Embedded Formative Assessment (Wiliam, 2011)

Chapter 4 Outline (Verbatim from author unless italic)

Eliciting Evidence of Learners’ Achievement

We discovered in chapter 3 the importance of being clear about what we want students to learn. Once we’ve accomplished that, we need to ascertain where the students are in their learning. In many classrooms, the process of eliciting such evidence is done mainly on the fly—teachers almost always plan the instructional activities in which they will engage their students, but they rarely plan in detail how they are going to find out where the students are in their learning. This chapter emphasizes the importance of planning this process and provides guidelines on what makes a good question, as well as some alternatives to questions. The chapter also offers some practical guidance on how to use questions effectively to adjust instruction to meet students’ needs.

Wiliam, Dylan (2011-05-01). Embedded Formative Assessment (Kindle Locations 1487-1492). Ingram Distribution. Kindle Edition.

  • Finding Out What Students Know
    • TIMSS example questions that “seem” the same many times aren’t.


    • Opinions vary on why these two sample questions are so different in success rate.
    • In answering the second question, 39 percent of the students chose B. Since this question was answered correctly by 46 percent of the students, it was answered incorrectly by 54 percent, but 39 percent chose the same incorrect answer.
    • This suggests that students’ errors on this item are not random but systematic.
    • Students seem to misapplying a rule from first experiences in fractions.
    • Smallest fraction=largest denominator (OK).  Largest fraction=smallest denominator (NOT OK).
    • In other words, there is strong evidence that many students who got the first question right got it right for the wrong reason.
    • Teachers need to ask questions better aimed at uncovering misconceptions.
  • Where Do Students’ Ideas Come From?
    • “upside-down triangle”


    • However, some students describe this shape as an upside-down triangle even though they know that the orientation does not matter for the naming of the shape, because they are using vernacular, rather than mathematical, language.
    • In the world outside the mathematics classroom, however, the word square is often used to describe orientation rather than shape…
    • What seems like a misconception is often, and perhaps usually, a perfectly good conception in the wrong place.
    • When a child says, “I spended all my money,” this could be regarded as a misconception, but it makes more sense to regard this as overuse of a general rule.
    • Some people have argued that these unintended conceptions are the result of poor teaching.
    • The key insight here is that children are active in the construction of their own knowledge.  correlation <> causation, trees and wind.
    • The second point is that even if we wanted to, we are unable to control the students’ environments to the extent necessary for unintended conceptions not to arise.
    • 2.3 * 10 <> 2.30.  But 7 * 10 =70!  We could make such a “misconception” less likely to arise by introducing decimals before teaching multiplying single-digit numbers by ten, but that would be ridiculous.
    • Thus, it is essential that teachers explore students’ thinking before assuming that students have understood something.
    • Pair of equations


    • When asked what a and b are, many students respond that the equations can’t be solved.  Skills versus beliefs.
    • The point here is that had the sixteen in the second equation been any other number at all, provided they had the necessary arithmetical skills, students would have solved these equations, and the teacher would, in all likelihood, assume that the class’s learning was on track.
    • Questions that give us insight into student learning are not easy to generate and often do not look like traditional test questions. Indeed, to some teachers, they appear unfair.  Example


    • The question is perceived as unfair because students “know” that in answering test questions, you have to do some work, so it must be possible to simplify this expression; otherwise, the teacher wouldn’t have asked the question—after all, you don’t get points in a test for doing nothing.
    • Example, “which is larger”


    • The fact that this item is seen as a trick question shows how deeply ingrained into our practice is the idea that assessment should allow us to sort, rank, and grade students, rather than inform the teacher what needs to be done next.
    • Example, what is between water molecules?  “water”.
    • Questions that provide a window into students’ thinking are not easy to generate, but they are crucially important if we are to improve the quality of students’ learning.
    • So the important issue is this: does the teacher find out whether students have understood something when they are still in the class, when there is time to do something about it, or does the teacher only discover this once he looks at the students’ notebooks?
    • As noted previously, questions that give us this window into students’ thinking are hard to generate, and teacher collaboration will help to build a stock of good questions.
  • Practical Techniques
    • Teacher-led classroom discussion is one of the most universal instructional practices.
    • So although many people assume that American teachers talk too much, they actually talk less than teachers in countries with higher performance. It would appear that how much students learn depends more on the quality than the quantity of talk.
    • Less than 10 percent of the questions that were asked by teachers in these [Brown & Wragg, 1993] classrooms actually caused any new learning.
    • I suggest there are only two good reasons to ask questions in class: to cause thinking and to provide information for the teacher about what to do next.  Example, triangle with 2 right angles.
    • The other reason to ask questions is to collect information to inform teaching…
    • …American classrooms were characterized by relatively low degrees of student engagement.
    • Student Engagement
      • Malcolm Gladwell, Outliers, ages of professional hockey players
      • Other sports, as well.
      • In almost any classroom, some students nearly dislocate their shoulders in their eagerness to show the teacher that they have an answer to the question that the teacher has just asked.
      • High-engagement=students working together and using language as a tool, show higher achievement.
      • …[Students] who are participating are getting smarter, while those avoiding engagement are forgoing the opportunities to increase their ability.
      • This is why many teachers now employ a rule of “no hands up except to ask a question” in their classrooms (Leahy, Lyon, Thompson, & Wiliam, 2005). The teacher poses a question and then picks on a student at random.
      • Some teachers claim to be able to choose students at random without any help, but most teachers realize that when they are in a hurry to wrap up a discussion so that the class can move on, they are often drawn to one of the usual suspects for a good answer.  Popsicle sticks.
      • The major advantage of Popsicle sticks can also be a disadvantage. It is essential to replace the sticks to ensure that students who have recently answered know they need to stay on task, but then the teacher cannot guarantee that all students will get a turn to answer.
      • Most teachers realize that being called upon at random will be a shock for students unused to participation in classrooms. However, moving to random selection can also be unpopular with students who participate regularly.
      • For other students, random questioning is unwelcome because they are unable to control when they are asked questions.
      • Lemov (2010).  Cold-calling and No Opt Outs.
      • Often, students will say, “I don’t know,” not because they do not know, but because they cannot be bothered to think.
      • “phone a friend”, “ask the audience”, “fifty-fifty”…All these strategies derive their power from the fact that classroom participation is not optional…
    • Wait Time
      • However, the amount of time between the student’s answer and the teacher’s evaluation of that answer is just as, if not more, important.
    • Alternatives to Questions
      • Asking questions may not be the best way to generate good classroom discussions.  Try statements.
      • The quality of discussion is usually enhanced further when students are given the opportunity to discuss their responses in pairs or small groups before responding (a technique often called “think-pair-share”).
    • Evaluative and Interpretive Listening
      • [John Wooden] was once asked why other coaches were not as successful, and he said, “They don’t listen.
      • When teachers listen to student responses, many focus more on the correctness [Evaluative listening] of the answers than what they can learn about the student’s understanding (Even & Tirosh, 1995; Heid, Blume, Zbiek, & Edwards, 1999).
      • However, when teachers realize that there is often information about how to teach something better in what students say—and thus how to adjust the instruction to better meet students’ needs—they listen interpretively.
    • Question Shells
      • There are a number of general structures that can help frame questions in ways that are more likely to reveal students’ thinking.  Example “Why is ___ an example of ___.”


      • Another technique is to present students with a contrast and then ask them to explain the contrast, as shown in table 4.2.


    • Hot-Seat Questioning
      • In hot-seat questioning, the teacher asks a student a question and then a series of follow-up questions to probe the student’s ideas in depth.
      • If teachers are to harness the power of high-quality questioning to inform their instructional decisions, they need to use all-student response systems routinely.
    • All-Student Response Systems
      • The problem with such techniques [“thinking thumbs” or “fist to five”] is that they are self-reports, and, as we know from literally thousands of research studies, self-reports are unreliable.
      • However, a very small change [asking cognitive not affective questions] can transform useless self-reports into a very powerful tool.
        • Example students that signal correct or incorrect know they will be followed up with.
        • Mercury phosphate equation example.
        • Example about the length of a line on a grid.
        • Example about incorrect classification of levers.
      • In each of these four examples, the teacher was able to ensure both student engagement—after all, it is very easy to tell if a student has not voted—and high-quality evidence to help decide what to do next.
    • ABCD Cards
      • Each student has a set of cards with letters.  Example.


      • Most students should recognize that A and B represent one-fourth, and hopefully, most students will also realize that in diagram C, the fraction shaded is not one-fourth.
      • The use of multiple correct answers allows the teacher to incorporate items that support differentiation, by including some responses that all students should be able to identify correctly but also others that only the ablest students will be able to answer correctly. Such differentiation also helps keep the highest-achieving students challenged and, therefore, engaged.
      • Example Heysel Stadium disaster 1985.  …she was pleased to see that the class now had a much more complex view of the tragedy…
      • One elementary school teacher takes the idea of cards one step further with what she calls “letter corners.”
      • ABCD cards can also be used to bridge two lessons.  Teacher verified that content covered that day was understood and content for the next day wasn’t understood yet.
      • A major difficulty with ABCD cards is that they generally require teachers to have planned questions carefully ahead of time, and so they are less useful for spontaneous discussion….
    • Mini Whiteboards
      • Whiteboards are powerful tools in that the teacher can quickly frame a question and get an answer from the whole class….
      • One teacher wanted to use whiteboards, but there was insufficient money in the school’s budget to acquire these, so instead, she placed sheets of letter-sized white card stock inside page protectors to provide a low-cost alternative.
    • Exit Passes
      • When questions require longer responses, teachers can use the exit passes….
      • Exit pass questions work best when there is a natural break in the instruction; the teacher then has time to read through the students’ responses and decide what to do next.
      • All these techniques create student engagement while providing the teacher with evidence about the extent of each student’s learning so that the teacher is able to adjust the instruction to better meet the students’ learning needs.
    • Discussion Questions and Diagnostic Questions
      • Example


      • Students who say A or B have some understanding but not of the value of the index of a sequence.
      • This question can lead to a valuable discussion in the mathematics classroom, since it allows the teacher to challenge the idea that mathematics is a right-or-wrong subject.
      • The teacher learns little just by seeing which of these alternatives students choose. She has to hear the reasons for the choices. That is why this question is a valuable discussion question, but it is not a good diagnostic question.
      • Example


      • In some ways, this is a sneaky question, because there are two correct—and four incorrect—responses.
      • The crucial feature of such diagnostic questions is based on a fundamental asymmetry in teaching; in general, it is better to assume that students do not know something when they do than it is to assume they do know something when they don’t. What makes a question useful as a diagnostic question, therefore, is that it must be very unlikely that the student gets the correct answer for the wrong reason.
      • Science Example


      • A teacher who has been focusing on Archimedes’ principle hopes that the students choose B, but there are valid reasons for choosing alternatives.
      • Example:  physics/forces.


      • A seems reasonable, but is naïve.
      • B has also an element of truth.
      • C was what the teacher had been hoping for, but is a little imprecise.
      • D is similar to B
      • E is clearly mystical and wrong.
      • The first and last responses, therefore, are obviously incorrect and related to well-known naïve conceptions that students have about the physical world.
      • Physics is an unnatural way of thinking about the world—if it were natural, it wouldn’t be so hard for students to learn.
      • This question is not really asking, “What’s happening here?” It is asking, “Can you think about this situation like a physicist?”
      • History question example.  World War II started 1937, 1938, 1939, 1940, or 1941?
      • Again, the point is that the teacher learns little about the quality of student thinking from hearing which answer a student chooses; the teacher needs to hear reasons for the choice, and that means hearing from every student in the classroom.
      • Example:  Diagnostic question in history.
      • It is the quality of the distractors that is crucial here—it is only because the distractors are so plausible that the teacher can reasonably conclude that the students who choose D have done so for the right reason.
      • Example:  Thesis statement question.
      • Example:  Modern foreign language question.
      • Diagnostic questions can be used in a variety of ways.
        • They can be used as “range-finding” questions to find out what students already know about a topic before beginning the instruction.
      • [D]iagnostic questions are most useful in the middle of an instructional sequence to check whether students have understood something before moving on.  [Hinge-point questions.]
      • Hinge-point question is a craft.  Two guidelines/principles.
        • First, it should take no longer than two minutes, and ideally less than one minute, for all students to respond to the question
        • Second, it must be possible for the teacher to view and interpret the responses from the class in thirty seconds (and ideally half that time).
      • Example NAEP flights from Newton to Oldtown.
      • [S]tudents who think there are a hundred minutes in an hour and those who know there are just sixty get the same answer.
      • Earlier in this chapter, I proposed that there are two good reasons to ask questions in classrooms: to cause thinking and to provide the teacher with information that assists instructional decision making.
        • Or, to put it another way, ideally it would be impossible for students to get the right answer for the wrong reason
        • One way to improve hinge-point questions is to involve groups of teachers.  See also Larry Cuban on this point.
        • No question will ever be perfect, but by constantly seeking to understand the meaning behind students’ responses to our questions, we can continue to refine and polish our questions and prompts
        • A second requirement of questions to assist instructional decision making—much less important than the first but still useful to bear in mind when designing questions—is that the incorrect answers should be interpretable. That is, if students choose a particular incorrect response, the teacher knows (or at least has a pretty good guess) why they have done so.
      • Multiple-choice questions are often criticized because they assess only low-order skills such as factual recall or application of a standard algorithm, although as the previous examples show, they can, if carefully designed, address higher-order skills.
      • In the classroom, these are much less important considerations, and multiple-choice questions have one great feature in their favor: the number of possible student responses is limited.
      • Sometimes it makes sense to administer the question as a series of simple questions.
      • Example lines of symmetry.  How many does each shape have?


      • She doesn’t try to remember how each student responds. Instead, she focuses on just two things: 1. Are there any items that a significant proportion of the class answers incorrectly and will need to be retaught with the whole class? 2. Which two or three students would benefit from individualized instruction?
      • She realizes that the incorrect answers may not necessarily indicate poor mathematical understanding.
      • In this episode, the teacher administered, graded, and took remedial action regarding a whole-class quiz in a matter of minutes, without giving herself a pile of grading to do. She does not have a grade for each student to put in a gradebook, but this is a small price to pay for such an agile piece of teaching.
      • However, high-quality questions seem to work across different schools, districts, states, cultures, and even languages [portability]. Indeed, sharing high-quality questions may be the most significant thing we can do to improve the quality of student learning.
  • Conclusion

Years ago, David Ausubel (1968) argued that the most important factor influencing learning is what the learner already knows and that the job of the teacher is to ascertain this and to teach accordingly. Students’ conceptions are not random aberrations but the results of sophisticated and creative attempts to make sense of their experiences. Within a typical classroom, there is clearly not enough time for the teacher to treat each student as an individual, but with careful planning and the thoughtful application of the techniques presented in this chapter, the teacher can make the classroom a much more engaging place for students and one in which the teacher is able to make rapid and effective instructional adjustments to meet the learning needs of all students. Once the teacher knows where learners are in their learning, she is in a position to provide feedback to the learners about what to do next—and this is the subject of the next chapter.



The Dayton Regional STEM Center. (2011).  STEM Education Quality Framework.  Retrieved July 10, 2012 from

Wiliam, D. (2011). Embedded Formative Assessment. Bloomington, Indiana: Solution Tree Press.

Trackbacks are closed, but you can post a comment.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: