I use MyMathLab to run my classes and I have all this great info on each assignment called an “Item Analysis”. Below is an item analysis for a sample exam in one of my algebra classes. Here’s my question, which are the most urgent problems to cover when I go to give my class the review? I’m not looking for an intuitive listing of the top problems, but a spreadsheet function that I can use to sort the problems. Note, something that makes this problem harder is that the questions aren’t given with the same frequency. (Hint: the most obvious is calculating the percent of time a question is gotten correct, but there are certain problems with that.)
Interesting problem. I’ve thought about it and come up with some general ideas on what I would do to try to solve it. Please visit
http://primepuzzle.com/py/weights.html
I’ve hooked your problem into my Facebook group but don’t let the Python program assignment scare you. You could probably do all this w/ Excel.
Lee, I’m glad you got your programming geeks working on this. You’re courageous. I didn’t even try to work time into my ordering. I was just worried about correctness. The problem I was having by just considering how many they got wrong was sometimes a question with 4 misses had a much better percentage than a question with 3 misses. For example, question A has 4 misses out of 20 attempts and question B has 3 misses for 4 attempts. Considering only the number missed would rank question A tougher. Percentage-wise question B would be much tougher though. There is a similar problem ranking by percentage.
Now that you’ve reiterated the problem you’re having I’m realizing my approach may be too simplistic. I think I need some clarification on the precise meaning of each column of numbers in the spreadsheet. Specifically … is the number of times a problem is attempted one of the columns? What is the meaning of the very first column (the one with the # heading)?
I use question pooling so the first column denotes the question’s number on the quiz. The column that identifies the number of unique questions is the 2nd. It’s hard to explain. This should help: so when a student loads a quiz, the computer either loads question 2.3.1 or 2.3.7 to be the first question on the quiz. It loaded 2.3.1 twenty eight times and 2.3.7 twenty one times. Does that make any sense?
OK. Now that I know that, it seems to me the questions to review most would be those that have the lowest values for the ratios # Correct/# Uses on Quiz. I’ve grabbed the data above and written something that sorts these ratios and put it up at
http://primepuzzle.com/tunxis/mml.data.html
Based on this, it looks like you should review
3.3.49
2.6.17
3.4.9
3.2.21
etc.
Does this look correct?
There is a problem in going for the min of #correct/# of times used. This time problems asked only a few times can throw things off. For example, I’d be inclined to review a question that was answered correctly 1 of 10 times rather than a problem that was answered correctly 0 of 1 time, or even 0 of 2. Do you see what I mean? There has to be some kind of discount taken on percentages of questions asked a few times. For questions that are asked a lot you have a much better idea of where the class is. Just now I’m thinking of a formula like this: Let M be the maximum of the # of times used column.
(#correct/# of times used) + (M-# of times used) (1/M). The problems I’d want to cover would be the minimum values of this formula. (One thing that might, or might not be realistic about this is that the discount is linear?
Thanks for your feedback. Your formula is a slick way to
address the problem you mention and seems like a better way to determine what should be reviewed. I have used this approach and updated the page
my previous post mentions. Please click the link below to learn more.
http://primepuzzle.com/tunxis/formula-post.html
Lee, this might be obsessive, but I was thinking about that formula. I don’t like that problems can end up with an adjusted percentage over 100%. Take a problem that was solved correctly 4 out of 5 times. If M for the assignment is 20, then the formula adjusts the naive percentage, 80%, up 5% for every attempt short of 20 the problem has. In short, the adjusted percentage correct for this problem is 155%. What do you think about this edit to the formula?
(#right/#attempts) + (M-#attempts)((1-#right/#attempts)/M)
Now this example of a problem solved correctly 4 out of 5 times in an assignment that has an M of 20 would now be adjusted to .95 or 95%. The amount questions are discounted with this edit is not just a function of M, but also of the size of the interval between the naive percentage correct (#right/#attempts) and 100%.
I hope I’m not screwing up the programmers each time we make these adjustments!
I have made the adjustments you mention above. I think you now have a pretty good way to prioritize what should be reviewed.
http://primepuzzle.com/tunxis/mml.data.html