Our study group has been taking some online tests. We’re going to stop. Scott has already passed the test (70-320); Steve and I will take it this month, and we expect to pass, but frankly at this point we’ve put in so much effort, we don’t care any more.

On one of the tests, we have 171 questions in the database. To test ourselves, we take a 50-question test. What this means is we select 50 questions at random from a database. The questions are then returned to the database and stand the same probability of being selected for the next 50-question test.

I was curious about how many 50-question tests I had to take to be confident that I tried out every one of the 171 questions. I wrote a C# program to calculate that, and the following graph shows my results.

Just to be safe I took one 171-question test (I got two wrong by the way). I was happy with that score, but I’m not sure how much it means any more. I’ve seen these questions so often by now that I recognize the answer without even having to read through all the question. I force myself to read all the question and reason out an answer, but unfortnately, I’ve memorized the answers by now, and so the test is not measuring knowledge any more.

The peak of the graph is at 15 tests. Here are the values around the peak

12       4280

13       7789

14       11100

15       12650

16       12624

17       11400

18       9449

19       7501

20       5814

21       4307

22       3178

23       2382

24       1653

I don’t know what the distribution is. It doesn’t look Gaussian; it’s not symmetric. I think it might be interesting to look at the equations in more detail to determine what distribution models the behavior best. Better, though, would be to look at the algorithm that chooses the questions and see what changes one could make to narrow the distribution. For example, one could weight the probability to preferentially select questions that have not been selected before.