Response from the statistician dealing with the poll about Israel that reverberated around the world

Dear Amiram,

Here are the replies to the questions that you sent me together with the letter from Mr. Maurice  Ostroff that you forwarded me.

1. In reply to Mr. Ostroff question, from the data that I got from Dialog, the sample included 82 Olim from the Russia (who came after 1991), and the distribution of their religious affiliation was as follows:  66 Non-Religious, 6 Traditionalists and 10 Religious. None of them was Ultra-Ortodox. The distribution mentioned in Mr. Ostroff’s letter counts the 82 persons twice (as Russians and in the respective religious categories).

1A. The design of the poll was good, and optimal under the constraints of the sample size. In particular  I learned that the design used in that study  is similar to that used in the majority of the polls performed for the major media.  Specifically, the   design is based on a random sample with quotas for sample sized in each domain (defined by the religious affiliation for the non-Russion) approximately proportional to the proportions of the domains in the population.

1B. Each individual from the population (who has a telephone) had an equal probability to get a phone call from the interviewer.

1C. Furthermore, since given the probabilistic nature of the sampling process, the sample sizes from each domain cannot be exactly equal  to the proportions in the populations, I’ve learned  that the data were  weighed by the correspondent  coefficients.

2. Since the newly coming Olim from Russia are by far the largest group who arrived in the recent years , I think that it is reasonable to analyze their opinion as a special group, and to weigh them according to their prevalence in the population. As for all the other groups of Olim they are represented in the sample by the power (and the beauty, in my opinion) of the random sample. Obviously, as a matter of research, we can focus on any group and then increase their particular sample size accordingly, but that has no bearing on the reliability of the results  for the entire population.

3. The sample size of 500 is the routine sample size for the polls in the media in Israel. A sample size of 1000 is better, as a sample size of 5000 is even better than 1000. The effect is on the margin of error (or, in technical terms on the lengths of the confidence intervals). A sample size of 500 has a margin of error of about 4.4% (with probability of 95%), while a sample size of 1000 has a margin of error of about 3.2% (with probability of 95%). A sample size of 500 is certainly more than reasonable.

4. As for the importance of places of residence my reply is similar to that regarding the different groups of Olim. As far the results regarding the population, the randomization takes care of that. But of course, if one wants to know what the people in Kibutzim think, one can build a specific sample (called stratum) for that sub-population and increase that sub-sample accordingly. Again, no bearing on the reliability of the findings regarding the entire population.

5. All the people who have telephones are part of the population (in particular, if you want,  residents across the Green line).

Letter from M. Ostroff to Dialog

Dear Sima

I am a freelance writer and as discussed I am trying to understand the recent apartheid poll

According to a press report, the respondents are made up as follows

1. The report says that 503 respondents took part but as you will see the total is 577.

It is also not clear if the Russians are part of the various religious categories in which case they should not be counted in the total. The total would then be 497.

Where does the 503 come from?

2. As replies from Russians are shown separately in the report don''t you think that we should compare them with olim from USA, Iraq, South Africa etc?

4. Don''t t you think that the sample should be at least 1,000 as used by Gallup?

4. Is it not important to factor in the places of residence as people in Haifa have different views than Bnei Brak or Kibbutzim?

5. Were any residents across the Green line included?

I would appreciate your response

Many thanks

Maurice Ostroff

Reply by Maurice Ostroff

Dear Amiram,                                     October 29, 2012

Thank you for the response from your chief statistician which I read with interest. It explains several aspects that were not clear.

Dealing seriatim with his numbered paragraphs I reply as follows

1. Thanks for the clarification of the religious affiliations of the Russian respondents. The information is significant. You will observe that although I included the Russians in the total in my letter to Dialog asking for clarification, I did NOT include them in my letter to you. In that letter to you I showed a total of 495 as opposed to the 503 mentioned in the press statement. No explanation has been given for this difference but I guess that it possibly comprises 8 persons who refused to participate.

1A) - Noted

1B – Noted

1C  I agree that sample sizes from each domain cannot be exactly equal  to the proportions in the populations, but if they differ too widely the results become distorted.  It appears that the randomization process was ineffective as  Ultra orthodox comprised 12% of the respondents whereas according to the CBS they comprise 8% of the population. In effect they were overrepresented by 50%. I would appreciate more details of the manner in which the data was weighted to take this into account.

2. I agree that it is reasonable to analyze the Russian opinion but my question has not been answered as to how you arrive at an average total for the entire population when the Russian responses are included side by side with the responses of religious categories (in which the Russians are included) as in the following example of question 8 translated into English. (All figures relate to percentages)

Are the Russian votes counted twice; once in the relevant religious column and again in the Russian column? And how is the total column arrived at? Is it the average of all columns or is the Russian column excluded?

Note that the total in the traditional column does not equal 100%

3. Gallup and other major organizations use sample sizes of between 1,000 and 1,500 because in their opinion this provide a solid balance of accuracy against the increased economic cost of larger and larger samples

4. The place of residence cannot be discounted especially in a sample of only 500. I disagree that it has no bearing on the reliability of the findings regarding the entire population.

5. Noted

Sincerely

Maurice