Review |
The Polls in the 2002 French Presidential Election: An Autopsy
CLAIRE DURAND is a professor of sociology at Université de Montréal. ANDRE BLAIS is in the Department of Political Science at Université de Montréal. MYLENE LAROCHELLE is in the Department of Sociology at Université de Montréal. We wish to thank the Social Sciences and Humanities Research Council of Canada (SSHRC) for its financial support. We also wish to thank the numerous academics, pollsters, and former pollsters for their help in explaining the methods used by French Pollsters. We particularly wish to thank Michel Brûlé, Jean-François Tchernia, and Loïc Blondiaux for their much appreciated help. Finally, our sincere thanks to the reviewers who greatly helped us improve the original manuscript.
Address correspondence to Claire Durand; e-mail: claire.durand{at}umontreal.ca.
The first-round results of the 2002 French presidential election came as a shock to both French voters and people around the world. The French presidential election is a two-round system: it takes an absolute majority of the vote to be elected in the first round and, whenever no candidate is elected in the first round, a second round opposes the top two candidates of the first round two weeks later. In the months preceding the election, polls asked about not only voter intentions for the first round but also voter intentions for the second round, offering a choice between Jacques Chirac, the incumbent president, RPR (Rally for the Republic, right) and Lionel Jospin, incumbent prime minister, PS (Socialist Party), the obvious candidates for the second round. What happened on the first-round election day was not forecast by the polls: contrary to predictions, Jean-Marie Le Pen, FN (National Front, an extreme right-wing party), finished second with 16.9 percent of the vote and moved on to the second round. The newspaper Le Monde (2003)
stated, "France is hurt, and many French people are humiliated."
In fact, no poll had even hinted that such a result was possible. The twelve polls published during the week preceding the first-round election day (see table A1 in the appendix for a list of the polls and methodological information) gave an average of 18.0 percent to Jospin, the presumptive second-place finisher in the first round, and 12.7 percent to Le Pen—an average of 5.3 points in favor of Jospin—and every poll put Jospin ahead of Le Pen by at least 4 points. The 2002 election was the first since 1977 in which polls could be published up to the very end of the campaign.1 Consequently, pollsters were under even more scrutiny than in previous elections, and the stakes were high. Since there is a possibility that the polls influence both turnout and voter choice (Blais, Gidengil, and Nevitte forthcoming), an examination of what happened during that campaign is unquestionably relevant.
|
This article discusses the various criteria that have been used to assess the quality of poll estimates and presents those utilized in this study. Then, we examine the French results to assess the extent to which the polls, collectively and individually, erred in their estimations. Finally, we look at the methodology used by French pollsters to investigate why the polls were wrong.
 |
How to Determine if the Polls Have Gone Wrong: Assessing Poll Accuracy |
|---|
 TOP How to Determine if... How Did the Polls...How Did the Pollsters...Comparing the 2002 French...What Went Wrong in...Sampling and Mode of...Question Wording and Proportion..."Adjustments"How Can We Know...ConclusionAppendixReferences |
|---|
How does one ascertain the accuracy of the polls? A related issue in the present case is obtaining comparable measures of accuracy that would allow the French election to be measured against other elections. In other words, how is it possible to compare the results of polls in single-round presidential elections (as in the United States.), in legislative elections where the head of government is the leader of the party that wins the majority of seats (as in Canada/Quebec, Great Britain, and Ireland), and in two-round presidential elections (as in France)? Does the varying number of major candidates in these different election processes hinder comparability?
Jowell et al. (1993) and Curtice and Sparrow (1997) measured the accuracy of the British polls by comparing the estimates of the final polls of the campaign with the vote for each party. In the Quebec context, Durand, Blais, and Vachon (2001, 2002) focused on the collective performance of pollsters and thus measured accuracy by comparing the actual election results with the average estimates of the polls published during the last week of the election, and with the vote forecast by time-series analyses of the campaign polls. Lau (1994), Vachon, Durand, and Blais (1999), and Durand (2002) measured individual poll accuracy using the difference between the estimates of each poll and estimates from time-series analyses.
For the U.S. presidential elections, Mitofsky (1998) summarized the methods used by the Social Science Research Council (SSRC) study of the 1948 preelection polls, with the goal of devising a single measure of accuracy for individual polls published as close to the end of the campaign as possible. Eight different methods had been devised at that time, all with virtues and drawbacks. He chose to retain four methods. Two of them, methods 1 and 2, assess the accuracy of the estimate of the leading party’s share of either the total vote or of the two leading parties’ vote; method 3 is the averaged absolute difference between voter intentions for each candidate and the final vote; and method 5 evaluates this same difference but only for the two leading candidates.
Mitofsky (1998) concluded that methods 3 and 5 gave the most satisfactory estimates of poll accuracy for U.S. presidential elections in which there are two main contenders, one round of voting, and the winner takes all.2 In a reply to Panagakis (1999), Mitofsky (1999) advocated the use of method 5 for comparability purposes. Traugott (2001) used methods 3 and 5 with and without the third-party candidate (Ralph Nader) to evaluate poll accuracy in the 2000 U.S. presidential election.
McElroy and Marsh (2003) used two estimates for the Irish election held on May 17, 2002. The average error—the sum of the absolute difference between the estimates of each of the six parties plus an "other" category and the vote, divided by the number of estimates—is similar to SSRC method 3. The systematic error—the sum of the signed difference between the estimates and the vote, divided by the number of estimates—gives an estimation of poll bias. Mitosfky (1999) used a difference of 1 point between a given poll and the vote as an indication of bias.
In the French situation, some problems arise. First, the French process is neither a legislative election nor a one-round presidential election. Second, the concepts of "major" and "third-party" candidates do not apply. In the 2002 French election, 16 candidates ran in the first round, half of whom were considered ideologically left-wing and half right-wing (see table A2 in the appendix for a list of candidates and political parties).3 Three candidates ended up with between 15 and 20 percent of the vote (Chirac, right; Jospin, left; and Le Pen, extreme right), four received between 5 and 10 percent (three from the left, one from the right) and the other nine candidates (four from the left and five from the right) got between 0.5 and 4.2 percent. In such a situation, using method 3, as suggested by Mitofsky (1998) and Traugott (2001) for U.S. presidential elections, for the three candidates who got more than 15 percent of the vote—recalculating the proportions so that the total equals 100 percent, as suggested by Crespi (Mitofsky 1998)—would be quite artificial since the three leading candidates got only 53 percent of the vote.4 Furthermore, to use method 5 one would have to focus on the two main candidates, whose combined vote was less than 40 percent.
|
Consequently, the following section examines overall poll performance by comparing the mean estimates of vote intention for each candidate and for each side (left and right) from polls published during the last week of the campaign with the election results. The article then examines the performance of individual pollsters. To evaluate the French situation against the U.S. situation, a comparison of the estimates of the total left- and right-wing vote is performed. This is similar to comparing the two leading candidates.
|
How Did the Polls Fare Overall? |
|---|
|
TOPHow to Determine if...How Did the Polls...How Did the Pollsters...Comparing the 2002 French...What Went Wrong in...Sampling and Mode of...Question Wording and Proportion..."Adjustments"How Can We Know...ConclusionAppendixReferences |
|---|
Table 1 compares the mean estimates of vote intentions for the 12 surveys published between April 12 and April 19 with the results of the first-round election held on April 21, 2002, for the eight candidates on the left and the eight candidates on the right. The table shows that overall, the left-wing vote was overestimated by 5.6 percentage points (48.5 percent estimated compared to the actual vote of 42.9 percent). The overestimation of the left-wing vote was not restricted to a few candidates, five out of the eight being slightly overestimated. The highest overestimation was for Robert Hue, the Communist Party candidate (2 points on a vote of 3.4 percent). On the other hand, the polls significantly underestimated the vote for only one right-wing candidate, Le Pen, by 4.2 percentage points, for a vote of 16.9 percent. Le Pen accounted for most of the underestimation of the right-wing vote.
|
Table 2 focuses on the three leading candidates, comparing the differences between estimates of voter intent during the last week of the campaign for Jospin, Chirac, and Le Pen with the actual vote. While the difference in the final vote between Jospin and Chirac (3.7 points) was only slightly underestimated (1.7 points) by the polls published during the last week, the difference between Chirac and Le Pen (3 points) was overestimated by 4.3 points, and the difference between Jospin and Le Pen (–0.7 points) was overestimated by 6 points.
|
|
How Did the Pollsters Fare Individually? |
|---|
|
TOPHow to Determine if...How Did the Polls...How Did the Pollsters...Comparing the 2002 French...What Went Wrong in...Sampling and Mode of...Question Wording and Proportion..."Adjustments"How Can We Know...ConclusionAppendixReferences |
|---|
One way to look at pollster performance is to compare the last published poll with the vote, focusing on the three leading candidates. Table 3 shows each pollster’s last estimate of the vote. There were six polling firms offering estimates of vote intention in the first round (see table A3 of the appendix). Two observations can be made. First, as with the average estimates calculated for the last week, the estimates for Jospin and for Chirac were quite accurate. However, all the estimates for Le Pen were inaccurate, with the differences between the estimates and the vote varying from 2.9 (BVA and CSA) to 6.4 points (IFOP). It should be noted that five out of the six pollsters had the same estimate for Jospin—18 percent; three pollsters put Chirac at 20 percent, and two had him at 19.5 percent.
|
|
An examination of the differences between the estimates for the three leading candidates shows quite clearly that the problem lies with Le Pen’s estimates, and especially with the estimates of the difference between Jospin and Le Pen. The gap between Le Pen and Chirac was overestimated by individual pollsters by 2 to 6.5 points; however, the difference between Le Pen and Jospin was overestimated by 4.7 (BVA, CSA, and IPSOS) to 6.7 points (IFOP). The assessment of individual pollster performance shows that they all erred in their estimation of Le Pen’s vote but that some were worse than others. IFOP stands as having seriously underestimated Le Pen at 10.5 percent, the highest estimate for Le Pen, at 14 percent, being offered by three firms, BVA, CSA, and IPSOS.
|
Comparing the 2002 French Presidential Election with the 2000 U.S. Presidential Election |
|---|
|
TOPHow to Determine if...How Did the Polls...How Did the Pollsters...Comparing the 2002 French...What Went Wrong in...Sampling and Mode of...Question Wording and Proportion..."Adjustments"How Can We Know...ConclusionAppendixReferences |
|---|
French and American elections are both presidential elections; however, a number of differences in the election processes may affect the comparison (two-round versus one-round elections, number of candidates, proportion of the vote held by leading candidates, etc.).
One way to get around the problem of comparability is to determine how well both sides (left versus right) were evaluated, which is the equivalent of having two candidates. Table 4 presents this information. The difference between the vote for the right-wing and left-wing candidates in the first-round of the 2002 French election was 14.2 (57.1 for the right and 42.9 for the left), while the average estimate from the pollsters’ last poll was 3.4 (51.7 right and 48.3 left). The average difference between the polls’ estimates and the vote is 5.4 (the last cell entry in the third column)5 compared to 1.7 for the 2000 U.S. election (Traugott 2001). Table 4 also shows that if ideological families are divided into extreme and traditional (which is how the French media usually presents results), the error is concentrated in two groups: the extreme right (minus 4 points) and the traditional left (plus 4.8 points).
|
|
What Went Wrong in the Polls’ Estimation of Le Pen? |
|---|
|
TOPHow to Determine if...How Did the Polls...How Did the Pollsters...Comparing the 2002 French...What Went Wrong in...Sampling and Mode of...Question Wording and Proportion..."Adjustments"How Can We Know...ConclusionAppendixReferences |
|---|
Two sets of reasons are given for erroneous poll estimates (Durand, Blais, and Vachon 2001
, Durand, Blais, and Vachon, 2002), one related to respondents, the other related to methods. Respondents may change their mind, abstain from voting, refuse to answer polls or to reveal their intentions, or they may be difficult to reach, and this differentially according to voting preferences. It has been argued that, because of these factors, the more conservative side (Curtice 1997; Curtice and Sparrow 1997; Durand, Blais, and Vachon, 2001, 2002; Jowell et al. 1993; Lau 1994), as well as the extreme parties (Boy and Chiche 1999; Breen 2000) or, more generally, the less popular parties in the media, may be underestimated by polls. The political reasons for the results of the first-round election, as well as for the failure of polls to predict these results, have been studied elsewhere (Cautres and Mayer 2004; Lewis-Beck 2004; Perrineau and Ysmal 2003).
Though shifting allegiances among the electorate after the last polls were conducted could explain some of the discrepancy between poll estimates and the actual vote, it is highly unlikely that they account for a significant portion of the error because no event or particular circumstances occurred in the final days that could explain such shifts and because polls were conducted until three days before the election. However, low voter turnout (a record low 71.6 percent, compared to 78.4 percent in 1995) and differential participation could explain some of the discrepancies between poll estimates and the actual vote. The outcome of the first round of the election was considered to be settled: everybody knew in advance the identity of the two top two candidates (Blais 2004), with the consequence that there was less interest in the election and a multiplication of candidates (Gerstlé 2003).
In this article we concentrate on methodological explanations, more precisely on sampling and selection, question wording, and finally pollsters’ adjustments. Since the French electoral law makes it mandatory for pollsters to file information on their methodology at the Commission des Sondages (Survey Commission)—this information being quite similar to American Association for Public Opinion Research’s standards for minimal disclosure —and gives the public access to this information, the methodology used by the French pollsters can be examined in order to see whether it could explain their failure.6
Although the law requires that all the methodological information be made available to citizens, the researcher in charge of obtaining the information encountered many difficulties.7 Because of these difficulties, we decided to select some surveys: the last published poll of each firm, when available, for the two rounds of the presidential election—April 21 and May 5—and the two rounds of the legislative elections held on June 9 and 16. We assembled as much information as possible (see note 10). Private discussions with pollsters and former pollsters allowed us to complete the information.
|
Sampling and Mode of Interview |
|---|
|
TOPHow to Determine if...How Did the Polls...How Did the Pollsters...Comparing the 2002 French...What Went Wrong in...Sampling and Mode of...Question Wording and Proportion..."Adjustments"How Can We Know...ConclusionAppendixReferences |
|---|
Five out of six pollsters used primarily telephone interviews,8 while the sixth used face-to-face interviews. All of the pollsters used what appears to be a form of probability sampling with quotas (PSQ) (Stephenson 1979
), as is common in Europe (Taylor 1995), based on the distribution of age, sex, and an indicator of social class—the head of the household’s socio-professional category (CSP). Region and size of municipality were also used in the stratification. The information on how the telephone numbers or addresses were selected is not provided in the methodological notes. Some pollsters informed us that, for telephone surveys, they used a sample of listed telephone numbers (provided by the national telephone company, France Telecom). Quotas were then applied to select within households. The polls were conducted very rapidly, usually two days for telephone surveys, with no callbacks or appointments. The face-to-face surveys usually took place over a longer period (four days). The polls were conducted primarily during weekends: 23 out of the 44 polls published during the last month were conducted between Thursday evening and Saturday. The polls of the last week were conducted during weekdays since their publication was banned on election day and the day before, that is, Saturday and Sunday. The tables filed by the pollsters at the Survey Commission show that the quotas were apparently very well respected: the samples almost perfectly reproduced the national distribution on sex, age groups, and CSP. The pollsters use, in addition to the two gender groups, five age groups and seven to eight socio-professional groups, strata constituted by six to nine different regions, and four categories of size of municipality. Quotas are thus very refined and may be very difficult to fill at the end of the fieldwork.
These types of quota-based samples may have a self-selection bias since no effort is made to reach respondents at different periods of the day or week or to convince respondents to participate.9 In the present case, the quotas are so detailed that interviewers may be tempted to cheat in order to fill them. The pollsters acknowledge that their samples are usually biased in favor of more educated people.10
|
Question Wording and Proportion of Non-Disclosers |
|---|
|
TOPHow to Determine if...How Did the Polls...How Did the Pollsters...Comparing the 2002 French...What Went Wrong in...Sampling and Mode of...Question Wording and Proportion..."Adjustments"How Can We Know...ConclusionAppendixReferences |
|---|
Another possible source of error is the way most pollsters ask the voter intent question. With the exception of one pollster, only one question was asked, and no leaning question followed (see table A3 in the appendix). The proportion of non-disclosers varied widely between, and sometimes within, firms. Respondents who refused to reveal their intention, who said they did not know, and who indicated they would spoil their ballot varied from a low of 9 percent (IFOP) to a high of 35 percent (CSA) during the last week (see table A1 in the appendix). During the whole campaign, the proportion of non-disclosers varied from a low of 5 to 31 percent to a high of 12 to 37 percent in the different firms. These variations suggest that the methods used to measure vote intentions may have a notable effect. Using a leaning question and relying on either a nonproportional attribution of vote based on the information from previous elections (Jowell et al. 1993
) or a statistical correction for selection bias (Berinsky 1999) might have improved the estimates. |
"Adjustments" |
|---|
|
TOPHow to Determine if...How Did the Polls...How Did the Pollsters...Comparing the 2002 French...What Went Wrong in...Sampling and Mode of...Question Wording and Proportion..."Adjustments"How Can We Know...ConclusionAppendixReferences |
|---|
The second sampling-related feature is weighting or, more precisely, "adjustments." Pollsters report that they adjust their figures based on the reconstruction of the vote in previous elections; that is, they apply a weight so that the self-declared vote in previous elections reflects the actual vote. This procedure raises the question of which election should be used as a reference point. One might think that the more recent the election, the better. However, French elections differ in their format and importance. The available reference elections were the two rounds of the preceding French presidential election of 1995, the legislative election of 1997, and the European election of 1999. All of these elections differed in voter turnout, mode of vote, and so on. The pollsters report that they adjust for all these elections, as well as for the likelihood of voting, the certainty of the choice, and sometimes, other "relevant" variables. A typical note in the file at the Survey Commission reads like this:
Diverse weights have been applied according to socio-demographic characteristics, the first-round vote in the 1997 legislative elections, the first-round and the second-round vote of the 1995 presidential election; the results are also analyzed in view of the results of various questions of opinion, and certainty of choice, etc. The BVA specialists determine the results taking into account these diverse weights and they rely on years of experience in electoral polls and measurement of vote intention. ... The determination of vote intention has essentially taken into account the observed poll results in accordance with the nature of presidential elections and the reconstitution of previous votes. In order to avoid any artificial evolution [of vote intentions] related to possible statistical errors specific to individual studies and not reflecting a real change in opinions, we take into account confidential inquiries conducted during previous days (BVA poll, published in Paris-Match, April 18, 2002).
How is it possible to adjust for past vote in many different elections plus other variables simultaneously? One pollster, who requested anonymity, explained in this way the process by which he decides on published estimates: "The statistician provides me with estimates according to different adjustments, using different previous elections as reference, together with likelihood of voting, certainty of choice. I look at the different columns and at the published estimates for the last week in order to figure out the most likely figure. Say a candidate had 2 percent the previous week and has 4 percent in most adjustments that week, I will put him at 3 percent. If he still has 4 percent in the next poll, then I will put him at 4 percent."
In summary, pollsters get different estimates and, then, either choose one figure over another, average the figures, or "smooth" them to arrive at a plausible adjusted published figure that takes many pieces of information into account, including their own evaluation of the likely estimates.11
This practice may have had an impact on the published figures in the sense that if a poll had shown Le Pen ahead of Jospin, and this may have happened, the figures might have been "smoothed" because such a result would have been viewed by pollsters—as by all the political class and media—as highly unlikely (Gerstlé 2003). This "adjustment" practice presents another problem: We cannot examine the impact of methodological practices on published figures because we were not provided access to the "raw data," that is, the results obtained with and without the different weights that were used, filed at the commission. A related problem is the fact that all the pollsters used the same smoothing practices. In fact, for many estimates there were no significant differences, and often no difference at all, between pollsters.
The aberrant 2002 polling results were not unprecedented in France. In the 1997 legislative elections, the victory of the left came as a surprise in face of the unanimity of polls predicting a right-wing victory. Jérôme, Jérôme, and Lewis-Beck (1999) attribute the failure of the polls in that election to interviewing conditions—nonresponse, particularly among the extreme-right voters who were aggravated by the use of face-to-face interviews—and to changes in voter preferences over time. In the 2002 first-round presidential election, however, polls conducted face to face, most of them conducted by BVA, fared better in their estimation of Le Pen’s vote than the telephone polls (Durand, Blais, and Larochelle 2004). In addition, polls were conducted and published until the end of the campaign, contrary to 1997 where the publication of polls was banned during the last week. And, in 1997 as in 2002, estimates were adjusted "according to the professional experience of the pollster."
|
How Can We Know Whether There Are Sampling-Related Problems? |
|---|
|
TOPHow to Determine if...How Did the Polls...How Did the Pollsters...Comparing the 2002 French...What Went Wrong in...Sampling and Mode of...Question Wording and Proportion..."Adjustments"How Can We Know...ConclusionAppendixReferences |
|---|
One way to assess the extent to which there were problems in the representativeness of the samples is to examine the information that was available for some of the pollsters at the Survey Commission, namely, how the vote in previous elections was reported by respondents in 2002. Table 5 presents the reported vote for the 1997 legislative election obtained in six different polls conducted by Ipsos, Sofres, and CSA.12 The vote of the extreme right (15.1 percent in that election) was systematically underrepresented or underreported, being estimated at 2.8 to 5.7 percent in the six selected polls.13 Concurrently, the vote for the traditional left (27.7 percent) was systematically overestimated, at 34.6 to 43.7 percent. These figures mean that the weight applied to respondents who declared having voted for the extreme right in 1997 could be as high as 5.4 (in order to get 15.1 percent of FN voters in 1997). In addition, close to 40 percent of the respondents could not or would not report their vote in the 1997 election, or did not have the right to vote at that time, therefore leaving the pollster with weights for only around 60 percent of the sample. Could the five-year time lapse between the two elections explain this situation? An answer to this question comes from the reported vote in the first-round presidential election of 2002 in subsequent polls, some conducted only days after the election.
|
Table 6 shows that the underestimation of the right-wing vote was also reproduced in a poll conducted on election day, a poll conducted for the second round of the presidential election held May 5, and three polls conducted for the first round of the legislative election held June 9. The underrepresentation of the vote for the extreme right is less substantial than for the 1997 legislative election, but it is clearly present. While 19.2 percent of the vote went to extreme-right candidates (Le Pen and Mégret), the estimates from the polls varied from 7.8 to 11.1 percent. The traditional left (32.5 percent of the vote, including Jospin at 16.2 percent) was declared at 37.9 to 42.1 percent, that is., at least 5.4 points higher than its actual share of the vote; the traditional right (37.9 percent of the vote, including Chirac at 19.9 percent) was estimated at 37.7 to 44.5 percent.
|
The preceding figures allow us to conclude that it is unlikely that the time lapse explains the inaccurate reconstitution of the vote in previous elections. Curtice and Sparrow (1997), in an analysis of the 1997 British election, conclude that the problem is with quota sampling: Though the British Election Panel Study arrived at a good reconstitution of the 1992 vote using a random sample, the reconstitutions by pollsters using quota samples differed substantially from the actual vote. Curtice and Sparrow (1997) suggest that it would be good practice to adjust for previous vote. However, this practice does not always improve the accuracy and may even worsen the situation according to Taylor (1995).14
In short, the results presented show that the problem is not only memory, but it is also sampling strategy. Though the quotas are apparently very well respected, some political preferences are underrepresented. Adjusting for the vote in previous elections does not adequately compensate for the bias in the samples, and the adjustment "according to the professional experience of the pollster" makes things worse, impeding the examination of the problem and the identification of remedies.
|
Conclusion |
|---|
|
TOPHow to Determine if...How Did the Polls...How Did the Pollsters...Comparing the 2002 French...What Went Wrong in...Sampling and Mode of...Question Wording and Proportion..."Adjustments"How Can We Know...ConclusionAppendixReferences |
|---|
People use polls to obtain a reliable assessment of voter intent. Though French pollsters produced good estimates of two of the leading contenders, their difficulty in estimating voter intent for the extreme right-wing leader Le Pen had important political consequences. Voters assumed that Jospin and Chirac would make it to the second round, and some decided to support one of the minor candidates to send a message to the two top contenders (Blais 2004
). The uniformity of the methods used by French pollsters, as was the case in Great Britain in 1992—all using similar quota-based methods and modes of adjustment—hinders a thorough analysis of the impact of methodological practices. The situation, however, is likely to provoke similar questions to those raised concerning the methodology used in electoral polls in Great Britain following the 1992 election. One would be tempted to issue the same recommendation as Jowell et al. (1993, p. 258): "Our recommendation to pollsters and their clients, the mass media, is that they should invest in a program of methodological work as soon as possible. Sampling methods need to be improved, and the rather primitive methods of forecasting employed by the polls need to be supplemented by more sophisticated techniques that draw on the massive body of data about voting behavior and political attitudes that is freely available."
French pollsters could question their reliance on quotas, the use of listed telephone numbers only, and the duration and timing of their polls’ fieldwork. They could reduce the proportion of non-disclosers by using a leaning question. Furthermore, the use of a form of adjustment that is arbitrary and cannot be checked raises questions about the credibility of the polls. The only appropriate methods are those that can be scrutinized by independent analysts. Recognized methods such as nonproportional attribution of vote intention to non-disclosers and imputation of intent based on voter profile or other questions on political attitudes can be used to deal with the problem of underrepresentation of extreme-right vote. Finally, it must be noted once again that merely reproducing demographics in polls does not mean that the estimates of other dimensions will not be biased.
|
Appendix |
|---|
|
TOPHow to Determine if...How Did the Polls...How Did the Pollsters...Comparing the 2002 French...What Went Wrong in...Sampling and Mode of...Question Wording and Proportion..."Adjustments"How Can We Know...ConclusionAppendixReferences |
|---|
|
Footnotes |
|---|
1 The 1977 law banned the publication of polls during the last week before the vote. However, this provision was declared noncompliant with European standards of freedom of speech and was consequently amended in early 2002 to remove restrictions on poll publication and to expand public access to information on polls published during electoral campaigns.
2 Following a debate on whether third-party estimates should be included and on the allocation of the undecided vote, Panagakis (1999) argued that a modified method 3—one which included third-party candidates (except the "other" category) and allocated the undecided vote—was more appropriate. Mitofsky (1999), in his reply to Panagakis, maintained his position, stating that the difference between the two leading candidates was the most relevant information and had the advantage of allowing for comparability.
3 Though the candidates are usually supported by political parties, it is individuals who run in the presidential elections. In order to be a candidate in the first round, one has to gather the signatures of 500 mayors or elected officials.
4 If the real difference between a poll’s estimate and the vote is 2 points on a total of 53 percent (sum of estimates of vote intention for the three leading candidates), this difference will be boosted at 3.8 points (2 divided by 53 multiplied by 100 percent) if we recalculate the proportion so that the total equals 100 percent. This can only be appropriate when the total is not that far from 100 percent.
5 The average absolute difference is the sum of the differences ((57.1 – 51.7) + (48.3 – 42.9)) divided by the number of differences (2). It is equivalent to calculate the difference between the estimates of differences in the election (14.2) and in the polls (3.4), divided by 2.
6 In 1977 France promulgated a law which gave a commission the mandate to control the polls conducted in an electoral context. The law—which also banned the publication of polls seven days before election day—required pollsters to provide technical information on the polls they conducted; more specifically, the publication and diffusion of a poll must be accompanied by information on who conducted the poll, for whom the poll was conducted, the number of respondents, and the dates of fieldwork, and a notice should be filed at the commission indicating the topic of the survey, the method by which the respondents were selected and the composition of the sample, the conditions in which the survey took place, the exact wording of the questions, the proportion of nonrespondents for each question, the limits of the study, and if appropriate, the method used in order to obtain "indirect" results. The law was modified in 2002 in order to allow the publication of polls until the day before election and to allow all citizens to have access to the methodological notices provided by the pollsters.
7 She was required to ask for an appointment and to specify in advance the files she wanted to consult. The personnel would not let her photocopy any document, so she had to write down the relevant information. Finally, she was informed that part of the information—the results obtained before adjustment—though required by law, would not be available unless the pollsters give their permission. We asked the pollsters for that permission, and we received a positive answer from one.
8 One pollster conducted five polls by telephone and two polls face to face.
9 Stephenson (1979), in a comparison of data from the General Social Survey (GSS) gathered with probability samples with quotas (PSQ) and with full probability sampling (FP), shows that the possible underrepresentation of less cooperative citizens may not always lead to substantial bias. However, in the present situation, one could argue that this possible underrepresentation may have played a role in the underestimation of the vote for the extreme right due to the sociological profile of its voters.
10 Our verification on polls conducted in Quebec using random samples show, however, that overrepresentation of more educated people is also present.
11 After the "catastrophe" of the first round, only two pollsters, IPSOS and CSA, decided to conduct polls for the second round of the presidential election. IPSOS, however, decided to publish mean estimates and the range of its estimates instead of one figure. The information published on April 29 gives an idea of the spread of estimates: Chirac got a mean of 78 percent in vote intention, ranging from 74 percent to 81 percent, according to estimates based on the certainty of choice.
12 The choice of polls reflects our selection and the information present in the files. However, because of the conditions we faced at the Survey Commission, the fact that some information is missing (sample size) does not mean that this information was not present in the files.
13 We selected one poll by pollster by election, usually the closest to the election itself when available, covering the two rounds of the presidential elections (April 21 and May 5) as well as the two rounds of the legislative elections (June 9 and 16). The estimates presented here come from two polls conducted for the first round and one for the second round of the presidential election—Ipsos is the only pollster who conducted polls during the second round, and it published mean estimates and a spread of estimates—and three polls conducted for the first round of the legislative election.
14 "In some countries, the data are routinely weighted by ‘past voting,’ bringing claimed voting in the last election into line with the actual results. In the U.S. we can demonstrate that this would have made some of our very accurate election forecasts extremely inaccurate" (Taylor 1995, p. 212).
|
References |
|---|
|
TOPHow to Determine if...How Did the Polls...How Did the Pollsters...Comparing the 2002 French...What Went Wrong in...Sampling and Mode of...Question Wording and Proportion..."Adjustments"How Can We Know...ConclusionAppendixReferences |
|---|
Berinsky, Adam J. 1999. "The Two Faces of Public Opinion." American Journal of Political Science 43(4):1209–30.
Blais, André. 2004. "Strategic Voting in the 2002 French Presidential Election." In The French Voter: Before and After the 2002 Elections, ed. Michael Lewis-Beck, pp. 93–109. Hampshire, UK: Palgrave.
Blais, André, Elisabeth Gidengil, and Neil Nevitte. Forthcoming. "Do Polls Influence the Vote?" in Capturing Campaign Effects, ed. Henry E. Brady and Richard Johnston. Ann Arbor : Michigan University Press.
Boy, Daniel, and Jean Chiche. 1999. "La qualité des enquêtes d’intention de vote : Le cas des régionales de 1998 (The Quality of Electoral Surveys: The Case of the Regional Elections of 1998)," in L’Etat de l’opinion (The State of Opinion), ed. Sofres, pp. 237–53. Paris: Editions du Seuil.
Breen, Richard. 2000. "Why is Support for Extreme Parties Underestimated by Surveys? A Latent Class Analysis." British Journal of Political Science 30(2):375–82.
Cautres, Bruno, and Nonna Mayer. 2004. Le nouveau désordre électoral, les leçons du 21 avril 2002 (The New Electoral Disorder, the Lessons of April 21, 2002). Paris: Presses de Sciences Politiques.
Curtice, John. 1997. "So How Well Did They Do? The Polls in the 1997 Election." Journal of the Market Research Society 39:449–61.[Web of Science]
Curtice, John, and Nick Sparrow. 1997. "How Accurate Are Traditional Quota Opinion Polls?" Journal of the Market Research Society 39:433–48.[Web of Science]
Durand, Claire. 2002. "Are Quebec Polls Biased Against Liberals?" Policy Options 23(2): 51–56.
Durand, Claire, André Blais, and Mylène Larochelle. 2004. "Une analyse des sondages de l’élection présidentielle française de 2002 (An Analysis of the Polls of the 2002 French Presidential Election)." In Echantillonnage et Méthodes d’enquête (Sampling and Survey Methods), ed. Pascal Ardilly, pp. 313–20. Paris: Dunod.
Durand, Claire, André Blais, and SébastienVachon. 2001. "A Late Campaign Swing or a Failure of the Polls? The Case of the 1998 Quebec Election." Public Opinion Quarterly 65:108–23.[CrossRef][Web of Science][Medline]
———. 2002. "Accounting for Biases in Election Surveys: The Case of the 1998 Quebec Election." Journal of Official Statistics 18(1):25–44.
Gerstlé, Jacques. 2003. "Une fenêtre d’opportunité électorale (An electoral opportunity)." In Le vote de tous les refus (The Vote of All Refusals), ed. P. Perrineau and C. Ysmal, pp. 29–52. Paris: Presses de Sciences Politiques.
Jérôme, Bruno, Véronique Jérôme, and Michael Lewis-Beck. 1999. "Polls Fail in France : Forecasts of the 1997 Legislative Election." International Journal of Forecasting 15:163–74.[CrossRef]
Jowell, Roger, Barry Hedges, Peter Lynn, Graham Farrant, and Anthony Heath. 1993. "The 1992 British Election: The Failure of the Polls." Public Opinion Quarterly 57:238–63.
Lau, Richard R. 1994. "An Analysis of the Accuracy of ‘Trial Heat’ Polls during the 1992 Presidential Election." Public Opinion Quarterly 58:2–20.
Lewis-Beck, Michael, ed. 2004. The French Voter: Before and After the 2002 Elections. Hampshire, UK: Palgrave.
McElroy, Gail, and Michael Marsh. 2003. "Why the Opinion Polls Got It Wrong in 2002." In How Ireland Voted 2002, ed. Michael Gallagher, Michael Marsh, and Paul Mitchell, pp. 159–76. London: Palgrave.
Mitofsky, Warren J. 1998. "Was 1996 a Worse Year for Polls Than 1948?" Public Opinion Quarterly 62:230–49.[CrossRef][Web of Science]
———. 1999. "Reply to Panagakis." Public Opinion Quarterly 63:282–84.[CrossRef]
Panagakis, Nick. 1999. "Response to ‘Was 1996 a Worse Year for Polls Than 1948?’" Public Opinion Quarterly 63:278–81.[CrossRef][Web of Science]
Perrineau, Pascal, and Colette Ysmal, eds. 2003. Le vote de tous les refus (The Vote of All Refusals). Paris: Presses de Sciences Politiques.
Stephenson, Bruce C. 1979. "Probability Sampling with Quotas: An Experiment." Public Opinion Quarterly 43(4):477–96.
Taylor, Humphrey. 1995. "Horses for Courses : How Survey Firms in Different Countries Measure Public Opinion with Different Methods." Journal of the Market Research Society 37(3):211–19.
Traugott, Michael. 2001. "Assessing Poll Performance in the 2000 Campaign." Public Opinion Quarterly 65:389–419.[CrossRef][Web of Science][Medline]
Vachon, Sébastien, Claire Durand, and André Blais. 1999. "Les sondages moins rigoureux sont-ils moins fiables? (Are less rigorous polls less reliable?)" Canadian Public Policy/Analyse de politiques 25:557–61.[CrossRef]
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  M. Callegaro and G. Gasperoni Accuracy of Pre-Election Polls for the 2006 Italian Parliamentary Election: Too Close to Call Int. J. Public Opin. Res., June 1, 2008; 20(2): 148 - 170. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||