BLOG
3. Juni 2022
Herberts World #3
The third edition of Herbert's World did not come out of the blue. No no. Rather, a well-known TV show provoked the replica by our managing director Herbert Höckel. The column appeared on 24 May - as always on marktforschung.de - or just here:
What do Böhmermann allow?!
Yes, that's exactly what he would have called out, Giovanni Trapattoni. You will have noticed: Jan Böhmermann attacked us opinion and market researchers in his show, ZDF Magazin Royal, on 13 May. How dare he? Aren't there enough other, more important problems in the world that his pompous, expensive and, above all, publicly financed editorial staff should take care of? What a disgrace and a mess! VENGEANCE!
Be offended? Revenge? No, of course not.
After all, how many coaching sessions teach us to stay on the factual rather than the emotional level when criticising and, above all, to listen first. And even to say THANK YOU. After all, someone or a whole department has dealt with our topic and, at least from their point of view, has compiled a number of critical and questionable points. To be offended now and verbally strike back? How un-sovereign would that be?
I actually say out loud: THANK YOU dear Jan.
Thank you and your team for prying into our industry from the outside to analyse, examine, evaluate and finally condemn it. Whether right or wrong: fundamentally, any perspective other than our own is always an enrichment. Full stop!
But first things first. What problems has the magazine brought to the table on the subject of opinion polls?
- The problem of suggestive questions: Several times, the moderator asked his viewers to answer various questions by pressing a button with previously distributed voting devices ("voting nauts"). In doing so, he demonstrated the distorting influence of suggestive questions. Of course, this was highly exaggerated, but in the context of satire and comedy, it's probably perfectly okay. Here is the example of the vote on whether the Böhmermann magazine should take over the broadcasting slot of the "heute-show" in future.
- Politics is made with polls: This thesis states that opinion polls influence politicians, the media and society. That is probably true. Who can deny that the politicians of our time use well-meaning polls for their agenda, but downplay critical results? It's a problem, sure. And it is a problem for all the figures in this game, who pursue their goals with variously bona fide motives and thus end up irritating, manipulating or at least straining the citizen.
- The observation that too many institutes are vying for attention is in the same vein. Here the magazine mentions forsa, INSA, Allensbach, CIVEY, YouGov, Forschungsgruppe Wahlen and infratest dimap. All of them are big players who "hold" a TV station in various forms. Or was it the other way round?
- Then it was on to the good old landline telephone: in the past, large surveys were usually carried out via these connections, which also ensured representativeness, but as is well known, this has not been possible for a long time. Böhmermann placed a nice old retro telephone on his table to demonstrate that THAT is really yesterday's news and that everything has to be done online nowadays. CIVEY had to serve as an appropriate representative for this, only to discredit her immediately with a current survey example (on SPIEGEL.de), in which CIVEY had to concede a margin of error of 11.8% in the footnote. Is that too much for a publication? At least Böhmermann left no doubt about that.
- The mysterious methods: An expert from the FU Berlin, Thorsten Faas, voiced further aspects critical of market research via an interlude . For example, how the institutes actually arrive at their figures. Just like the concrete survey method, however, this remains a secret. The moderator then comes to the conclusion that opinion research does not measure opinions, but only whims. Whims that are one way today but another tomorrow and that in turn can be influenced by the published research results. The cat bites the dust
- The personal entanglement of various institutes with certain parties: Example Manfred Güllner from forsa and the SPD. Or Hermann Binkert of INSA with the AFD. I myself would like to add to this fact that such constellations have not only been accused since today of pursuing their own agenda and using "slightly coloured" questioning methods to achieve exactly the results that suit them.
Another point of criticism concerns the presentation of survey results, which with obviously precise percentage points and scientific-looking diagrams and graphics pretend to be serious, but which all too often cannot really be understood - see the lack of transparency in methods and processes.
This is as far as this small magazine inventory goes, because nothing is to be swept under the carpet here. I don't want to comment on all the points. As in every industry, there are good and bad performers, free riders and even corrupt journeymen. In such cases, of course, we must never stop at critical voices from the outside; we ourselves must also raise our voices again and again. After all, it is our most important asset that is at stake, the trust in our services. As well as our reputation, which is being held hostage by phonies.
Of course, 20 minutes is not enough time to cover a very complex subject in an entertaining way. This is probably also why some topics are presented in an extremely abbreviated and erroneous manner. Therefore, I would like to take the liberty of making a few small corrections for all non-market research readers of this column - by the way, without questioning the tenor of the programme as a whole.
The landline telephone
The issue of the old landline telephone has been trotted out by the show as if we were a bunch of technical illiterates who have not recognised the signs of the times. And the alternative of surveys via the internet is presented as if it only existed at the price of indisputable error tolerances.
So, dear ZDF editors, most institutes that used to conduct surveys exclusively by means of telephone interviews (CATI - Computer Assisted Telephone Interview) have long since switched to mixed-mode methods. In addition to telephone interviews, other participants are reached or interviewed on the digital level in order to solve the problems of representativeness. For those who want to know more: Here is a further explanation.
If a survey is nevertheless conducted exclusively by telephone, a majority today no longer have a fixed network connection, but as a rule have a mobile phone. These are the so-called "mobile-only" respondents. In the end, a survey sample is then partly covered by the fixed network, the other part by the mobile network. Both samples still have to be weighted correctly, and the "dual-frame method" from the Arbeitskreis Deutscher Markt- und Sozialforschungsinstitute (ADM) is finished. That was a VERY simplified presentation, so here is a more detailed explanation.
As a reminder: We do this professionally, i.e. opinion research. We therefore have experts for all methods, procedures and design processes in order to tackle and solve precisely those problems that were touched on in the programme. And at least for every serious market researcher I can claim that all of them align their services with the following four quality criteria:
- Objectivity: The survey results must not be influenced by the questioner.
- Reliability: That is, reliability. The repetition of the survey under the same conditions must have the same result.
- Validity: Can the results be described as valid? Have the results covered what was the subject of the survey?
- Representativeness: Do the respondents correspond to the so-called population in structure and composition? Here too is a friendly link for further information.
Last point: The suggestive questions.
Here's just this: every serious market researcher will try as hard as possible to keep the survey as un-suggestive as possible. Because otherwise - and Böhmermann makes an important point here - the answer is not worth the chart on which such information ends up.Quality, quality, quality! That's exactly what matters. Always. From careful planning to exact implementation, precise analysis to viable interpretation of the data.
Back to the beginning - not to be offended: Dear ZDF team: Thank you again for your criticism, even if it came across as rather superficial in parts. I at least felt well entertained and feel even more than usual the motivation to question and examine myself and my institute: Where do we have to face the accusations and improve processes?
And we should always look beyond the current status quo, because the world around us does not stand still. Here's a little saying from the calendar: Today's methods will be yesterday's tomorrow! Consequently, our development must never stand still. The qualification and integrity of the market researchers themselves must also be constantly re-examined. Addressed to my guild: Please let us regularly ask ourselves what role we really want to play in the spectrum of politics, society and economy?
I invite you all!
Back to you, Jan & Team: How about inviting me to Düsseldorf, where I can present my world of market research to you over a round of delicious Altbier? So that you can see with your own eyes that certainly not everything is bad in our industry. But to gain this insight, you have to be prepared to immerse yourself completely in the subject matter. Are you? Invitation extended!And hey, maybe you'll do another programme on the subject in a year's time? For a hopefully positive before-and-after view, I'll also help with the research - completely neutrally, of course! So much for my comment on your programme. I have finished!
Yours Herbert Höckel
Herbert Höckel ist geschäftsführender Gesellschafter hier bei bei der moweb research GmbH. Seit mehr als 25 Jahren ist er Marktforscher. 2004 gründete er die moweb GmbH, welche er bis heute als Inhaber führt. Die moweb aus Düsseldorf ist international tätig und eines der ersten deutschen, auf digitale Verfahren spezialisierte Marktforschungsinstitute.