JTI’s response to Peter Kellner

A few days after ASH wrote to Japan Tobacco asking for the methodology underpinning its retailer survey, the company was asked the same questions by former ASH trustee Peter Kellner. Kellner, a polling expert who was also Chairman, and then President of YouGov from 2001-16, specifically asked about JTI’s response to the Government’s consultation on “Creating a Smoke-Free generation” in December which stated that:

“A recent survey of 1,000 UK independent retailers commissioned by JTI found that 62% percent believe that implementation will prove costly to their business and 66% believe that the Government does not have funding nor resources for police enforcement.”

On Thursday 11th April, JTI responded but have still not answered all the questions put to them.

Peter Kellner has provided the following comment to ASH about JTI’s response:

The company that JTI appointed for this survey was Acorn Retail Promotions. It describes itself as a “telesales” company, not a research agency. It is not a member of the British Polling Council, and therefore not covered by BPC’s rules on transparency.

JTI says Acorn ”used its nationwide database of retailers” that sell tobacco. No indication is given as to how the database was constructed, or whether its profile matches that in, for example, ONS business statistics.

The questionnaire refers to a ”Generational Ban”. It does not explain the term. If it was explained in the preamble, this is not shown. Indeed, as this was a telephone survey, there would certainly have been a preamble for the Acorn representative to read out. This also means that we don’t know how Acorn described the purpose of the survey.

As for the questionnaire itself, there is no indication that the implementation of the generational ban will take place, year-by-year over decades, with only a small incremental impact being felt by retailers each year. This should have been made clear. It wasn’t in the questions, and the preamble remains a mystery.

It is best practice not to identify the client, but whether the client is identified or not, the questionnaire should be designed so that, when it concerns a debate between two sides of an argument, a reasonably intelligent respondent should not be able to work out the client’s position. It is abundantly clear that this questionnaire was commissioned by opponents of the Government’s proposals.

This can matter, especially with face-to-face and telephone surveys, where the respondent is speaking to a human being (rather than ticking boxes on a computer). My judgement is that intelligent respondents would have quickly worked out, if they had not already been told, that this survey was for a client that came from, or was working for, the tobacco industry. Respondents sometimes prefer to give the “correct” answers – that is, those they think the interviewer wants to hear. Such “social satisficing” is a well-established phenomenon.

Moreover, the survey contains three yes/no questions. (a) When seeking opinions and facts, yes/no questions should be avoided, as some people, often subconsciously, prefer to say yes than no. (b) On each occasion, the “yes” position favoured JTI’s policy. No attempt seems to have been to remove any risk of either social satisficing or “yes” bias.

(If, for some reason, yes/no questions are to be used, there should be an equal balance between those where “yes” favours the client’s position, and those where it opposes that position. This deals with the yes-bias, but of course still leaves the risk of social satisficing.)

In short, I would have rejected any questionnaire with these faults, on any subject, when I helped to run YouGov. I believe any reputable research agency would have taken the same position.