Are Africans negative about Chinese business impact? Good folks, Bad research

In February 2014, the Ethics Institute of South Africa (EthicsSA) released the results of an opinion survey on Chinese business in Africa. “Africans are generally negative about the impact of Chinese business on the continent,” EthicsSA announced. “This is the key finding.”

By now, even though the findings surprisingly contradict other public opinion surveys which are generally positive about China's impact in Africa, and they have been cited multiple times by reporters and researchers, I have never seen any critique of the selection and confirmation biases in their (pretty flawed) methodology.

In the past week I have seen this study cited yet again by two papers, including a draft working paper by one of our SAIS-CARI research teams — which reminded me that this blog post — which I see I've started and abandoned four times (!) is way overdue.

The EthicsSA team are no doubt good folks, but their report has severe methodological weaknesses that should lead any researcher to treat it not as a proper public opinion survey, but as an excellent example of selection and confirmation bias.

Of course, as a researcher, when I got my first alert about this survey, I went to the full report. It is immediately obvious that it is a bit hard to say something like “Africans are generally negative” on the basis of data primarily from three countries (the online survey had 1056 respondents, mainly from South Africa, Nigeria, and Kenya).  But this may be the least of the significant flaws of the study.

1. Selection Bias

“Selection bias is the selection of individuals, groups or data for analysis in such a way that proper randomization is not achieved, thereby ensuring that the sample obtained is not representative of the population intended to be analyzed (Wikipedia).” 

Is this a representative sample of “Africans” or even the population of the three countries where the survey mainly took place?

Hardly. The full report notes that the researchers elicited responses from 15 African countries — via the internet — but received low response rates. Here's how they did it:

“Invitations to participate in the survey were extended to contacts of the Ethics Institute of South Africa (EthicsSA) and of Globethics.net in the respective countries. … The initial objective was to achieve a representative sample for each of the participating countries. However, reliance on only the networks of EthicsSA and Globethics.net in the participating countries and on the media to encourage participation were insufficient to attract the required response. Thus, the data collected was not representative of the populations of the participating countries. The only countries that produced sizable samples were South Africa (299 respondents), Nigeria (197 respondents) and Kenya (204 respondents). 

So how did this method of attracting respondents bias the sample?

The report notes that the “low response rate can partly be attributed to the online format of the questionnaire. In many African countries, internet access still remains a luxury rather than the norm.”

And in keeping with the fact that internet access is “a luxury,” the report notes that 54% of the respondents had post-graduate degrees, while another 27% had university degrees. For Nigeria, this bias was even more pronounced with 74.5% of the respondents having post-graduate degrees, and 24% graduate degrees: i.e. 98.5% of the Nigerian respondents were very highly educated. So now we know that very highly educated Nigerians, South Africans, and Kenyans — and perhaps foreigners living in these countries who are on their contact lists — do not think positively of the Chinese presence.

But there's more: who are these highly educated respondents?

Recall that the Ethics SA sent invitations to participate to their (highly educated, internet-savvy, deeply concerned with ethics) contacts and those of Globethics.net in the respective countries. This creates even more bias. One might call it confirmation bias.

2. Confirmation Bias

“The tendency to search for, interpret, favor, and recall information in a way that confirms one's preexisting beliefs or hypotheses, while giving disproportionately less consideration to alternative possibilities (Wikipedia). 

To see how confirmation bias worked in this survey, imagine the US Chamber of Commerce doing a survey on free trade by sending it out only to its subscriber list.

And imagine the same survey sent to the subscriber list of an important trade union group, American Federation of Labor and Congress of Industrial Organizations (AFL-CIO).

Then imagine either organization issuing a press release saying: “Americans are generally … [negative/positive] … about free trade.”

Now you can see how providing a questionnaire to one's own contact list is likely to reach a biased sample — of people who are likely to confirm your own beliefs, rather than alternative beliefs.

3. Other Problems

The report had other problems. Let me mention two. There was no effort to find out if those surveyed actually had direct experience of a Chinese company in their country, or had direct experience of Chinese-built infrastructure. Yet they were asked questions that presumed some prior knowledge. This led to some strange responses.

• Chinese worker myth. More than half of the respondents — 53% — agreed or strongly agreed with a pervasive myth: that Chinese companies only use Chinese citizens as employees. On the other hand, 59% agreed that “Chinese companies create employment opportunities for Africans.” Go figure.

• Quality of Chinese construction. In South Africa, only 10% are happy with [Chinese-built] infrastructural projects.” Yet due to protections of local services in South Africa and the strength of local contractors, Chinese builders do a very small amount of business in that country: annually the South Africa market is less than 1% of Chinese companies' African construction business. How many South African respondents would actually know anything about a Chinese-built infrastructure project? On the other hand, in Kenya 88% were satisfied with their Chinese construction projects. Given the large visibility of Chinese-built roads in Nairobi, this is more likely to reflect actual experience. But how do we know?