Thursday, April 24, 2014

Open Letter to the Canadian Centre for Policy Alternatives

To: Ms. Kate McInturff
Senior Researcher, Canadian Centre for Policy Alternatives
Dear Ms. McInturff,
 
It is with concern that I read your most recent publication from the CCPA, "The Best and Worst Places to be a Woman in Canada". Your analysis looked at five categories for Canada's top 20 cities, normalized and ranked them, then averaged the rank for each to form a list for gender inequality across Canada.
 
Two weeks ago, I did an analysis where I looked at six factors for Canada's top 20 cities, normalized them, averaged the score for each, and ranked each city for zombie preparedness. My piece was intended as a joke but using real statistics, and I am extremely discouraged that what appears to be a similar level of statistical rigour was applied to your analysis as was applied to mine. Though I'm sure I agree in general with the principles behind your analysis, I have several specific concerns:
 
First of all, I would like to express some confusion as to how your final ranking was determined. You mention that:

The scores for the indicators in each category (i.e. health, education) are averaged to produce a final score for that category. Each indicator is given equal statistical weight in the calculation of the score for each category. The cities are then ranked according to their score. The overall ranking of the cities is produced by averaging their ranks in each category.
At first glance, this seems pretty straightforward. When I took a look at Appendix B, though, this didn't quite add up. Québec city is still very much in the lead (ranked 6th, 2nd, 3rd, 7th, and 8th; average: 5.2), but I'm confused about why Montréal (ranked 11th, 9th, 11th, 6th, and 7th; average: 8.8) is ranked above Sherbrooke (ranked 7th, 10th, 10th, 11th, and 2nd; average: 8.0). In fact, it looks like most of the cities ranked 3-9 are somewhat shuffled:
This is pretty minor in the long run, but I am curious about how you got this final result.
 
Secondly, though the concept of using similar indicators as the Gender Inequality Index and the Global Gender Gap Report certainly seems valid, weighting seventeen indicators in five categories equally before ranking the categories and averaging them ignores much of the analytics required to do a study like this justice. (As an aside, stating that something is "well-supported by medical research" without citing any research makes it tough to follow up on...)
 
An easy example of even weighting potentially causing issues is in the Education category. The female to male completion ratios are calculated from the National Housing Survey for High School, Apprenticeships, College, and University, then all four were averaged together to come up with an inequality score, which is then ranked for the full category.
 
Of the four indicators, apprenticeship rates are consistently the lowest for females. However, the fraction of all people that go into the trades at all is also very low, and varies more city-to-city than the female enrollment rate (with a coefficient of variation 1.75 times higher). As a result, weighting all four forms of education equally penalizes cities with low number of tradespeople in general by effectively giving the apprenticeship figure twice the influence it should.
 
If we account for this and weight each indicator based on its overall population size, it turns out that no city has fewer women than men in total education attainment at all. The cities most helped by weighting all forms of education the same were the cities with the highest number of people in the trades, which are Montreal, Quebec, and Sherbrooke. These cities drop to rank 17, 18, and 13 respectively for Education after accounting for this, and likely would have an effect on the conclusions drawn about Quebec as a province. 
 
Thirdly, the system of averaging rankings, though fun in a zombie analysis sort of way, both addresses all five categories completely equally and undermines the variation between cities within each category. Even without accounting for any changes in the Education category, it is a category with equal or nearly-equal participation between men and women. If all categories are treated equally and the ranking is the variable under consideration, then we are essentially saying that a city like Vancouver, which was ranked 15th in Education with a perfectly equal score of 1.00, is just as bad as a city like Oshawa, which was ranked 15th in Leadership with a score of 0.27.
 
Finally, I find the name of your paper to be alarming and misleading. Your report specifically (and in italics) reinforces that it examines "the gap between men and women, rather than overall levels of well-being." Saying that Edmonton is the worst place in Canada to be a woman sounds like a comment on well-being, especially after mentioning that it's a great place for median income, just not quite as awesome for women as it is for men. On the other hand, cities with dangerous crime rates might be considered great for women, as long as the crime is equally distributed. I guess "Gender Inequality Index of Canadian Cities" wasn't catchy enough.
 
It's time that gender equality statistics are taken more seriously than zombie statistics. I hope that future studies reflect this.
 
Michael Ross

1 comment:

Neekoo said...

Mike, I am a very big fan of your blog. Loving your easy-to-access writing, and your interesting ideas!