BBC Focus Magazine

[M Paul Lloyd]

Focus ran a quite interesting article on statistics a while back but it seems that a degree of doubt as to their validity might be creeping in, which has serious ramifications for a good many fields.

http://www.sciencenews.org/view/feature ... _Its_Wrong

[Ush]

Reading through papers I keep finding that researchers include the statistical significance of findings from one study. Now, I have an intuitive grasp what it is supposed to mean but I also had a gut feeling that calculating the significance of results from one study was dodgy. Anyway, I'm glad I'm not the only one who doesn't really know what it supposed to mean.

Just yesterday I was going to ask the forum how statistical significance works.

Relatedly, has anyone ever read Government documents? They take a year's figures for say, asylum refusals, compare that number to the previous year's number and conclude it significant if there is say, a 20% decrease but then if you look at the long term trend the "significant decrease" is nothing but ordinary variation. The annual reports don't present that long term trend though. Maybe politicians are trying to hide their failures or maybe they are just innumerate?

[M Paul Lloyd]

As has oft been said in the past 'you can make statistics say anything' and this is a prime example. Although it is not just the world of medicine that uses statistics these days, the climate change 'hockey stick' graph created by D. Mann relied heavily upon statistical data as does an awful lot of science, climate based or otherwise.

As for politicians, well they just pick and chose whatever suites their needs at the time, bad results get ignored positives expanded, tis the way of the world.

[Shadowwolf]

Oh yes, politicians just cherry pick the numbers that suit the situation.

Though I think that statistics may be getting a bad rap and in the view of a number of commenters lower down, it is not the best article and is confusing misuse with intrinsically broken.

As far as climate science is concerned, well the hockey stick graph was a result of a misuse of statistics and not anything wrong with statistics. Nor does a misuse of statistics invalidate anything that uses statistics, so whilst there may be misuse it does not bear a huge reflection on the entirety of climate science.

[M Paul Lloyd]

Oh no, don't get me wrong I don't have a problem with statistics as such and the article does point out that statistics have their uses but as you say when misused and relied upon too much they can lead to some rather anomalous results.

[Jamie]

There are two kinds of statistics, the kind you look up and the kind you make up.

[M Paul Lloyd]

There are two kinds of statistics, the kind you look up and the kind you make up.

True Jamie, so very true, but I really worry when they insert algorithms into the equation which can 'tweek' the results quite dramatically.

[Ush]

This is from Wiki:

The significance level is usually denoted by the Greek symbol α (lowercase alpha). Popular levels of significance are 5% (0.05), 1% (0.01) and 0.1% (0.001). If a test of significance gives a p-value lower than the α-level, the null hypothesis is rejected. Such results are informally referred to as 'statistically significant'. For example, if someone argues that "there's only one chance in a thousand this could have happened by coincidence," a 0.001 level of statistical significance is being implied. The lower the significance level, the stronger the evidence required. Choosing level of significance is an arbitrary task, but for many applications, a level of 5% is chosen, for no better reason than that it is conventional.[3][4]

Does this strike any one else as dumb and very unscientific?

This is free to access:

http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124

Why Most Published Research Findings Are False

Summary

There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.

[The Beige Avenger]

Statistical significance is very important...

If you've got a data set that looks to be say, increasing with x, but your errors are big enough to allow for multiple interpretations then the increase is not statistically significant.
Sure, you'd comment on the fact that the data appears to be showing a trend but you'd also have to show your errors to allow other people to see and question the results.

[Ush]

I see what your saying but just arbitrarily choosing a value and then pronouncing if results are statistically significant or not seems very strange to me.

[The Beige Avenger]

I'd be inclined to agree...

[Ush]

Speaking of statistics, I've wanted to teach myself some maths for a while and think it's about time I got started on the task. Can anyone recommend a good introductory textbook on statistics?

[Shadowwolf]

I think that a limited value probably does the job, its whether the latitude is allowed to be excessive or not.

Like almost anything the system is open to abuse by those looking to circumvent the process, it is probably not to an extent that just about everything is at fault but undoubtedly some is below par.

[The Beige Avenger]

ush, it might be worth while checking out some uni's maths departments who might have notes freely available for download and if not you can always email a teacher to ask them if there are any freebies.

I've got a booklet I could forward you but I'm not sure if it's for registered students only.

mathworld (wolfram) is a great maths resource but they jump right into the terminology.

[M Paul Lloyd]

This might have what you want Ush.
http://www.audiencedialogue.net/stats.html

[Ush]

Cheers folks.

[The Beige Avenger]

ps "probability" might be the unit you're after.... it's sometimes bundled in with stats and, IMO, it's actually far more enlightening.