tag:blogger.com,1999:blog-3621026.post2978835709245958154..comments2021-07-30T19:18:46.717+02:00Comments on Robert's Stochastic thoughts: Roberthttp://www.blogger.com/profile/14455788499385673507noreply@blogger.comBlogger2125tag:blogger.com,1999:blog-3621026.post-45595375253962676742008-03-11T15:58:00.000+01:002008-03-11T15:58:00.000+01:00Actually I think I understand how Kirsch et al got...Actually I think I understand how Kirsch et al got their results. I get a weighted average difference of change of 1.<BR/><BR/>notation: changeij is the average change in HRSD of patients in study i who got the SSRI (j=1) or the placebo (j=0). Nij is the sample size of patients in trial i who get j pills.<BR/><BR/>In one calculation I used changeij/dij as the standard deviation of the change and thus (changeij/dij)^2/Nij as the estimated variance of the average change. The *separately* for drug and placebo data, I calculated the precision weighted average over i of changeij. This gave me an average change of 7.809 for the placebo and 9.592 for the SSRI treated for a difference of 1.78.<BR/><BR/>I guess this is what they did. I the confidence intervals are screwy and d is as described in the paper.Roberthttps://www.blogger.com/profile/14455788499385673507noreply@blogger.comtag:blogger.com,1999:blog-3621026.post-26363650200538474752008-03-11T14:40:00.000+01:002008-03-11T14:40:00.000+01:00Hi, interesting stuff, but I'm not sure if you can...Hi, interesting stuff, but I'm not sure if you can explain the findings of Kirsch et al by this alone - essentially you are pointing out fairly subtle differences in detected effect sizes, but Kirsch et al find an effect size that is substantially different from you, or me.<BR/><BR/>As I discuss <A HREF="http://pyjamasinbananas.blogspot.com/2008/03/final-analysis.html#c1289319061217140175" REL="nofollow">here</A>, I'm not convinced that Kirsch et al found their effect size measure 'd' by dividing the change score by the SD of the change, even though it does indeed look like that is what they are claiming in the paper - which is why I derived my estimates for the SD of the change from the confidence intervals.<BR/><BR/>What is interesting is that you find pretty similar effect sizes to me, and I just can't see how Kirsch et al arrived at their figure of a change of 1.8 in the HRSD - just looking at the numbers for the studies I can't see how you could combine them together to reach their figure, and look at the scatter plot and regression <A HREF="http://pyjamasinbananas.blogspot.com/2008/03/regression-in-depression.html" REL="nofollow">here</A>, how are they getting an estimated effect size that seems so far outside the region of the data?<BR/><BR/>I'd also point out that their measure 'd', is a pretty odd measure of effect size, if it is the HRSD change/change SD that is a very strange measure indeed, but even a properly formed SMD (dividing by estimates of the SD of the HRSD score, not the change in the HRSD score) makes it difficult to assign any meaning to subtracting the drug 'd' from the placebo 'd' to come up with an effect size, and it is difficult to compare such a measure with the Cohen d, which is a true SMD.<BR/><BR/>But even then, <A HREF="http://pyjamasinbananas.blogspot.com/2008/03/final-analysis.html#c106287257847544247" REL="nofollow">a cursory look</A> at the relationship between the difference between 'd's and the difference between change scores still suggests an effect size much larger than 1.8 HRSD points, indeed more like the 2.7 points that every other method of analysis seems to suggest.<BR/><BR/>In actual fact I find that calculating a true meta-analytic SMD (where it is the difference in change scores that is normalised to the SDs of the change scores) that can be compared with a Cohen d showed quite a small effect size (.25) so I don't know why, if they sought to deliberately underestimate the 'clinical efficacy' of anti-depressants, they didn't just compare that to NICE's Cohen d > .5 criterion.<BR/><BR/>You'd think, in these days of online open access journals they'd have given a bit more detail and supplemental material so us obsessive types could tell exactly what it was they did - and then slag it off!pjhttps://www.blogger.com/profile/06832177812057826894noreply@blogger.com