post-hoc power analysis

[Norman B. Grover]

I tested the same hypothesis on three independent data sets by means
of one-tail Mann-Whitney tests, and combined the three p-values using
Fisher's chi-square test for combined probabilities: chi-square ~ -2*ln(p1
*p2*p3) with df=6. Now a journal wants a post-hoc power analysis.
I am against post-hoc power analyses for the obvious reasons, but
this was done for a colleague of mine and I am at a loss as to how to
proceed­--other than to suggest he find another journal!
Is there any way to get even a rough estimate of the power? I was
thinking of non-central chi-square, and have access to the standard
commercial sample-size/power-analysis software and an electronic table of
the non-central chi-square distribution. (Please don't suggest simulation.)

[David Duffy]

Post by Norman B. Grover
I tested the same hypothesis on three independent data sets by means
of one-tail Mann-Whitney tests, and combined the three p-values using
Fisher's chi-square test for combined probabilities: chi-square ~ -2*ln(p1
*p2*p3) with df=6. Now a journal wants a post-hoc power analysis.
Is there any way to get even a rough estimate of the power?
(Please don't suggest simulation.)

Which is of course what you need to do ;) Why not redo the analysis as
an ANOVA or whatever, show the results are comparable to your nonparametric
approach, and give power or CIs about relevant effect measures.

2c, David Duffy.

[Norman B. Grover]

Post by David Duffy

Post by Norman B. Grover
I tested the same hypothesis on three independent data sets by means
of one-tail Mann-Whitney tests, and combined the three p-values using
Fisher's chi-square test for combined probabilities: chi-square ~ -2*ln(p1
*p2*p3) with df=6. Now a journal wants a post-hoc power analysis.
Is there any way to get even a rough estimate of the power?
(Please don't suggest simulation.)

Which is of course what you need to do ;) Why not redo the analysis as
an ANOVA or whatever, show the results are comparable to your nonparametric
approach, and give power or CIs about relevant effect measures.
2c, David Duffy.

1. Because there are over 10 such experiments.
2. Because the sample sizes are very small (mostly < 5) and so the
parametric results will most likely not be comparable.
3. Because there is a limit to my friendship ;)

[Norman B. Grover]

Post by Norman B. Grover
I tested the same hypothesis on three independent data sets by means
of one-tail Mann-Whitney tests, and combined the three p-values using
Fisher's chi-square test for combined probabilities: chi-square ~ -2*ln(p1
*p2*p3) with df=6. Now a journal wants a post-hoc power analysis.
I am against post-hoc power analyses for the obvious reasons, but
this was done for a colleague of mine and I am at a loss as to how to
proceed­--other than to suggest he find another journal!
Is there any way to get even a rough estimate of the power? I was
thinking of non-central chi-square, and have access to the standard
commercial sample-size/power-analysis software and an electronic table of
the non-central chi-square distribution. (Please don't suggest simulation.)

On further thought, a simple solution would be to compute the noncentrality
parameter lambda from p1, p2 and p3, if that were possible, and use the non-
central chi-square distribution.

[Rich Ulrich]

On Sat, 20 Dec 2014 11:43:15 +0200, Norman B. Grover

Post by Norman B. Grover

Post by Norman B. Grover
I tested the same hypothesis on three independent data sets by means
of one-tail Mann-Whitney tests, and combined the three p-values using
Fisher's chi-square test for combined probabilities: chi-square ~ -2*ln(p1
*p2*p3) with df=6. Now a journal wants a post-hoc power analysis.
I am against post-hoc power analyses for the obvious reasons, but
this was done for a colleague of mine and I am at a loss as to how to
proceed­--other than to suggest he find another journal!
Is there any way to get even a rough estimate of the power? I was
thinking of non-central chi-square, and have access to the standard
commercial sample-size/power-analysis software and an electronic table of
the non-central chi-square distribution. (Please don't suggest simulation.)

On further thought, a simple solution would be to compute the noncentrality
parameter lambda from p1, p2 and p3, if that were possible, and use the non-
central chi-square distribution.

Don't you get to ignore the orignal p's? The power is just a gross
re-statement of the observed p-value for the observed test situation.

I think you get your noncentrality from the overall N and the observed
chi-square. And some tables might incorporate the N, so that the
chi-square (or chi-square divided by d.f.?) is actually the value
that you want.

[Norman B. Grover]

Post by Rich Ulrich
On Sat, 20 Dec 2014 11:43:15 +0200, Norman B. Grover

Post by Norman B. Grover

Post by Norman B. Grover
I tested the same hypothesis on three independent data sets by means
of one-tail Mann-Whitney tests, and combined the three p-values using
Fisher's chi-square test for combined probabilities: chi-square ~ -2*ln(p1
*p2*p3) with df=6. Now a journal wants a post-hoc power analysis.
I am against post-hoc power analyses for the obvious reasons, but
this was done for a colleague of mine and I am at a loss as to how to
proceed­--other than to suggest he find another journal!
Is there any way to get even a rough estimate of the power? I was
thinking of non-central chi-square, and have access to the standard
commercial sample-size/power-analysis software and an electronic table of
the non-central chi-square distribution. (Please don't suggest simulation.)

On further thought, a simple solution would be to compute the noncentrality
parameter lambda from p1, p2 and p3, if that were possible, and use the non-
central chi-square distribution.

Don't you get to ignore the orignal p's? The power is just a gross
re-statement of the observed p-value for the observed test situation.
I think you get your noncentrality from the overall N and the observed
chi-square. And some tables might incorporate the N, so that the
chi-square (or chi-square divided by d.f.?) is actually the value
that you want.

What I had in mind was to use Fisher's chi-square for combined probabilities
to estimate lambda: lambda = -2*ln(p1*p2*p3)-6, since E[noncentral chi-
square] = df + lambda. Is that being naive?

[Rich Ulrich]

On Sat, 20 Dec 2014 23:21:55 +0200, Norman B. Grover

Post by Norman B. Grover

Post by Rich Ulrich
On Sat, 20 Dec 2014 11:43:15 +0200, Norman B. Grover

Post by Norman B. Grover

Post by Norman B. Grover
I tested the same hypothesis on three independent data sets by means
of one-tail Mann-Whitney tests, and combined the three p-values using
Fisher's chi-square test for combined probabilities: chi-square ~ -2*ln(p1
*p2*p3) with df=6. Now a journal wants a post-hoc power analysis.
I am against post-hoc power analyses for the obvious reasons, but
this was done for a colleague of mine and I am at a loss as to how to
proceed­--other than to suggest he find another journal!
Is there any way to get even a rough estimate of the power? I was
thinking of non-central chi-square, and have access to the standard
commercial sample-size/power-analysis software and an electronic table of
the non-central chi-square distribution. (Please don't suggest simulation.)

On further thought, a simple solution would be to compute the noncentrality
parameter lambda from p1, p2 and p3, if that were possible, and use the non-
central chi-square distribution.

Don't you get to ignore the orignal p's? The power is just a gross
re-statement of the observed p-value for the observed test situation.
I think you get your noncentrality from the overall N and the observed
chi-square. And some tables might incorporate the N, so that the
chi-square (or chi-square divided by d.f.?) is actually the value
that you want.

What I had in mind was to use Fisher's chi-square for combined probabilities
to estimate lambda: lambda = -2*ln(p1*p2*p3)-6, since E[noncentral chi-
square] = df + lambda. Is that being naive?

Okay, you have a lambda defined that way. I don't remember using
something like that lambda, but it might be my memory or it might be
that my few excursions into hand-computed land did not do chi-squared.

It might be clever to know that; it would only be naive to *trust*
that your source for the non-central is using that parameterization
for what it reports.

[Norman B. Grover]

Post by Rich Ulrich
On Sat, 20 Dec 2014 23:21:55 +0200, Norman B. Grover

Post by Norman B. Grover

Post by Rich Ulrich
On Sat, 20 Dec 2014 11:43:15 +0200, Norman B. Grover

Post by Norman B. Grover

Post by Norman B. Grover
I tested the same hypothesis on three independent data sets by means
of one-tail Mann-Whitney tests, and combined the three p-values using
Fisher's chi-square test for combined probabilities: chi-square ~ -2*ln(p1
*p2*p3) with df=6. Now a journal wants a post-hoc power analysis.
I am against post-hoc power analyses for the obvious reasons, but
this was done for a colleague of mine and I am at a loss as to how to
proceed­--other than to suggest he find another journal!
Is there any way to get even a rough estimate of the power? I was
thinking of non-central chi-square, and have access to the standard
commercial sample-size/power-analysis software and an electronic table of
the non-central chi-square distribution. (Please don't suggest simulation.)

On further thought, a simple solution would be to compute the noncentrality
parameter lambda from p1, p2 and p3, if that were possible, and use the non-
central chi-square distribution.

Don't you get to ignore the orignal p's? The power is just a gross
re-statement of the observed p-value for the observed test situation.
I think you get your noncentrality from the overall N and the observed
chi-square. And some tables might incorporate the N, so that the
chi-square (or chi-square divided by d.f.?) is actually the value
that you want.

What I had in mind was to use Fisher's chi-square for combined probabilities
to estimate lambda: lambda = -2*ln(p1*p2*p3)-6, since E[noncentral chi-
square] = df + lambda. Is that being naive?

Okay, you have a lambda defined that way. I don't remember using
something like that lambda, but it might be my memory or it might be
that my few excursions into hand-computed land did not do chi-squared.
It might be clever to know that; it would only be naive to *trust*
that your source for the non-central is using that parameterization
for what it reports.

That reply is very reassuring, especially coming from you. I plan to use
G*Power. One of the options there is to compute the post-hoc power using a
'Generic chi-square test', given lambda, alpha and df. But I will first make
sure that its results are consistent with those from other software.
Thank you Rich, thank you very much.

[Norman B. Grover]

Post by Norman B. Grover

Post by Rich Ulrich
On Sat, 20 Dec 2014 11:43:15 +0200, Norman B. Grover

Post by Norman B. Grover

Post by Norman B. Grover
I tested the same hypothesis on three independent data sets by means
of one-tail Mann-Whitney tests, and combined the three p-values using
Fisher's chi-square test for combined probabilities: chi-square ~ -2*ln(p1
*p2*p3) with df=6. Now a journal wants a post-hoc power analysis.
I am against post-hoc power analyses for the obvious reasons, but
this was done for a colleague of mine and I am at a loss as to how to
proceed­--other than to suggest he find another journal!
Is there any way to get even a rough estimate of the power? I was
thinking of non-central chi-square, and have access to the standard
commercial sample-size/power-analysis software and an electronic table of
the non-central chi-square distribution. (Please don't suggest simulation.)

On further thought, a simple solution would be to compute the noncentrality
parameter lambda from p1, p2 and p3, if that were possible, and use the non-
central chi-square distribution.

Don't you get to ignore the orignal p's? The power is just a gross
re-statement of the observed p-value for the observed test situation.
I think you get your noncentrality from the overall N and the observed
chi-square. And some tables might incorporate the N, so that the
chi-square (or chi-square divided by d.f.?) is actually the value
that you want.

What I had in mind was to use Fisher's chi-square for combined probabilities
to estimate lambda: lambda = -2*ln(p1*p2*p3)-6, since E[noncentral chi-
square] = df + lambda. Is that being naive?

I got it wrong.
Fisher's chi-square for combined probabilities assumes that the p-value for
each independent test ~ U(0,1). This is of course true under H0 but not, as
I now realize, under H1.
What I have to use instead is a combining method like Tippett's or
Stouffer's (or Stouffer's weighted). The former keeps the original p-value
but modifies the alpha and the latter transforms the p-values into z's (or
weighted z's).
Sorry for wasting bandwidth.