- Paper with responses worked in (Last updated on 2016 April 4)
- I find the observation discussed in this paper to be interesting but hardly worth a separate publication, especially since the authors correctly point out that the issue is discussed in the statistics literature (ref 23). For the paper to be considered for the publication the authors need to clearly point out what is innovative in this study.
>> We appreciate the reviewer's candid assessment. Of course we disagree that it isn't worth a separate publication, but agree completely that it is important that the paper make it clear and unambiguous as to why it is worth publishing. Clearly the previous version did not do this. For this reason we have improved the introduction and conclusion to make it explicit why this study is indeed important and should be of value to the readership. Since the journal is Nuclear Instruments and Methods, its primary audience is scientists who desire important information about methods. Our specific goal is to assist scientists with measurement methodology, and in particular for results that are directly applicable for future collider asymmetry measurements. The authors note from personal experience a number of examples of ongoing/unpublished analyses where there is a real danger that studies can potentially make a wrong assumption (and where analyzers have made this wrong assumption). While part of the understanding of why this is a potential problem is discussed in the statistics literature (Cauchy distribution), we know of no reference that clearly points out this pitfall in reference to asymmetry measurements. Indeed, many scientists we discussed this result with did not know of this statistics example. As such, this problem is likely to be overlooked by most practicing physicists. The problem isn’t that they don’t know about the Cauchy distribution, it’s that they don’t know to worry that they might have a problem. We aim to aid researchers in being made aware of this potential pitfall. Our study provides them with an understanding of both when and why they are in danger, and what they can do to avoid making a bad measurement. It is extremely useful when you can use a simple technique and get it right, and it is equally useful to know when doing the simple thing will get it very wrong. The bottom line is that we see this as a highly relevant study; this has already been relevant for measurements made at the Tevatron, and has potential to be more generally applicable at various collider experiments. We see this important tool in the experimental scientist’s arsenal as highly relevant and useful and therefore worthy of publication in NIM.
For convenience, we quote the first paragraph of the introduction and the conclusion here:
- Introduction:
Measurements of production asymmetries have a long history at colliders [1–21], so examination of the experimental techniques used to make them is important. Most measurements are performed by first measuring the asymmetry within a restricted geometric region – the region covered by the detector – and then extrapolating to the inclusive region. In some cases an extrapolation based on a constant multiplicative factor is advantageous, but a potential pitfall exists in estimating the multiplicative factor via simulations. Because this sort of technique is widely applicable to experimental measurements, we explore it in detail here and identify where and why this potential pitfall arises.
- Conclusion:
We have studied the use of a simple multiplicative extrapolation method in asymmetry measurements. This method has already been used for measurements made at the Fermilab Tevatron of the tt¯ forward-backward asymmetry, and has potential for wide use. Perhaps most important for future experiments is that, if the correction factor and its uncertainty are to be estimated from a simulated sample, more statistics than expected may be needed, especially when the simulation yields small asymmetry values. We find that the number of simulated events needed for reliable measurements rises as 1/(Ainclusive)^2
- This paper sheds light onto issues encountered when extrapolating measured cross sections from a visible phase space to the full phase space needed to compare most easily to theoretical predictions. Hence, it adds knowledge to a topic which is currently under intense discussion in the experimental communities given the huge sample sizes of the continued run of the LHC. The paper is also of interest for the legacy measurements of the asymmetries at the Tevatron and I propose to publish this manuscript with modifications as suggested by my comments given below. I split my comments into comments on the content (given first), purely editorial and purely cosmetic comments on Figures and References at the very last.
>> Thank you for your comments. We are gratified that the Reviewer clearly understands the intent of this paper, and confirms our belief that it is important and should be published. We have worked hard to address all of the concerns pointed out to us. See specific responses below.
CONTENT COMMENTS
- Strongly suggest to add a paragraph / few sentences on the fact that nowadays even for leptonic asymmetries the limited detector resolution and hence the migrations between bins are of more importance. Typically these require to use more sophisticated methods of correcting the data and extrapolating to the full phase space / region. One of the most widely used techniques is matrix unfolding with some sort of regularization. Typically first detector effects are corrected for in a visible phase space while the extrapolation is done purely based on MC to the parton or full phase space / region while both numbers are published. While I don't argue for adding that topic in here I think a brief discussion of this development is very helpful for the reader to digest the topic discussed here in this paper and its applications. The best place in my view is either l8-13 and/or l23-31, or briefly introduce it in one additional sentence in l8-13 and expand a little bit in l23-31.
>> We agree with the reviewer that the issues of bin migration are clearly important. For this reason we have added two citations to the "more advanced" techniques:
arXiv:1402.3803
arXiv:1501.07383v2
and a brief comment on line 29. On the other hand, we feel that this issue is just one of many that need to be addressed in doing any measurement. Our manuscript is not meant to survey the various methods that are currently being used. However, because it is relevant and important for some readers, we have added a brief discussion in Reference [25].
- For the conclusions I also suggest (along the same lines as the previous comment) to mention in one sentence the more recent developments on how these measurements are done and how this applies
>> See above comment, discussion added into Reference [25].
- l80 Regarding the "significant systematic uncertainty", this is typically encapsulated in the limited MC statistics category or the Unfolding/Extrapolation method category. Is that correct ? of course while employing large amounts of MC statistics to avoid issues and ensure stable methods.
>> Yes, in principle this systematic would be accounted for in a "limited MC statistics" uncertainty. However, what we are reporting is that naive methods are almost certain to underestimate it and in this paper we've shown that the amount of MC statistics goes to infinity quickly as AFB-->0, so in some circumstances it may not even be feasible no matter how large the MC sample is. In the extreme case, the uncertainty actually becomes infinite! The primary problem is that a naive reader using simple binomial statistics will get the wrong answer, by a lot, and not realize it's wrong and in some cases asymmetric.
- l87 Suggest to move the text from l114-117 to l87, so that the reader has an idea of the properties of the Cauchy distribution without looking up the reference.
>> Done.
- l92 "somewhat arbitrary": can this not be quantified, e.g. state that 0.5 is X sigma away from the nice Gaussian in the high-statistics-region ? Otherwise it begs the question of why not using 0.65, which indeed I guess does not change the message as you say, but shuffles a lot of PEs around (looking at Figure 4b).
>>Replaced the original text:
"The choice here of \(R<0.5\) is somewhat arbitrary, and the results do not depend on this choice. However, it is chosen to capture information on the lower tail of the \(R\) distribution and since it is many standard deviations away from the large \(N\) answer we require \(f\approx0\) for a reliable measurement of \(R\)."
with:
"The results here do not depend qualitatively on the choice of \(R<0.5\). This inequality is simply chosen to capture information on the lower tail of the \(R\) distribution; since \(R=0.5\) is many standard deviations below the mean from the large \(N\) answer, we can require \(f\approx0\) for a reliable measurement of \(R\)."
EDITORIAL COMMENTS
- l8 Suggest to add the refs to asymmetry measurements already here and not only in l33/27, but right after "…at colliders [xxx], …"
>> Done.
- l8 Can "some" not be a bit more specific, suggest to re-phrase to "the relevant techniques at the Tevatron" or something along those lines.
>> Dropped "some of" in the second sentence. It is generically important to study the techniques. In this paper, we have studied one of them, contributing to this overall important program of studying the techniques.
- l24 The math typeset of "C=A^{incl…}…" looks odd with no space around "=" sign, suggest to fix.
>> Fixed.
- l36 Suggest to use just d\sigma / dx instead of frac{}{} since it makes symbols tiny.
>> Fixed.
- l37,47,etc. "Appendix A" looks like an additional space/blank added between Appendix and letter A… maybe a not-needed \xspace or so ?
>> Fixed.
- l78-80 Suggest to replace the second "thus" with a "hence", avoids repetition.
>> Fixed.
- l99 Suggest to avoid a 1 line text after a Figure and right before a page break.
>> Fixed. We will work with the journal to make sure all of these typesetting issues are worked out.
- l135/6 "standard error propagation techniques", Suggest to just use standard error propagation since that is what it is. It is not a specific technique I would say.
>> Fixed.
REFERENCE COMMENTS
- Can the references be checked once more carefully, I encountered several issues and it is not clear to me if intentionally or by accident. For references are not "balanced" means citing same work of other collaborations than CDF. For example the leptonic asymmetry measurement by D0 is not mentioned (Phys. Rev. D 90, 072001) or there is a typo in Reference [4], which is the combined Tevatron cross section.
>> The reviewer is quite right and we are indebted to them for pointing these out. We went through them again, most notably fixed the typo in Ref. [4], got rid of two duplications (one which was pointed out by the reviewer), they should all be good now.
Added five citations (one replaced which was a typo).
- [4] if intended to be Tevatron combination needs CDF & D0 collaboration.
>> Fixed, this was a typo, and indeed meant to be the suggested D0 paper.
- I believe References [5] and [18] are double.
>> Indeed they were. Got rid of the duplication.
FIGURE COMMENTS
- l123 Maybe I am missing something, but why does it need to be yet another value for \mu ? It would be more consistent and better to follow if again 10^-3 or 0.1 is used.
>> Updated to use mu=1E-3.
- Figure 1: Suggest to overlay a) and b) into one figure and use different colors/line style, that way the differences are easier to see/grasp.
- Figure 3,4: Suggest to overlay these two sub-figures into one with different colors. Given the very different x-axis scale it makes the message much more prominent since a) will look like a really sharp peak in b). Similar to what is done in Figure 8, where a) and b) have the same scale.
>> Updated Figs 1,3 to be overlaid; we decided to remove Figure 4 from the manuscript completely.
Figure 2,3,4,6,7,8: These look like jpegs and not eps files, if at all possible suggest to use eps. my printer messes resolution/fonts/tick marks up a bit. Maybe this is also solved with the final journal typeset.
>> Fixed.