The Rebuttal - Writing a robust defence of a journal submission

I’ve had a fair amount of experience in negotiating a publication out of largely negative reviews, and also a fair amount of experience in writing reviews for Journal publications. The best guidance I’ve read about the process of writing a rebuttal is here, a process which I have unwittingly duplicated over the years. I will, however, attempt to rephrase these points in terms of my personal priorities, and embellish these with examples.

Introduction

It is important to distinguish between the reviewing process of a journal and a Conference: traditionally conference papers have a quick turnaround, and in many disciplines are reviewed purely on a short abstract rather than a full paper. Computer Graphics is an exception: authors are often enticed to attend conferences with the promise that their work will be published in a journal: for this reason full papers are almost always required. The implication of this is that the there is little flexibility - the paper must be publishable or near-publishable in it’s submitted form as there is little time to make significant changes. In my opinion, this limits the scope of the rebuttal to the acceptance of minor corrections and clarifications of misunderstandings (which will often still not result in a satisfactory outcome).

The type of rebuttal I’ll be writing about is one which accompanies a journal paper resubmission, which implies that you’ve already managed to get your foot in the door with a reasonably coherent submission. For journal submissions, my experience is that reviewers have the flexibility to “take a punt” on research that may be under-developed, poorly presented and simply not currently publishable - as long as there is an original idea lurking in the manuscript which could represent a significant contribution. Here is an actual reviewer result from a journal submission:

Recommendation: Author Should Prepare A Major Revision For A Second Review
Comments: As mentioned above, I find the combination of ideas in this paper interesting and useful, and I believe that eventually the method will merit publication. However, the paper is poorly written and some recent ﴾and also not so recent﴿ work is omitted and should be discussed.

You can't get away with that in a SIGGRAPH submission. The reviewer has seen the potential cunningly hidden under the poor presentation, and encouraged me to resubmit for a second review. My opinion of this process is that the reviewer him/herself has actually contributed substantially to the production of this manuscript in identifying the importance and encouraging publication, which is a win for everyone.

This should set the tone for your rebuttal: you must be both respectful and authoritative:

• Respect is vital if you want the reviewers to accept a 2nd draft: do not be dismissive of their ignorant or ill-informed remarks. Their discipline area may be quite distant from your own, and their misconceptions may arise from commonly held wisdom in another discipline area - in effect, you might both be correct. If you’re rude, it insults both the reviewer (who has put in the time to read and assess your work and is often a prominent researcher in the field) and the editor (who’s judgement of reviewer allocation is called into question).
• Authority is important in establishing your expertise in the area (which you reinforce by using evidence, see below), and also essential for reassuring the editor and reviewers alike that your contribution is going to be significant.

Below I’ll go into a little more depth of the process I use of writing a rebuttal.

Classify reviewer comments

The first and probably most vital thing you’ll need to is to actually read the review and identify in bullet points all the specific good and bad things from each reviewer. Then group together similar points made by different reviewers. As I’m ridiculously organised I’d often put this in some sort of table indicating the priority, if it is a positive or negative point, which reviewer(s) made the point, and details of the praise / criticism.

Ranking the priority is important: you should address the concerns of the reviewers in the order of the importance. This should be in the order of *their* perceived importance rather than their own. The clue is in the text - if it says *it would be nice if* is obviously less urgent than *is essential for publication*. However, do not be fooled: you **must address every negative comment made by every reviewer**.

Grouping similar comments together is going to be very important if you’re going to keep to the rebuttal page limit (which is often 5). Some flexibility is often required in grouping together similar but not exactly similar comments. Make sure you address both variants in your rebuttal.

Construct your argument

A rebuttal, like a paper or grant proposal, must be constructed rather than simply written. This distinction is vital in the presentation of any argument - particularly if you’re in a debate and you don’t want to sound like a rambling loon. You will need to construct an argument where you deconstruct and address each negative statement. There are, I suppose, four possible ways to argue a point. In each case, your argument must be evidence based, otherwise it may simply be rejected out of hand.

1. Agree and implement: you thank the reviewer for the excellent suggestion, and provide a page/line number of where your discussed / implemented this in your paper.

C6.1: Especially it is not clear how the energy scaling is written for a general system, the expression of $u_{local}$ for energy scaling and time scaling could be given in appendix.
On this reviewers recommendation, we have included the expressions for $u_{local}$ in the main text body, and included a substantial appendix in which the local controllers are derived (Appendix B). We feel this is a useful addition to the paper, especially for readers not familiar with the application of Lie group symmetry to dynamic systems.

2. Agree, but argue that it is beyond the scope of the paper. Reviewers sometimes want it all. Requests can be unreasonable for a single incremental unit of research (i.e. the journal paper). Sometimes a reality check is necessary. This is a difficult argument to make, as it requires to some extent an argument to be made that not only argues for the quality of the results in the paper to be sufficient and representative, but also that the additional work suggested will not add anything substantive to the presentation. Make sure that your argument demonstrates understanding of the subject area, and you provide evidence to justify this point of view. Here are a couple of examples:

C2.1: The drawback is that the illustration concerns very simple system with a low number of state and the extension of the calculation to the case of a 3D humanoid robot is not trivial. For example the calculation of the basin of attraction for a large system is not obvious. It is often difficult to explain clearly new idea in the case of complex system because the complexity can hid the interest of the strategy, thus this present form is convenient but I would appreciate more detailed calculation in the appendix.
We agree that this is an important topic requiring further investigation. Due to limited space we have not discussed the various possible approaches by which this method could be scaled up to more complex systems. In the work of Ames, Gregg and Spong 2007, the complexity is resolved by dividing the system into many components which are coupled together through mechanical coupling. Another approach is to treat additional degrees of freedom as noise and discard them. We do not yet know the best approach, but it will certainly be the subject of future work.

C3.3: My main complaint is the vague reference to the actual deformable model used in the paper, and the constant confusion between Laplacian editing, LSmeshes and other (nonlinear) surface representations.
This opens up an interesting academic discussion, even if it is about semantics. We have defined Least squares meshes in the introduction as “anchor points and a matrix encoding the differential and topological properties of the surface” (slightly changed from original version), which extends the LS mesh definition of Sorkine and Cohen-Or [5], which is “a planar graph with arbitrary connectivity and a sparse set of control points with geometry”. Effectively, we consider a LS mesh as a form which can be deformed (the representation) using Laplacian mesh editing (the action). Using this definition enables us to use alternative differential operators with our reconstruction. Section 4 has been added to explain what operators are used when and why.

3. Disagree proactively: This is the situation when the reviewer misunderstands some aspect of your work and do not feel that this needs to be clarified further in your text. You need to phrase your response in a manner that is authoritative but not critical and also must remain respectful of their expertise.

C3.5: Last paragraph of section A. You wrote that passive dynamic models use significantly lower energy compared with ZMP-based walkers like ASIMO. Lower passive dynamic models have less joints than the ASIMO I believe.
The differences in the published numbers of the cost of transport of ZMP robots vs passive dynamic models are quite stark, even taking the number of joints into account. BigDog, for example, uses between 15 and 30 times the amount of energy to the Cornell Ranger, which is still not as efficient as a real human. The lower energy consumption relates to the highly efficient design of the passive walker rather than the number of joints. The passive biped of Collins and Ruina [12] has both knees and arms, and consumes no energy when walking downhill.

4. A dismissive response: While this should rarely be used, you do occasionally have to respond to a reviewer that may not have read your article, knows nothing about the area, or is being unprofessional in their assessment. I believe that a polite beat-down is required in these cases, establishing the ignorance of the reviewer but also your authority in the irrelevance of their comment. This could be controversial, but I think it may assist the editor in identifying reviewer comments which are simply not worth the time of day. Here are some of my favourites:

C4.1: There are many claims which are not justified or are incorrect.
It is not clear which claims the reviewer is referring to, so we are unable to provide rebuttal or corrections relating to this point.

C4.2: There is no demonstration of the system and results are largely anecdotal and unconvincing.
We demonstrated our method with 3 different dynamic models. These are not anecdotal, but are genuine dynamic models used to drive and balance existing robots. In addition, we have justified our methodology by evaluating the energy efficiency and the stability of the dynamic system, and by comparison with the motion adaptation strategy of a real human walker.

C4.3: The methodology is unclear with no mention of the actual structure of the animated models used and how they are driven.
In the Applications section we clearly define the dynamic model for the bouncing ball. The walking model is a modified version of the model of Chen [9], as explained in the paper.

...and on it goes.

Sources of Evidence

At the review stage, only the following forms of evidence would really be admissible, in decreasing order of authority:

1. An existing publication: A reputable publication is probably your best form of evidence when refuting a reviewer argument. This might not necessary be from a directly related subject area, any may be outside of the reviewers discipline area. Here’s a really good example:

C3.3: Under Neural Computation, you wrote "This makes it impossible for the neural system to carry out the complex computation necessary for real-time optimisation". Really?
This is an agreed hypothesis amongst the biological motor control community. Good references for this are the book by Latash and the paper of Nishikawa et al. This is easy to justify: we observe that animals with very simple neural systems are capable of controlling complex motion. Even flagellates (for example, sperm), a class of very simple organisms are capable of a swimming motion by whipping a flagellum (whip-like tail). However, current work in simulating dynamic models which include muscle actuation using optimisation can take many days on large clusters of computers.

2. **A comment from another reviewer**: A good reason to keep positive comments handy is that they can be turned into evidence against another reviewers argument. This is a powerful device for negating a criticism as it plays the reviewers off against each other. Here's a reasonable example:

C5.6: Besides, I consider it not necessary to detail so much on the least-squares meshes and Principle Component Analysis in the related work, as they are commonly familiar to the community.
These sections were requested by other reviewer(s). They also serve the double purpose of defining terminology used within the paper.

3. **Evidence found within your paper**: Your paper is currently under scrutiny to deduce whether it is sufficient for publication. By citing evidence within the paper you are treading a dangerous path, so ensure that your evidence is strong. In most circumstances, it might be a prudent approach to use the reviewers comments to tighten up your evidence, produce additional results etc in order to prove your case. 4. **Common sense**: I don't think I've ever had to do this. In some guides they recommend that an emotion appeal may be an effective rebuttal strategy, but I would strongly recommend avoiding such an approach for scientific publication.

The Cover Letter

A cover letter (or email or whatever) is a good opportunity to summarise the general trends of the reviewers, and also to possibly address any of the serious reviewer concerns in a respectful manner. I have also used the cover letter as an opportunity to address any outstanding queries which may arise due to conflict between the reviewers, or space issues. Here is an example:

We are thankful to the reviewers comments, which are largely very positive and helpful. Many of their comments are very interesting, questioning fundamental assumptions relating to human and synthetic locomotion. These arise from the multi-disciplinary nature of the work, and we have addressed these by referencing work in complimentary disciplines such as motor control, neurobiology and robotics. The substantive changes we have included are:

1. We have provided a full derivation of the expressions for the local controller parameters (Appendix B), as requested by reviewer 5. In addition, these local controllers are included under the relevant application sections in the main text body.

2. We have included an introduction to Lie group symmetry applied to dynamic systems in Appendix A.

3. We have added a figure demonstrating how the ball would bounce without being coupled to a neural oscillator.

It should be noted that amendments to the paper have taken the form of including additional material rather than correcting factual errors. One area of concern is what background needs to be incorporated in a final version to be accessible to the reader. Currently the paper contains an introduction to dynamic systems, which was criticised by Reviewer 1 for being to obvious. Additionally, Reviewer 4 requested a definition of Lie group symmetry, which has necessitated the inclusion of a introduction to the application of Lie group symmetry to differential equations (Appendix A). Depending on space limitations, these sections can be included in the main text, as an appendix, or removed (as adequate references were provided), but they do provide the reader with an accessible and intuitive introduction to these tricky topics, and we hope they can be accommodated in the final text.

Conclusion

In answer to the question "Should I Write a Rebuttal", it is dependent on whether you're able to provide sufficient defence of your article, or are able to revise your article in a manner that meets with the reviewers criteria. I think of writing a paper as trying to plug holes in a dam wall.

The first thing an author needs to accept is: there are probably always going to be holes - your results might not contain sufficient examples, your comparisons may not be with the absolutely latest work, your method may not be rigorously proven - although you should obviously do your best to minimise these. The secret of writing the paper is to ensure that there are as few holes as possible, so the reviewers are satisfied that you’ve done a good enough job. The purpose of the rebuttal is to plug as many of the holes as possible, from largest to smallest. Even if you choose not to submit a rebuttal (I’ve been there) I would strongly recommend going through the exercise above anyway to get a better understanding of how you can improve your research.

So in summary:

• Be respectful and authoritative in tone.
• Read and classify the reviewers comments.
• Address every negative reviewer comment.
• Provide evidence in constructing your argument.
• Provide a broad summary of changes in the cover letter.