Teaching Claude to review like an editor

A pre-submission diagnostic looks like a peer review but has to invert its norms, because the audience is inverted: when the author is the one asking, exhaustiveness is a form of respect. I built that inversion into a Claude skill using my field's editorial philosophy, real decision letters from top journals in my field, and one instructive bad review, then tested it against an editorial record it had never seen. The corpus, the judgment calls, and the test design are mine. Here is what worked and where it broke.

The problem: fluent reviews that don’t work like reviews

the specimen taught more than the exemplars did

I had an unusual artifact sitting in a folder: an AI-generated peer review of one of my own submitted manuscripts. It looked like a review, with numbered major concerns, minor issues, questions for the authors, and a recommendation. And it exhibited a pathology set anyone who has asked a chatbot to review a paper will recognize. It said the same thing in three places. It hedged performatively, as above. It cited literature into existence with the confidence of a scholar who has read everything, including papers that may not exist. It padded its praise so the ratio would look fair. Individually these are small. Together they are why AI feedback on manuscripts reads like a review but does not function as one.

I am a postdoctoral researcher in management on the academic job market, and pre-submission feedback is the scarcest resource I have. Co-authors are busy, friendly reviews take weeks, and the reviews that matter arrive only after an editor has already made half the decision. I wanted a diagnostic I could run on my own drafts: something that flags everything wrong, explains why, suggests a fix, and holds the standard of the journals I target. So I built one as a Claude skill.

A prompt is a bad place to keep a corpus

version-controlled expertise, not prompt archaeology

I had prompts. Everyone has prompts. A prompt can hold a great deal, including pasted reference material, so the problem is not capacity. The problem is maintenance. Prompts drift: you paste different versions on different days, you forget which file had the good clause, and quality varies with your clipboard hygiene. And what makes feedback good in a specific field is not an instruction like “be rigorous.” It is calibration against what good and bad actually look like there, and that calibration lives in documents that need to be versioned, not in adjectives that need to be remembered.

A Claude skill is a small folder: one instruction file the model always reads when the skill triggers, plus reference files it consults at defined moments, packaged once and installed at the account level. After that, uploading a draft and asking for fresh eyes pulls the whole apparatus in automatically. The gain is persistent, version-controlled packaging of instructions plus calibration material. Infrastructure instead of improvisation.

The raw material: philosophy, exemplars, and one very useful bad review

four ingredients; the mix matters more than any one

• Editorial philosophy. Three Academy of Management Review editor’s essays: Ragins (2015), the field’s flagship statement on developmental reviewing; Ballinger and Johnson (2015) on the four Rs of respect, reasons, recommendations, and recognition; and Lepak (2009). The field’s own account of its standards, written by the people who enforced them.
• Positive exemplars. Real decision letters from top journals in my field. Direct, hierarchically numbered, reasoning before verdict, specific to the page and paragraph.
• A negative exemplar. The AI-generated review, which became the single most useful document in the pile.
• My own protocols. A twelve-section review prompt I had been iterating on, and a Reviewer 2 persona with a one-to-five harshness dial that I ultimately rejected.

The design decision everything follows from: a diagnostic is not a review

the audience inversion: when the author is the reader, exhaustiveness is the respect

The editors’ essays tell reviewers to prioritize five to eight concerns and not to kick dead horses, because the author did not ask for the review and a hundred-item list is demoralizing. Correct advice, for a reviewer writing to an author. But I am the author, and I did ask. When the author is the reader, exhaustiveness is not cruelty. It is the entire point, because the author will triage.

So the skill inverts the reviewing norm on three axes: flag everything, across theory, empirics, prose, numbers, citations, and tables; be direct without social cushioning; and never repeat, because each issue gets exactly one home, handled with the discipline flag, explain, suggest, move on. That last rule is not mine. AOM’s own reviewer guidance says to avoid repeating the same critique in multiple places and to consolidate related points, which is the field telling you that repetition is a defect rather than thoroughness.

I kept the philosophy and dropped the manners. Respect, reasons, and recommendations survive intact. Recognition survives in lean form: name what to protect in revision, never pad praise. There is no harshness dial, because the editors are explicit that the Reviewer 2 posture is bad reviewing rather than rigorous reviewing, and my own letters confirm it: the toughest ones are firm and warm at once, and their toughness comes from the specificity of the reasoning rather than the temperature of the prose. I kept “be real” as the stance and stopped expecting it to do any work. The procedural rules below do that.

The autopsy: eight named anti-patterns became the spec

"don't sound like an AI" is not executable; "never write 'It is worth noting that'" is

The most productive step in the build was treating the AI-generated review as a specimen and cataloguing its failure modes as named anti-patterns, each with a verbatim example and a corrected version. Eight made it in:

1. Performative throat-clearing. The specimen at the top. A social move, not an analytical one.
2. Saying the same thing in three places. One measurement issue appeared as a major concern, again as a question for the authors, and again in a note to the editor. Length without information.
3. Formatting as a substitute for argument. Bold-italic headers announcing importance the content doesn't earn.
4. Theatrical hedge constructions. "It is worth noting that", "To be fair", "I would be remiss not to". The sentence after the hedge is the actual point; write that.
5. Citing-into-existence. Fluent, confident, specific references that may not exist. The encoded rule: point to streams of literature and tell the user to verify; never fabricate specifics.
6. Padding the strengths section so the praise-to-criticism ratio looks balanced.
7. Rhetorical questions as evasion. "Can you provide any evidence that...?" when the honest sentence is "the paper provides no evidence for X."
8. Theatrical section headers. Urgency in the label doing work the content should do.

Naming these with examples matters because “don’t sound like an AI” is not an instruction a model can execute. “Never write ‘It is worth noting that’” is.

Architecture: one file always read, two consulted on demand

progressive disclosure: protocol up front, calibration on call

The skill is three files. SKILL.md, about 270 lines, is always loaded: the stance, the input handling, and a twelve-section protocol. references/tone_and_register.md holds the voice calibration and the eight anti-patterns, consulted before writing. references/exhaustive_checklist.md holds the scanning catalog for the section-by-section sweep: numerical consistency across tables and text, construct drift, hedge language, citation hygiene, per-section and per-method checks, and an opt-in reference style audit keyed to journal conventions, because house styles differ and change, and the model should not guess.

Three moves in the protocol earn their keep more than the rest. First, actual versus claimed contribution, including a required sentence stating what a hostile reader would say the paper is really about. That one sentence is worth the exercise. Second, hypothesis diagnosticity: classify every result as non-diagnostic, suggestive, discriminating, or highly probative. Many papers’ first hypotheses are non-diagnostic, and it is better to hear that from your own tool than from Reviewer 3. Third, cut, rewrite, keep: the review ends with a revision memo, not a mood.

The skill also knows that real drafts are messy. It reads tracked changes with deletions intact, unpacks Word comments from the document XML, and treats bracketed notes as signal rather than noise: a “[TODO: explain why this differs from X]” sitting next to the paper’s core construct is not a reminder, it is an unresolved theoretical problem, and the review says so.

Test against an external editorial record, not vibes

n = 1, but out-of-sample

Most people evaluate AI tools by whether the output sounds smart. I had something better: one of my own manuscripts for which I held the full editorial packet, meaning the decision letter and all three reviewer reports from a top management journal. That packet was produced by independent readers before the skill existed, which makes it an out-of-sample benchmark rather than an answer key. The distinction matters, and my own result shows why: the most important thing the skill found is something no reviewer flagged. Against an answer key that would score as an error.

I ran the installed skill in a fresh chat, so the conversation that built it could not contaminate the execution.

One limitation worth naming rather than burying: the catches are checkable against the editorial packet, but the failure list is my own audit of my own tool, produced in the session that built it. An interested party graded the homework. That is the weakest joint in the evaluation, and the fix is other people running it on their own drafts.

Every failure traced back to a loose instruction

templates create demand; slots get filled

The useful discipline was tracing each failure to the sentence that permitted it. The punted reference check: I had written “always do a bidirectional consistency check,” and the model read “always do” as “always mention.” Version two replaces it with numbered do-it-now steps and forbids the punt phrasing. The padded strengths: I had given a four-item template, and templates create demand, so the model filled the slots. Version two reframes the items as prompts to scan, states that the number is not fixed, and encodes the rule that competence is the floor rather than an asset: if a candidate strength is merely competently executed, exclude it.

The confabulated citations resist a prompt-level fix. The warning was already written down and the behavior survived it. Two lessons. Instructions in the always-loaded file bind harder than instructions in consulted reference files, so the highest-stakes rules belong in the main file. And some behaviors survive prompting entirely, which makes the mitigation procedural on my side: I verify every named citation before acting on one. A tool that is right about the confound and shaky on the bibliography is still enormously useful, as long as you know which is which.

Which leaves two lessons the exercise reinforced. Procedural instructions beat attitudinal ones. And negative exemplars are underrated, because failure is more legible than excellence: the bad review taught the skill more than either editors’ essay did.

What ships is the generic version

calibration is personal; the shipped file is deliberately weaker

The version I use is calibrated on my own material: my manuscripts, my decision letters, anti-patterns drawn from a review of my own paper, checklist examples keyed to the constructs and designs I actually work with. That specificity is most of what makes it good.

None of it ships. Before releasing the skill I genericized all three layers. The decision letters became descriptions of their moves rather than quoted passages. The anti-pattern examples lost the manuscript they came from. The checklist examples lost the constructs that made them concrete. Two different reasons, worth separating: the letters are other people’s writing, produced under an expectation of confidentiality, while the manuscript examples are mine but belong to papers still under review, and I would rather not circulate their internals.

So the download is a scaffold, not a finished instrument, and the honest recommendation is to re-personalize it. Upload the three skill files to Claude along with whatever editorial material you have, whether decision letters from your own submissions, reviewer reports, or a rejection whose phrasing you still remember, and ask it to recalibrate the skill from them. What comes back is tuned to your field’s register and your own recurring failure modes rather than mine. That is the version worth having. If you ever share yours, run the same genericization pass first, for the same two reasons.

If it catches something your co-authors missed, or produces one of the failure modes it was designed against, I want to hear about it. The paper-authoring counterpart is the next build.

References

Ballinger, G. A., & Johnson, R. E. 2015. Editors' comments: Your first AMR review. Academy of Management Review, 40: 315-322.

Lepak, D. 2009. Editor's comments: What is good reviewing? Academy of Management Review, 34: 375-381.

Ragins, B. R. 2015. Editor's comments: Developing our authors. Academy of Management Review, 40: 1-8.