Title: ENTROPY DYNAMICS DO NOT PROVIDE RELIABLE EXECUTION-FREE SELECTION SIGNALS FOR CODE GENERATION FARS
PDF: edis-code-bestofn.pdf
Score: 4.2
Verdict: Reject
Confidence: 0.60
Elapsed: 53.9s

Strengths:
1. Pre-registered success criterion with explicit refutation condition (Section 3.4) is a methodological strength rarely seen in ML papers. The criterion requires nEDIS to outperform both confound baselines on both benchmarks, and the authors honor the refutation when it fails on MBPP vs Length-only (p=1.0, Table 2). This prevents post-hoc rationalization and strengthens the credibility of the negative result.
2. Thorough failure mode analysis identifying entropy sparsity (88.3% zero entropy values, Section 4.4) as a mechanistic explanation for why EDIS transfers poorly from math to code. This is a concrete, measurable finding that future work can directly account for, rather than a vague 'it didn't work' conclusion.
3. The optimization paradox analysis (Section 4.4) is a valuable insight: nEDIS v2 inverts the selection direction (argmin→argmax) and multiplies by length T, effectively converting the method into a length-bias proxy with no theoretical justification. This honest dissection prevents the illusion that 'tuning fixed it' and reveals the method is not capturing meaningful entropy dynamics.

Weaknesses:
1. Extremely limited model diversity: only DeepSeek-Coder-6.7B-Instruct is evaluated (Section 4.1). The entropy sparsity finding (88.3% zeros) may be specific to this model's instruction tuning or size. Whether the same failure occurs with larger models (e.g., DeepSeek-Coder-33B, CodeLlama-34B), different architectures (e.g., StarCoder2), or non-instruction-tuned variants is entirely unknown. A negative result on a single 6.7B model does not establish that 'entropy dynamics do not provide reliable signals for code generation' as the title claims.
2. The nEDIS v2 design (Equation 4) is not a principled adaptation but a post-hoc optimization that fundamentally alters the method's semantics. The authors acknowledge this (Section 4.4), but presenting it as a named variant ('nEDIS v2') and including it in the main results table gives it an air of legitimacy it does not deserve. Including a method that is effectively length × CV_H as an 'entropy dynamics' method muddies the evaluation.
3. Missing comparison to more competitive execution-free baselines. Self-certainty (Kang et al., 2025) is mentioned but the paper notes it is equivalent to Mean H in the single-model setting (Section 3.5), making it a redundant baseline. Key alternatives like semantic similarity between candidates, consensus-based selection (majority vote on outputs), or even simple token-level logprob averaging are not tested. Without these, the paper cannot convincingly claim that 'alternative approaches are needed'—some alternatives may already work.

Must Fix Items:
1. Evaluate on at least one additional code model (e.g., a larger model or non-instruction-tuned variant) to test whether the entropy sparsity finding generalizes. The current title makes a universal claim based on a single model experiment.
2. Add a consensus/majority-vote baseline (a natural execution-free method) to strengthen the conclusion that entropy-based methods are specifically inadequate, rather than just poorly designed for this task.
3. Reduce the title's scope: 'Entropy Dynamics Do Not Provide Reliable Execution-Free Selection Signals for Code Generation' implies a general finding, but the evidence supports only 'EDIS-style entropy dynamics fail with DeepSeek-Coder-6.7B-Instruct for code generation selection.'

Runs:
- run=1 score=4.2 verdict=Reject confidence=0.6 error=None