{
  "pdf": "284f2298-130d-4e58-b969-4c93533626e4.pdf",
  "title": "TYPED-DSL CONSTRAINED DATA RECIPES",
  "elapsed": 190.8,
  "runs_mode": 1,
  "valid_runs": 1,
  "avg_score": 4.2,
  "scores": [
    4.2
  ],
  "score_std": 0.0,
  "final_verdict": "Reject",
  "final_confidence": 0.72,
  "conference_scores": null,
  "strengths": [
    "Clean ablation design separating JSON format from typed operator constraints (Table 3), which clearly demonstrates that the improvement comes from typed operators rather than structured output format alone. The JSON-Wrapped Python baseline is a useful control that rules out the trivial explanation that JSON formatting alone drives gains.",
    "The failure mode audit methodology (Section 3.1, Table 1) provides a principled motivation: categorizing failures into DSL-addressable vs non-DSL-addressable and validating that >50% are structurally fixable. This top-down analysis-before-solution approach is methodologically sound.",
    "Honest reporting of the enum constraint's mixed results (Table 3, Section 4.4): the paper notes that on OpenFinData, removing enum constraints slightly improves performance and that zero hallucinations occurred without the enum. This negative nuance is preserved rather than hidden, which adds credibility."
  ],
  "weaknesses": [
    "The core contribution is trivial when stripped of packaging: constraining an LLM's output via a typed schema/schema-validated DSL to eliminate syntax and format errors is exactly what constrained decoding systems (Outlines, XGrammar, SynCode — all cited in the paper's own Related Work) were designed to do. The paper applies this to DataChef, but the 'insight' that structural errors can be eliminated by structural constraints is a tautology, not a research contribution. The DSL itself (Section 3.2) contains only 7 operators (SelectDataset, FilterByKeyword, MapToShareGPT, Deduplicate, Sample, Mix, LLMTransform) — a minimal wrapper around existing toolbox functions with no algorithmic novelty.",
    "Evaluation is dangerously narrow: 2 tasks, 1 model (Qwen3-32B), N=32 samples per task, no multiple seeds, no statistical significance tests. The main comparison in Table 2 rests on 64 total samples. A single lucky or unlucky batch could swing the executable rate by ~15pp. The 5.8–29× improvement claims are derived from these tiny sample sizes with no confidence intervals. With N=32, even a 1-sample change shifts the executable rate by 3.1pp, making the precision of reported numbers (e.g., '90.6%' vs '84.4%') misleading.",
    "Inconsistency in failure mode classification: Table 1 classifies 'Field Mismatch' as non-DSL-addressable (×), but Section 4.5 and Figure 2 claim 100% elimination of field mismatches (16→0) under the DSL. If field mismatches are not DSL-addressable, how does the DSL eliminate them? If the DSL does eliminate them, then Table 1's classification is wrong and the DSL-addressable fraction should be higher. This contradiction undermines the paper's causal attribution story. The most likely explanation is that the DSL's MapToShareGPT operator hard-codes field mappings, which is a task-specific fix, not a general DSL mechanism — making the '100% elimination' claim misleading about the DSL's generality.",
    "No downstream evaluation of actual model performance: DVS (Data Verifier Score) is a proxy metric using Qwen3-235B-A22B as a judge. The paper never demonstrates that the generated recipes produce training data that actually improves a fine-tuned model on the target benchmarks. High DVS does not guarantee training effectiveness. The entire value proposition — better recipes lead to better models — remains unsubstantiated."
  ],
  "must_fix_items": [
    "Report statistical significance or confidence intervals for all comparisons in Tables 2 and 3. With N=32, bootstrap CIs or binomial proportion CIs would be straightforward. Without them, the 5.8–29× claims are unsupported.",
    "Resolve the Field Mismatch classification inconsistency between Table 1 (non-DSL-addressable) and Section 4.5 (100% eliminated). Clarify whether elimination came from the DSL mechanism or from task-specific hard-coding in MapToShareGPT, and adjust claims accordingly.",
    "Evaluate on at least 2–3 additional tasks and/or a second generator model to demonstrate that results are not idiosyncratic to ClimaQA/OpenFinData and Qwen3-32B."
  ],
  "runs": [
    {
      "run": 1,
      "score": 4.2,
      "verdict": "Reject",
      "confidence": 0.72,
      "strengths": [
        "Clean ablation design separating JSON format from typed operator constraints (Table 3), which clearly demonstrates that the improvement comes from typed operators rather than structured output format alone. The JSON-Wrapped Python baseline is a useful control that rules out the trivial explanation that JSON formatting alone drives gains.",
        "The failure mode audit methodology (Section 3.1, Table 1) provides a principled motivation: categorizing failures into DSL-addressable vs non-DSL-addressable and validating that >50% are structurally fixable. This top-down analysis-before-solution approach is methodologically sound.",
        "Honest reporting of the enum constraint's mixed results (Table 3, Section 4.4): the paper notes that on OpenFinData, removing enum constraints slightly improves performance and that zero hallucinations occurred without the enum. This negative nuance is preserved rather than hidden, which adds credibility."
      ],
      "weaknesses": [
        "The core contribution is trivial when stripped of packaging: constraining an LLM's output via a typed schema/schema-validated DSL to eliminate syntax and format errors is exactly what constrained decoding systems (Outlines, XGrammar, SynCode — all cited in the paper's own Related Work) were designed to do. The paper applies this to DataChef, but the 'insight' that structural errors can be eliminated by structural constraints is a tautology, not a research contribution. The DSL itself (Section 3.2) contains only 7 operators (SelectDataset, FilterByKeyword, MapToShareGPT, Deduplicate, Sample, Mix, LLMTransform) — a minimal wrapper around existing toolbox functions with no algorithmic novelty.",
        "Evaluation is dangerously narrow: 2 tasks, 1 model (Qwen3-32B), N=32 samples per task, no multiple seeds, no statistical significance tests. The main comparison in Table 2 rests on 64 total samples. A single lucky or unlucky batch could swing the executable rate by ~15pp. The 5.8–29× improvement claims are derived from these tiny sample sizes with no confidence intervals. With N=32, even a 1-sample change shifts the executable rate by 3.1pp, making the precision of reported numbers (e.g., '90.6%' vs '84.4%') misleading.",
        "Inconsistency in failure mode classification: Table 1 classifies 'Field Mismatch' as non-DSL-addressable (×), but Section 4.5 and Figure 2 claim 100% elimination of field mismatches (16→0) under the DSL. If field mismatches are not DSL-addressable, how does the DSL eliminate them? If the DSL does eliminate them, then Table 1's classification is wrong and the DSL-addressable fraction should be higher. This contradiction undermines the paper's causal attribution story. The most likely explanation is that the DSL's MapToShareGPT operator hard-codes field mappings, which is a task-specific fix, not a general DSL mechanism — making the '100% elimination' claim misleading about the DSL's generality.",
        "No downstream evaluation of actual model performance: DVS (Data Verifier Score) is a proxy metric using Qwen3-235B-A22B as a judge. The paper never demonstrates that the generated recipes produce training data that actually improves a fine-tuned model on the target benchmarks. High DVS does not guarantee training effectiveness. The entire value proposition — better recipes lead to better models — remains unsubstantiated."
      ],
      "must_fix_items": [
        "Report statistical significance or confidence intervals for all comparisons in Tables 2 and 3. With N=32, bootstrap CIs or binomial proportion CIs would be straightforward. Without them, the 5.8–29× claims are unsupported.",
        "Resolve the Field Mismatch classification inconsistency between Table 1 (non-DSL-addressable) and Section 4.5 (100% eliminated). Clarify whether elimination came from the DSL mechanism or from task-specific hard-coding in MapToShareGPT, and adjust claims accordingly.",
        "Evaluate on at least 2–3 additional tasks and/or a second generator model to demonstrate that results are not idiosyncratic to ClimaQA/OpenFinData and Qwen3-32B."
      ],
      "conference_scores": null
    }
  ]
}