openmaic/OpenMAIC/eval/pbl-v2-planner/judge-prompt-completability.md

You are an expert reviewer for PBL v2 runtime feasibility. Evaluate whether an auto-generated PBL project is completable by a real learner in the actual OpenMAIC Live PBL v2 runtime. This is a feasibility judge, not a pedagogy/style judge.

Requested project

• Topic: {{topic}}
• Description: {{description}}
• Target skills: {{targetSkills}}
• Proficiency tier: {{proficiency}}

Generated project JSON

{{project}}

Actual runtime capabilities

Ordinary PBL runtime

An ordinary PBL project has only:

• A left roadmap of milestones and microtasks.
• A center Instructor chat.
• A right submission panel where the learner can paste text, upload their own work, or submit a link.

Ordinary PBL does NOT have a right-side briefing tab, resource panel, reference tab, preloaded image, preloaded screenshot, attached PDF, downloadable starter file, provided dataset, or hidden document viewer. The learner can use their own external tools, editor, browser, or files, but every required project-specific material must be present in visible milestone/task/instructor text.

If an ordinary project says "see the right-side briefing", "look at the provided image", "use the attached PDF", "open the starter file", "read the dataset", "参考右侧资料", "查看右侧图片", or similar, it is blocked unless that material is fully reproduced as visible text in the project.

Judge this semantically across languages, not by matching those example phrases. If a task tells the learner to read/inspect/analyze a brief, case note, material, image, dataset, excerpt, map, table, or scenario facts, verify that the actual content needed for the task is present in visible project text. A label like "read the brief below" is not enough when the brief itself is absent.

Scenario PBL runtime

A scenario PBL project has a top-level scenario block and runs as:

• prep: Instructor explains the premise and rules.
• one or more roleplay stages: Simulator characters interact with the learner.
• wrapup: Instructor consolidates what happened.

Scenario projects may use the scenario briefing panel after prep, because that is part of the scenario runtime. Roleplay beats must still be advanceable: each roleplay microtask needs a concrete observable successWhen unless it is a prep/wrapup task. Hidden character facts may live in private characterObjective, but the learner must receive enough visible context to take the next action.

What "completable" means

A project is completable only if a learner can progress from the first milestone to the final outcome using:

• visible project text,
• Instructor/Simulator interaction provided by the runtime,
• the learner's own external tools and own created artifacts,
• paste/upload/link submission when evidence is needed.

Do not require the learner to know private facts, inspect nonexistent assets, open unavailable panels, or use a platform capability that the runtime does not provide.

Blocker codes

List every blocker code that applies:

• C1 hidden-unavailable-resource: completion depends on a right-side briefing/resource/reference panel, preloaded image/screenshot, attached PDF, starter file, provided dataset, hidden document, or other material not available in the actual runtime. Also use C1 when the project refers to a brief/material/dataset as if it exists elsewhere but the actual content is not visible in the project text.
• C2 missing-prerequisite-material: the task assumes domain context, example data, source text, case facts, API keys, account access, or setup that the project never gives and a learner could not reasonably create themselves. Also use C2 when the task asks the learner to extract facts from a brief/case/material, but those facts are not included in visible text.
• C3 unclear-done-evidence-path: a learner cannot tell what to produce, how to show it, or what observable evidence lets the task/stage complete.
• C4 unavailable-platform-capability: the project requires runtime behavior the platform does not provide, such as automatic code execution, built-in spreadsheet/database tools, browsing, grading private files, or branch-changing scenario logic.
• C5 impossible-ordering: a task needs the output of a later task, or the ordering prevents progress.
• C6 scope-too-large: the project asks for too much to complete in one focused sitting of roughly 15-45 minutes for the requested proficiency tier.
• C7 scenario-cannot-advance: scenario skeleton is not prep -> roleplay(s) -> wrapup, or a roleplay beat lacks a concrete observable successWhen / action needed to advance.
• C8 private-unseen-info-required: the learner is asked to use or answer with information that is private, hidden, or never surfaced through visible text or scenario interaction.

Scoring

• 5: Clearly completable. All required materials are visible or learner-created; done/evidence paths are clear.
• 4: Completable with minor ambiguity or friction, but no blocker.
• 3: Possibly completable, but enough ambiguity or missing context that many learners may stall. Usually pass=false unless you can explain why no blocker applies.
• 2: Likely blocked by at least one concrete runtime/material/evidence issue.
• 1: Impossible in the actual runtime.

Set pass=true only when the score is 4 or 5 and blockers is empty. Any blocker code must make pass=false.

Set riskLevel:

• low: score 4-5, no blockers.
• medium: score 3 or minor uncertainty.
• high: score 1-2 or any blocker.

Output

Output exactly one JSON object and nothing else (no prose, no code fences):

{ "score": <1-5>, "pass": <true|false>, "blockers": ["C1 hidden-unavailable-resource"], "riskLevel": "low" | "medium" | "high", "rationale": "<2-3 sentences explaining whether the learner can complete the project in the real runtime, naming the biggest blocker if any>" }

Use an empty array for blockers when there are no blockers.