Almost all of the public argument about AI and assessment has been about writing, which is odd, because the first assessment format to fall over was the weekly problem set. A model that can produce a competent literature review can certainly produce a correct integral with working shown, or a functioning implementation of a data structure with comments explaining the choices.
In computer science this happened first and hardest, and the response in many departments was to make the problems harder. That did not work and could not: difficulty is exactly the axis on which models have improved fastest.
Why the usual fixes underperform
Harder problems. Raises the bar for the honest student more than for the dishonest one. The student using a model is not solving the problem, so its difficulty is not their constraint.
Novel problems. Helps for a term. Then the problems are online, or the model is better, or both.
More weight on the exam. Concentrates everything into one high-anxiety session and tells students the coursework is theatre. It also removes the formative value that made problem sets worth setting.
The problem set was never valuable because the answers were valuable. It was valuable because producing them changed the student.
What survives
The formats that hold are the ones where the artefact was never the whole point.
- Ask about the solution, not for it. Two minutes on "why does this converge" or "what happens if I remove this line" separates the student who worked from the student who collected.
- Grade the debugging, not the code. Hand them a broken version of their own submission and watch what they do. This is closer to the actual professional skill anyway.
- Make problem sets ungraded and frequent, then assess the understanding separately. Several departments have moved this way and report that it removed the incentive to outsource almost entirely, because there was nothing left to gain.
That last one is worth sitting with. A great deal of the integrity problem in STEM is a grading-weight problem: we attached marks to an activity whose value was formative, then acted surprised when students optimised for the marks.
The oral version of this is short. Five minutes on a submitted problem set, questions drawn from the student's own working, and the answer to who understands their solution stops being a guess.