This is the title of a paper by my student, Stefano Albrecht, which we have recently submitted to a conference. The core idea is to address model criticism, as opposed to the better studied concept of model selection, within the multi agent learning domain.
For Informatics folks, he is giving a short talk on this paper on Friday at noon, to the Agents group (in IF 2.33). The abstract is below.
The key for effective interaction in many multi-agent applications is to reason explicitly about the behaviour of other agents, in the form of a hypothesised behaviour. While there exist several methods for the construction of a behavioural hypothesis, there is currently no universal theory which would allow an agent to contemplate the correctness of a hypothesis. In this work, we present an novel algorithm which decides this question in the form of a frequentist hypothesis test. The algorithm allows for multiple metrics in the construction of the test statistic and learns its distribution during the interaction process, with asymptotic correctness guarantees. We present results from a comprehensive set of experiments, demonstrating that the algorithm achieves high accuracy and scalability at low computational costs.