Chile has more than two decades of assessing the quality of education through a national censal system called SIMCE. However, there are scarce studies that address this assessment system from an up-to-date concept of validity. The aim of this study was to critically analyse the validity of SIMCE, considering the unitary concept of validity, the argument-based approach, the consideration of the whole process involved in the assessment system, the inclusion of the interpretations of all the stakeholders involved in the process, and threats to validity. Interviews to key stakeholders were carried out along with analyses of official documents about SIMCE. An iterative analysis was executed on the basis of the theoretical considerations, focusing on the fairness of the assessment system as understood in international standards. Some of the main findings indicate SIMCE lacks fitness-for-purpose in relation to its aim of improving the equity of the education system. It generates negative consequences which can be detrimental to students with more learning difficulties and from more vulnerable groups. And, it does not consider in its construction the diversity of students who have to take the tests.