Information for decision-makers: The validation argument of Deutschtest.digital – Evidence-based decision confidence through modern language diagnostics
The assignment of a language test to the Common European Framework of Reference for Languages (CEFR) represents a complex validation process. The claim that a test corresponds to a specific CEFR level is not a directly observable fact, but must be supported by theoretical and empirical evidence. The collection and evaluation of this evidence takes place within the framework of validating the corresponding claim to validity.
(see Council of Europe, 2009, p. 7)A language certificate documents measured values. Correct interpretation is crucial!
If, as a decision-maker, you require proof of language proficiency from an applicant for a specific purpose, you will generally be presented with a language certificate or a corresponding test result. Such a document essentially contains a measurement result determined under specific methodological and examination-related conditions. If you use this, for example, in an official procedure to assess language competence, it can serve as evidence (see Section 26 Paragraph 1 Sentence 1 and Sentence 2 No. 3 of the German Administrative Procedure Act). However, such a measurement result only gains its significance through a professionally sound and evidence-based interpretation. The crucial question, therefore, is not merely what result is written on paper, but on what scientific basis this measurement result can be reliably evaluated.
Whether such an interpretation is valid depends largely on whether the underlying testing processes are scientifically sound, technically reliable, and organizationally controlled. Therefore, what matters is not the mere measurement value, but the quality of the entire system that generates it.
To assess the reliability of a language certificate, it is therefore necessary to examine the conditions under which test data are collected, processed, and interpreted. These controlled processes generate the necessary evidence to enable a scientific evaluation of the test quality.
However, these individual pieces of evidence do not yet constitute a conclusive justification for the interpretation of a test result. Only through their systematic integration within a validation argument does a scientifically comprehensible basis emerge for why a specific statement about a person's language competence can be derived from a concrete test result.
The following sections first explain the infrastructure of evidence generation – that is, the standards, processes, and control mechanisms through which reliable evidence is produced. Subsequently, they describe how this evidence is systematically integrated within our validation approach and on what scientific basis the interpretation of our test results is founded.
1. The infrastructure for generating reliable evidence
To objectively demonstrate that a certificate is not an empty promise, a testing organization needs robust empirical evidence (psychometric data, reliability analyses, validation documentation). Modern test research emphasizes a central premise: there are no inherently "valid tests" or "valid test results"—validity can only ever be achieved through the interpretation and use of these results for a specific purpose.
Only a methodically comprehensive quality management system can guarantee that a test provider continuously generates clean, unbiased raw data, which is essential for justifying such a valid interpretation of language proficiency. Since such evidence is only trustworthy if it is not generated arbitrarily, a rigorous, independent system architecture is required. This architecture rests on five pillars:
Pillar a) The standard
Info box / Definition: What is a standard?
A standard is a documented set of rules that describes requirements, criteria, or recommendations for specific processes, procedures, or services. It creates a common framework to systematically ensure quality, comparability, transparency, and traceability. Standards can be developed and published by various stakeholders and serve to define best practices, technical requirements, and quality benchmarks in a transparent and comprehensible manner.
Which standard does Deutschtest.digital use?
Deutschtest.digital bases the development, administration, evaluation, and continuous quality assurance of its language tests on the requirements of DIN ISO 29992. With ISO 29992 ("Evaluation of the outcomes of learning services"), the international standards committee ISO/TC 232 ("Education and Learning Services") has published a dedicated standard aimed at organizations that provide learning services and develop, select, or apply assessments. The international standard ISO 29992 has been adopted into the national standards system by the German Institute for Standardization (DIN) as DIN ISO 29992.
The standard formulates specific quality requirements for the planning, development, administration, and evaluation of assessments and explicitly includes the measurement of language proficiency according to the Common European Framework of Reference for Languages (CEFR). Unlike purely test-theoretical guidelines, ISO 29992 primarily stems from the logic of quality-assured service management. For Deutschtest.digital, it forms the central organizational and methodological framework for quality assurance. It supports us in generating reliable evidence, transparently documenting testing processes, and ensuring the consistent quality of our language tests according to verifiable and traceable standards. At the same time, it provides decision-makers with comprehensive legal, institutional, and methodological certainty.
What does ISO 29992 include?
The requirements of ISO 29992 are structured into the following main chapters:
- Chapter 1 The scope of application of the standard is specified. Two main applications are mentioned: (1) the measurement of individual learner results and (2) the determination of learning progress.
- Chapter 2 refers to references to other standards (which are not present in the case of standard ISO 29992:2021).
- Chapter 3 defines a series of technical terms related to diagnostics/performance measurement.
- Chapter 4 This standard provides specific recommendations regarding the planning of diagnostic instruments (consistently referred to as "assessment" in the text). According to DIN ISO 29992:2021-04, the planning phase should include a needs analysis, resource planning documents, and a framework document containing information on the scope of application of the diagnostic instrument; decisions made based on the instrument; the interpretation of its results; the intended consequences of using the diagnostic instrument; the relationship between the instrument's objectives and the assessment methodology used; the relationship between results, their interpretations, uses, and consequences; the communication of test results; conditions of use, scheduling, and costs; assessment procedures; the assessment environment (e.g., premises); prerequisites for participation in the assessment and applicable standards for instruments and methods used in the assessment of learning outcomes; quality assurance procedures; stakeholders involved in the development and/or implementation of the assessment; an operational plan for the diagnostic instrument; and rules of procedure. as well as the availability of the diagnostic instrument and the degree of coverage of the test specifications requirements.
- Chapter 5 It contains a number of recommendations for the development of the diagnostic instrument. These include aspects such as the timing of the instrument's use (before, during, or after the learning process); the qualifications of the test developers; the format of the test items/exam questions; and the spatial, temporal, organizational, and technical specifications for administering the diagnostic instrument. Furthermore, it provides guidance on the test quality criteria of objectivity, reliability, and validity; the process of developing test items/exam questions; procedures for determining the results of the diagnostic instrument; the framework for how and when the results are communicated; procedures for challenging the results; and requirements for technical documentation.
- Chapter 6 This focuses on the administrative aspects of a diagnostic instrument. Facets of this include guidelines for administration; a plan to ensure test safety; the qualifications and responsibilities of individuals supervising the administration of a diagnostic instrument; and the qualifications of raters.
- Chapter 7 This refers to the operational use of a diagnostic instrument. Documents relating to operational use are required, such as evidence of reliability and validity; regulations for administering the diagnostic instrument; and statistical procedures for test-methodological quality management and related processes. Furthermore, Chapter 7 requires a detailed plan for how an existing diagnostic procedure will be continuously revised and further developed.
- Chapter 8 This section addresses the issue of fairness. From this perspective, it outlines features such as a formal agreement to be concluded between the provider of a diagnostic instrument and the test candidates. This agreement should include, for example, information on test fees, the rights and obligations of the parties involved, participation requirements, a description of the diagnostic instrument, data protection information, and details regarding the timeline. Chapter 8 further addresses the prohibition of discrimination; the rights of test candidates; the neutrality of the diagnostic instrument; codes of conduct for those involved with the diagnostic instrument; and information that is made available to test candidates.
- Chapter 9 DIN ISO 29992:2021-04 addresses further ethical considerations regarding the use of diagnostic instruments. In this respect, it requires a clear definition of the responsibilities of all parties involved. Furthermore, it demands truthful and accurate information about the diagnostic instrument. Finally, a diagnostic instrument should guarantee various aspects of information security, such as the confidentiality of collected data, the protection and security of data, and the security of communication channels. Ultimately, a communication management system should also be planned and documented.
- Appendix A Specifies the information that should be collected in a structured manner in the technical documentation for a diagnostic procedure. This includes all documents relating to the entire life cycle of the diagnostic instrument, from instrument planning and development to operational application. Relevant documents include the needs analysis; the framework for the use of the diagnostic instrument; the scoring specifications; a manual for administering the diagnostic instrument; a training manual for those involved; forms for raters; specifications for the qualification and training of examiners/raters, proctors, etc.; documents on procedures for developing test items; documentation of test audits and validation studies of the diagnostic instrument; evidence of regular quality assurance; evidence of the psychometric quality of individual test items; evidence of procedures focused on eliminating unsuitable test items; evidence of verification and procedures for the manual scoring of results; and evidence of the conformity of live scoring with the test specifications. Evidence of the reliability of machine-based evaluations; evidence of the arbitration and appeals procedure; and evidence of the procedure in the event of re-evaluations. This very broad range of evidence is further supplemented by requirements for technical documentation on how users of the diagnostic instrument should be supported in its application. Details regarding the storage of technical data and documentation conclude the substantive part of the standard.
What are the key advantages of DIN ISO 29992?
DIN ISO 29992 provides Deutschtest.digital not only with a formal quality framework, but also creates a systematic basis for the development, administration, and continuous review of our language tests. The key advantages of applying this standard lie particularly in the following areas:
Ensuring a binding security architecture: The standard defines requirements for a structured security plan. Organizations applying DIN ISO 29992 must systematically implement preventive measures against manipulation (fraud prevention) and safeguard the integrity of tasks as well as information and identity security through defined processes.
Sustainable quality assurance through lifecycle management: By requiring a continuous maintenance and monitoring process (maintenance plan), ISO 29992 supports the long-term assurance of testing quality. Instead of a one-time review, the standard considers the continuous evaluation and improvement of testing procedures as an essential component of quality assurance. This allows changes in test quality to be identified early and corrected through empirically supported measures.
Technical linkage with the Common European Framework of Reference (CEFR): The structure of the standard makes it possible to systematically transfer general quality requirements to language diagnostics. This allows the requirements of ISO 29992 to be linked with the theory-based competence descriptions of the Common European Framework of Reference for Languages (CEFR) and enables the assignment of test results to language levels to be comprehensibly justified.
Transparent justification of the interpretation and use of results: ISO 29992 requires a clear presentation of the relationship between assessment results and their interpretation. This provides the basis for a scientifically comprehensible description of what conclusions can be drawn from a test result and for what purposes a certificate may be used.
Basis for a scientifically sound validation argument: A key advantage of ISO 29992 is that it supports the structured generation and documentation of relevant evidence. Combined with psychometric methods, this creates a robust evidence base upon which a scientifically sound validation argument can be developed. This argument enables the interpretation and use of test results to be justified transparently, comprehensibly, and with sound professional evidence.
Pillar b) An independent certification body
Info box / Definition: What is an independent certification authority?
An independent certification body is an external, impartial organization that verifies and confirms that an organization meets specified quality requirements or standards. It does not evaluate the certified organization's own performance, but rather conducts an independent audit of its documented processes, structures, and evidence. This external assessment creates an additional layer of trust, as quality assurance is not solely the responsibility of the organization itself.
A key feature of independent certification is the separation between the assessment and the certification decision. The review of processes and evidence is carried out by expert auditors, while the final certification decision is made by an independent body. This avoids self-assessments and potential conflicts of interest, ensuring the objectivity and credibility of the certification.
Which independent body does Deutschtest.digital collaborate with?
For external quality assurance, we collaborate with the independent certification body DeuZert . DeuZert guarantees the complete integrity, credibility, and acceptance of its certification decisions. With more than two decades of expertise, DeuZert possesses comprehensive specialist knowledge in the education and certification sector.
What are the advantages of independent certification by DeuZert?
The key added value of independent certification by DeuZert lies in the fact that the quality of the implemented processes is not solely assessed by the organization itself, but is subject to external expert review. This creates an additional level of trust for decision-makers. The following aspects are particularly relevant:
- Institutional independence and objective evaluation: DeuZert acts as an independent certification body and assesses the implementation of defined requirements using transparent testing procedures. The separation between the audited organization and the external assessment body prevents self-assessments and strengthens the objectivity of the certification decision.
- Continuous quality assurance through surveillance audits: The quality of the implemented procedures is not assessed once, but continuously monitored through regular reviews. This external control ensures that methodological and organizational requirements are consistently met and that further developments are systematically reviewed.
What formal proof certifies that Deutschtest.digital complies with the DIN ISO 29992 standard?
Following successful completion of the auditing process, DeuZert has issued us a certificate of conformity with DIN ISO 29992. This document confirms that the procedures implemented at Deutschtest.digital for planning, developing, conducting, and evaluating assessments in the field of German language skills meet the requirements of the standard and have been evaluated as part of an independent external audit.
The certificate of conformity thus serves as formal proof that Deutschtest.digital aligns its testing processes with an internationally recognized standard for the assessment of learning outcomes. It confirms the systematic implementation of standards-compliant procedures to ensure transparency, comparability, and quality of the test results.
view certificatePillar c) Independent experts and auditors
Info box / Definition: The role of auditors
International standards stipulate a strict separation between assessment and certification decision-making. The actual review of the testing organization (the audit) is conducted by independent, highly qualified auditors. They examine the empirical evidence, evaluate the system architecture in practice, and document their findings in an audit report, which serves as a neutral factual basis for the certification body's subsequent decision.
Who conducts the audits at Deutschtest.digital?
The surveillance audits are conducted by a network of experienced education experts, examiners, auditors, and assessors acting on behalf of DeuZert. The use of this highly specialized team ensures that the evaluation of our processes and evidence is not merely administrative, but also carried out with the highest level of expertise in content and language testing theory.
Pillar d) Regulations on the audit procedure
Info box / Definition: What are the regulations governing the audit process?
Audit regulations are binding procedural rules that precisely define how a certification body must conduct an audit. They govern the process of document reviews, sampling, interviews, and on-site or system audits to prevent arbitrariness and guarantee reproducible audit quality.
Which audit regulations apply to certification according to ISO 29992?
The audit procedure for ISO 29992 is subject to the requirements of the overarching standard DIN EN ISO/IEC 17065. This standard does not regulate the test itself, but rather the working methods, independence, and competence of the certification body. This ensures that the external audit is free from commercial or internal conflicts of interest. The result is independent external confirmation of the standard-compliant implementation of the testing and assessment processes, based on a transparently documented professional and methodological foundation.
Pillar e) Evidence
A sound assessment of language skills requires a reliable data basis. Evidence of test quality relies primarily on the following key categories of evidence:
1. Planning and concept documents
These documents establish the strategic and contractual foundation before the actual test development begins. They document systematic needs analyses of the target group as well as binding agreements regarding the precise purpose of the test.
2. Specification and task review documents
This category includes detailed blueprints and test protocols for content quality assurance. They demonstrate that each individual test item meets the technical specifications and the targeted CEFR level.
3. Administration and security documents
They guarantee a completely uniform, standardized, and tamper-proof examination process at all testing locations. This includes manuals for invigilators, strict identity checks of candidates, and detailed IT security plans.
4. Evaluation and scoring protocols
These documents govern the objective, error-free, and unbiased awarding of points. They contain answer keys, assessment guidelines for examiners, and evidence of the accuracy of automated evaluation procedures.
5. Qualification and training certificates
They demonstrate the professional, academic, and diagnostic qualifications of all testing and assessment personnel. Furthermore, they document the systematic training and regular refresher course concept to ensure uniform assessment standards.
6. Empirical evidence of quality and validation reports
They provide the scientific and statistical evidence for the actual psychometric quality of the test. They include calculations on measurement accuracy (reliability), fairness across subgroups, and a theory-compliant dimensional structure.
7. Maintenance and revision processes
These processes control continuous quality monitoring and the regular renewal of tests during operation. They define statistical metrics for the lifecycle of tasks to ensure that consumable materials are replaced in a timely and standardized manner.
8. Fairness, justice and appeals
They safeguard the rights, dignity, and protection of exam candidates throughout the entire process. This includes transparent appeal and complaint guidelines, easily understandable user support, and strict data security concepts.
2. From the evidence base to the validation argument: The scientific justification of test interpretations
The evidence generated within the quality assurance architecture initially constitutes individual pieces of evidence regarding the quality and functionality of the testing procedure. Only through their systematic integration within a validation argument does a comprehensible justification emerge for why a specific statement about a person's language competence may be derived from a test result.
What is a validation argument?
If test developers consider scientific quality criteria in isolation, they risk losing sight of the overarching logic and objective of the test. The quality of a language test does not result solely from individual tasks or statistical parameters, but from the interplay of all components of the testing procedure. In modern language testing research, the validity of a test is therefore understood—influenced, among others, by the work of Bachman & Palmer, Michael Kane, and Carol Chapelle—as a comprehensive and multidimensional quality concept. The crucial question is not whether individual quality criteria are met, but whether the conclusions drawn from a test result can be justified overall.
This question is particularly important because the results of language tests often have far-reaching consequences for educational, career, and residency decisions. Test providers therefore bear a special professional and ethical responsibility. They must not only ensure the quality of their testing procedures but also be able to demonstrate this quality transparently. It was precisely from this need that the concept of the so-called validity argument developed in modern validity research.
Historically, this approach emerged in the 1990s and 2000s as a groundbreaking response to the previously very abstract scientific validity models, which offered little concrete guidance for real-world testing practice. The American researcher Michael Kane applied the practical argumentation theory of philosopher Stephen Toulmin to test theory. The renowned linguist Carol Chapelle subsequently developed this logic further specifically for language testing research – she is world-famous for her pioneering validation of the TOEFL iBT starting in 2008. She refined the model into a seamless chain of seven precise inference steps, which today represents an established modern approach to the systematic justification and documentation of test interpretations.
The crucial benefit of this approach lies in freeing language testing organizations from the passive role of mere "certificate issuers ." Instead of simply submitting disorganized individual documents or rigid checklists to accrediting authorities or ministerial decision-makers, test providers can now assume an active, proactive "advocacy role ." This enables them to argue the validity, fairness, and legal certainty of their examinations to decision-makers independently, transparently, and in a scientifically unassailable manner, proving it themselves through a systematically constructed chain of evidence.
A validation argument thus describes this structured approach used to justify the validity of test results. The starting point is the assumption that every test makes a claim about a person's language proficiency. For example, if a person is assigned level B1 of the Common European Framework of Reference for Languages (CEFR), this implies that they possess the language skills defined for this level. The validation argument has the task of systematically supporting and comprehensibly justifying this claim.
This involves constructing a coherent chain of reasoning that extends from the test tasks through administration and evaluation to the interpretation of the results. For each individual step, theoretical foundations and empirical evidence (so-called backing) are used to demonstrate that the assumptions made are justified. The validation argument thus connects scientific evidence with practical testing decisions and makes transparent why a particular test result can be considered meaningful evidence of a defined language competence. In this way, it creates the basis for trust, transparency, and the legal accountability of language tests.
What logical chain links constitute a validation argument?
To provide irrefutable professional evidence of the quality of a language test, modern diagnostics follows a clear logical structure. Only when empirical evidence exists for each of these seven levels is a test considered fully reliable for practical application.
The architecture of the validation argument of Deutschtest.digital
In the following, we present the methodological architecture of our validation argument for the assessment of German language proficiency. Since a validation argument is always geared towards a specific target domain, a defined proficiency level, and a particular purpose, the underlying evidence base must be specifically structured for the intended interpretation and use of a test result. The architecture presented below illustrates how the individual conclusions (inferences) within a validation argument are linked and what types of evidence (backing) are required to scientifically substantiate the resulting statements. A complete validation argument with all empirical and statistical evidence would exceed the scope of this presentation. Therefore, the corresponding evidence documents are listed here as examples.
1 Domain definition
| Categories | Our quality standards | Evidence document (backing) |
|---|---|---|
| Domain description | The relevant linguistic requirements and communicative situations of the target group have been accurately and completely identified. | Target domain specification |
| representation | The test items adequately represent the domain defined in the test specifications by covering the required content and cognitive levels in the correct proportions. | Exam Blueprint/ Mapping Tools |
| relevance | The test items are relevant in terms of content and precisely calibrated to the desired level of competence; they do not contain any inappropriate sources of difficulty that are not part of the construct being measured. | Item review protocols, item evaluation using rating sheets (including proof of the CEFR level check) |
| Quality control | The development process includes systematic editorial control mechanisms that ensure tasks exhibiting qualitative deficiencies, ambiguities, or cultural biases are necessarily revised or discarded. | Item quality control protocol (item review) |
Scientific background
In psychometric and language test theory literature, the use of test blueprints and mapping tools is the mandatory methodology for ensuring content validity and demonstrating the appropriate representation of the target domain. The Standards for Educational and Psychological Testing explicitly recommend creating a logical structure that maps test items to the content domain (mapping).
To ensure this coverage in practice, Alderson (2005) explicitly defines test specifications as the "blueprint" that test and item authors must follow to establish the construct validity of a test. Fulcher and Davidson (2007) also refer to test specifications literally as "generative blueprints."
2 evaluation
Part 1: Administration Conditions & Tamper Protection
| Our quality standards | Evidence document (backing) |
|---|---|
| The examination regulations are of high quality, containing comprehensive and precise guidelines that ensure strictly standardized and uniform test administration, as well as binding regulations for fair and construct-compliant accommodations (accessibility). The invigilators unequivocally validate the candidates' identities in accordance with the applicable guidelines using reliable authentication procedures and continuously monitor the testing environment (room control), thereby structurally preventing proxies and unauthorized information exchange. | Examination regulations, handbook for conducting assessments - room control & identity verification |
| A tamper-proof examination environment – regardless of whether the examination takes place in person or digitally – reliably prevents access to unauthorized aids and completely eliminates cheating attempts through format-compliant control mechanisms (physical proctoring staff, secure browser, remote proctoring). | List of physical and electronic security requirements, emergency plan in case of breach of security regulations, countermeasures against unauthorized activities by examination candidates |
The unambiguous identification of test participants and the seamless monitoring of the testing environment constitute the absolute conditio sine qua non (essential prerequisite) of any legally sound testing procedure. If the organizational procedures are flawed and the physical identity of the test taker cannot be unambiguously linked to the test performance, all subsequent statements regarding content and statistical validity become invalid.
Parts 2 & 3: Scoring quality, protocols & evaluators
| Our quality standards | Evidence document (backing) |
|---|---|
| Precise rules and criteria exist for all evaluation methods and the results recording process, thus ensuring objectivity and procedural reliability. The assessment scales, based on the CEFR reference levels ("Can-Do" statements) and adapted to specific contexts, represent valid indicators of the language sub-competencies being measured. | Handbook for conducting the assessment, handbook of assessment specifications: minimum requirements for the testing software, assessment criteria for productive performance. |
| The assessors employed possess the necessary professional qualifications (e.g., relevant university degrees) and relevant didactic experience. | References for the assessor's qualifications |
| The systematic training program provides assessors with the necessary skills to apply assessment guidelines consistently and accurately. | Training certificates of the assessors |
| Systematic manual quality controls and plausibility checks by qualified specialists (human-in-the-loop) verify the accuracy of the automated evaluation processes and the correctness of the solution keys. | Proof of the technical processing accuracy of the machine evaluation |
| The evaluation of examination performance is independent of the individual strictness of the examiners and is free from systematic biases towards specific tasks or groups of people. | Confirmation of the objective evaluation quality (multifaceted Rasch analysis) Target values: Fit values in the range of 0,5 to 1,5. Interaction effects not significant (t ≤ 2,0). |
In psychometric and language test theory literature, the validity of performance assessments is required to rely on standardized, externally anchored criteria (criterion-referenced measurement). To exclude subjective standards, rating scales must be precisely aligned with an established, theory-based reference system.
Part 4: Empirical Quality / Statistical Integrity
| Our quality standards | Evidence document (backing) | Psychometric target values |
|---|---|---|
| The test items have sufficient item discrimination to effectively and reliably differentiate between candidates with higher and lower language proficiency. | Proof of statistical item discrimination | Target selection power (r) ≥ 0,50 (values between 0,30 and 0,49 are acceptable). Discrimination index (D) ≥ 0,40. Distractors must have a selection frequency > 0. |
| The difficulty levels of the test tasks adequately cover the entire performance range of the target population. | Evidence of appropriate item difficulty | Target range p-value between 0,40 and 0,59 (0,20–0,80 acceptable). Mean difficulty of the overall test of approximately 0,50. |
| The test items function invariantly across relevant subgroups (no bias due to gender or first language). | Confirmation of test fairness (Differential Item Function / DIF) | Differential Item Function (DIF) according to Mantel-Haenszel procedure (Category A: safe according to ETS classification). |
3 Generalization
Consistency across test formats and tasks
| Our quality standards | Evidence document (backing) | Psychometric target values |
|---|---|---|
| Consistency across test forms: Test results are consistent across different test versions (level stability). | Proof of the quality of the test form equivalence (parallel test reliability) | Target correlation r ≥ 0,90 (from r ≈ 0,80 acceptable). t-test without significant differences (p > 0,05). |
| Consistency across tasks: The test items are homogeneous (internal consistency). | Proof of internal consistency and homogeneity | Cronbach's alpha α ≥ 0,90 (0,80 - 0,89 acceptable). Discrimination of remaining items ≥ 0,30. |
Consistency across reviewers
| Our quality standards | Evidence document (backing) | Psychometric target values |
|---|---|---|
| The assessment of examination performance by a single examiner is stable and consistent over time (consistency of judgments). Additionally, the assessments by different examiners show no significant differences in rigor (neutrality). | Empirical evidence of the consistency and neutrality of judgments | Inter-rater comparison: No significant mean differences (p > 0,05). ICC or Kappa ≥ 0,90 (target value). |
4 Declaration
| Our quality standards | Evidence document (backing) | Psychometric target values |
|---|---|---|
| Construct representation: The internal structure of the test results corresponds to the theoretically defined construct (GER). | Empirical evidence of construct dimensionality (EFA/CFA). | KMO value ≥ 0,80. Good fit values: RMSEA ≤ 0,05, SRMR ≤ 0,08, CFI/TLI ≥ 0,95. |
| Nomological network: The test results correlate with established standardized test procedures (external criterion). | Empirical evidence of criterion validity (external criterion) | Strong, significant correlation of r ≥ 0,50 (p < 0,05). |
5 extrapolation
| Our quality standards | Evidence document (backing) | Psychometric target values / analysis |
|---|---|---|
| Target domain performance indicators: Test results correlate with actual ability to act in practice. | Document demonstrating concurrent validity (teacher assessment) / Document demonstrating predictive validity (practical validation by relevant experts) | Correlations in the range of r Approximately 0,30 to 0,50. |
| Comparability of performance: Test items and evaluation criteria comply with the established standards of the target domain. | Confirmation of the criteria relevance through expert validation. | Systematic comparison with subject matter experts (AERA, APA & NCME standards). |
| Authenticity and correspondence: Input materials and formats correspond to typical practical requirements. | Confirmation of situational authenticity (input analysis). | Structural performance equivalence (MANOVA). Global test not significant (p > 0,05). |
| The language performances show no significant structural deviations from authentic performance samples. | Empirical evidence of structural performance equivalence (MANOVA). | |
| The test participants apply construct-relevant cognitive problem-solving strategies (genuine understanding instead of guessing). | Evidence of construct-compliant mental processes (process analysis). | - |
6 Usage
| Our quality standards | Evidence document (backing) | Psychometric target values |
|---|---|---|
| Suitability / Usefulness: Stakeholders (universities, lecturers) consider the results useful for making justifiable decisions. | Proof of the practical applicability of the test results. | Overall accuracy ≥ 0,90. Reliability index (Livingston & Lewis) Φ ≥ 0,70. |
| Decision rules: Cut-off scores are the result of a qualified standard-setting procedure. | Assessment framework - Standard-setting procedures / Cut-scores. | |
| Classification quality and consistency: Classifications largely correspond to an independent reference assessment. | Proof of classification accuracy (cut scores). | |
| Interpretability and communication: Test results are clearly interpretable for stakeholders. | Confirmation of the interpretability and usefulness of results reports. | |
| Application fidelity: Decision-makers consistently apply established decision rules in their operations. | Proof of adherence to application. | - |
7 Consequences
| Our quality standards | Evidence document (backing) |
|---|---|
| The realized consequences of the test usage actually achieve the intended benefits, while negative side effects remain minimal. | Evidence of the consequences of the test results (User Benefit & Washback) |
| The rights, dignity and sensitive data of test participants are protected by strict guidelines (including complaint procedures). | Appeals guidelines, manual for the information management system |
| The implementation of the test has a positive washback effect on the education system (focus on skills instead of test strategies). | Evidence of the consequences of the test results (macro-impact & social benefit) |
In modern test theory, the implementation of appeal procedures and information security guidelines is not viewed as a purely administrative act, but as a mandatory methodological instrument for ensuring consequential validity (cf. Messick, 1989; 1995). A regulated arbitration procedure serves as a necessary qualitative counter-check for identifying construct-irrelevant variance.