Introduction

Cognitive diagnosis models (CDMs) are psychometric models that assess one’s mastery of latent skills being tested. CDMs provide detailed feedback, including the probability of mastering a certain topic. Owing to CDMs’ effectiveness in determining strengths and weaknesses in the topics to be tested, researchers in the field are becoming more aware of CDMs and “assessment for learning rather than assessment of learning” (Ravand & Robitzsch, 2015).

Multiple formulations of CDMs have been proposed in psychometric literature such as deterministic inputs, noisy “and” gate (DINA) (de la Torre, 2009), generalized DINA (GDINA) (de la Torre, 2011), and log-linear cognitive diagnosis models (LCDM) (Henson et al., 2009). There are multiple packages to fit different CDMs, such as the cdmTools and CDM packages (Nájera et al., 2022; Robitzsch et al., 2022). These packages help researchers use CDMs to learn more about the examinees’ latent attributes based on their responses.

An important component of CDMs is the Q-matrix that informs the dependency structure between the J test items and K latent attributes (Li et al., 2022; Xu & Shang, 2018) because the Q-matrix can be effectively used to design intervention strategies. An example Q-matrix is shown in Table 1. ‘1’ in the matrix means that Skill K is required for mastery of Item J. Thus, Q-restricted latent class models have gained popularity in fields such as educational proficiency assessments, psychiatric diagnosis, and many more disciplines (Xu & Shang, 2018). A well-known usage of CDMs is to study the dependency between mathematical questions (items) and their latent skills for the topic of fractions as shown in Table 2. Let the six attributes tested in this topic be:

The first item (mathematical question) in the test is $2\frac{3}{4}+1\frac{1}{2}$, where “find the lowest common denominator,” “add fractions,” and “convert mixed numbers to improper fractions” (skills 1, 2, and 6) are required for this question to be answered correctly. Thus, the rows of the Q-matrix corresponding to this item will contain the vector (1,1,0,0,0,1) as shown in Table 2.

The Q-matrix plays an important role in CDMs because it can be used to categorize test items and design future assessments (Li et al., 2022). However, not all assessments can be explicitly specified with a Q-matrix. Even if there is an explicitly specified Q-matrix, the Q-matrix may not be accurate due to the following reasons: (i) design error by the assessment provider; and (ii) one test item may be linked to multiple attributes, but not all attributes are found and identified. For example, error (i) is committed in the second row of Table 2 because skill 6 is not required to correctly answer $2\frac{3}{4}-1\frac{1}{2}$. Thus, it is important to be able to learn more about the Q-matrix from the responses in order to have a better understanding of the relationship between the test items and latent variables’ attributes.

Models

In this paper, the models of interest are the deterministic inputs, noisy “and” gate (DINA) model, the generalized-DINA (GDINA) model, and the restricted Boltzmann machines (RBMs). These three models are used in this paper to perform the following:

• Test the accuracy of RBMs used by Li et al. (2022) on the data generated with a small number of latent attributes $K\in \left\{3,4,5\right\}$.

• Compare the outputs from (i) with the results from Xu & Shang (2018).

• Compare the results generated from the “CDM” package with the results from (i) and (ii) (Robitzsch et al., 2022).

Data

The data are simulated with latent attributes dimension $K\in \left\{3,4,5\right\}$ and the number of test items, $J=20$. The true Q-matrices chosen are identifiable and similar to those used in Xu & Shang (2018). The three true Q-matrices are

In this study, the data is simulated from the DINA latent class model. The ground truth response probabilities for all items are between 0.2 and 0.8, and both the slipping and the guessing parameters are set to 0.2. The dependency of latent attributes, $\rho$, is set to $\rho \in \left\{0,0.15,0.25,0.5\right\}$. The two-step simulation of true latent profiles follows those set in Xu & Shang (2018). First, $x_i$ is generated with $x_i=\left(x_{i1},\mathrm{...},x_{iK}\right)\stackrel{\text{i}\text{.i}\text{.d}\text{.}}{~}N\left(0,\Sigma \right)$ for $i=1,\mathrm{...},N$ where $\Sigma =\left(1-\rho \right)I_K+\rho 1_K{1}_K^T$. The attribute profile ${\alpha }_{ik}$ is set to be 1 if $x_{ik}\ge 0$ and 0 otherwise. The response data is then generated using the ‘sim.din’ function from the CDM package.

Estimating the Q-matrix

The Q-matrices are estimated using the gdina function from the CDM package. As the response data follows the DINA model, a GDINA model can be fitted as the GDINA model is a generalized version of the DINA model. The gdina function will be used to fit the response data using both LASSO and Truncated LASSO Penalty (TLP). The delta matrix returned by the function will be converted into a $J\times \left(2^K-1\right)$ binary matrix (intercept column removed). The idea behind this is that because $\delta =\beta \times q$, if $\delta$ is not 0, $q$ is definitely not 0, where $\beta$ and $q$ are elements in Equation (1). Values that are close to 0 in the delta matrix (smaller than 0.1) will be forced to be 0 and everything else to be 1. The $J\times 2^K$ binary matrix will be collapsed into a $J\times K$ binary matrix by grouping up the latent attributes that are required to master the item $J$.

Let $\alpha \in \left\{0,1\right\}$, $1\le k\le K$, and ${\delta }_{ji}={\alpha }_{iK}\mathrm{...}{\alpha }_{i1}$ be the binary representation index of i^th element in the j^th row of the delta matrix. ${\delta }_{ji}$ will be transformed to have a value of 1 if it is greater than the threshold and 0 otherwise.

For example, let $\delta =\left(1.4,1.32,0.08,2.1,0.0003,0.0001,0\right)$, $J=1$, $K=3$, and threshold = 0.1, then applying Equations (6), we get,

In Equation (6), the columns of the $J\times \left(2^K-1\right)$ binary matrix refer to (Attr1, Attr2, Attr3, Attr12, Attr13, Attr23, Attr123). The matrix is then collapsed into a $J\times K$ matrix by summing up all the 1s into their respective latent attributes, where the columns refer to (Attr1, Attr2, Attr3). The estimated Q-matrix in Equation (7) is expected to be identifiable only up to rearranging the orders of the columns. This is because when estimating the Q-matrix, the columns do not contain information about the latent attributes (e.g., the n^th column of the Q-matrix might not refer to the n^th latent attribute). Thus, the estimated Q-matrix will be reordered so that each column shows the lowest possible average congruent coefficient with the True Q-matrix’s columns. This process is done using the “orderQ” function in cdmTools (Nájera et al., 2022).

Results

In this study, the Q-matrices are estimated completely from the data. For $K=3$, the following crossover design is applied for the DINA model, three sample sizes, and four attribute-dependent levels: $\left\{DINA\right\}\otimes \left\{N=500,1000,2000\right\}\otimes \left\{\rho =0,0.15,0.25,0.5\right\}$. For $K=4$ and $K=5$, the designs are $\left\{DINA\right\}\otimes \left\{N=1000,2000\right\}\otimes \left\{\rho =0,0.15,0.25,0.5\right\}$.

Table 3 shows the simulation results for 50 replications. From Table 3, it can be observed that, on average, TLP and RBM outperform the LASSO method. In the case of small N and small $\rho$, RBM usually outperforms the TLP method. However, as N and $\rho$ become larger, the TLP method is able to estimate a Q-matrix more similar to the true Q-matrix. As N increases, accuracy also increases. This is because if there is more response data, the model has more data to learn and train from, resulting in a more accurate model. Surprisingly, across the three different methods, the accuracy increases as the correlation among attributes increases. This may be because the higher the dependency among the attributes, the lesser the number of possible attribute patterns, making estimation relatively easier (Li et al., 2022).

Conclusion and Future Direction

In conclusion, it is shown in Table 3 that the CDMs with TLP method outperformed the ones with LASSO method. Moreover, it is interesting to see that the RBM models have stable performance for $K\le 5$. The RBM models always have an accuracy of 82% or more for these data while CDMs perform badly when N is small.

The future work of interest would be to explore different ways to include interactions between latent attributes so that the assumptions set in RBM will not be violated. In practice, it is hard to find latent attributes that do not correlate with one another. Thus, by addressing this latent attribute interaction problem, the RBM method that has higher accuracy can be created. One potential way to address this problem may be integrating deep learning into the RBM method.

Owing to the success and stability of the RBM method in learning dichotomous item responses, it will also be interesting to implement the RBM method in research that uses polytomous item responses. This is because a lot of questionnaires contain responses in the form of a 5-point or 7-point Likert scale. It will be interesting to study how different levels of responses correlate with mastering a certain skill or how the slipping and guessing parameters are affected by the way the questions were phrased. For example, an examinee may have the skills to answer a mathematical question correctly, but because the questions contain ambiguity and poor word choices, the examinee may be unable to answer the question.