There are many ways to conduct a differential item functioning (DIF) analysis. jMetrik provides several useful statistics including the Mantel-Haenszel chi-square procedure, commons odds ratio effect size, standardized p-dif effectsize, and ETS DIF classification levels. These statistics allow you to judge the statistical and practical significance of DIF. You must complete item scoring before you conduct a DIF analysis. See FAQ #8 for more information about the DIF analysis options and output. To conduct a DIF analysis you need a matching variable such as a sum score. If one does not exist in your data table, you must create it using the Test Scaling procedures and possibly the Ranking procedures.
- Create a matching variable – If a matching variable does not exist in your data you can create one by computing a sum score. See the instructions for Test Scaling for directions on how to create a sum score. If a matching variable exists, then use it in the next step.
- Choose thick or thin matching – Thin matching involves all levels of a sum score. To use thin matching, select a sum score variable as your matching variable in a DIF analysis. This method provides the best control over the measured trait, but it may result in sparse tables and omitted responses. Think matching preserves more of the data but gives you less control over the measured trait. For think matching group examinees into ordered groups such as deciles. Use the Deciles option of the Ranking procedure to rank examinees into ten groups. Use the new decile variable as your matching variable in the DIF analysis.
- Click Analyze > DIF: Mantel-Haenszel to start the DIF analysis dialog.
- Select the items you would like to study and move them to the top right list by clicking the first select button.
- Select the matching variable and move it to the Matching Variable field by clicking second select button.
- Select the DIF grouping variable and move it to the Group By field by clicking the last select button. An example grouping variable is gender.
- Identify the Focal and reference group codes that are in your grouping variable. For example, the code F might indicate females in your DIF group variable and the code M might represent males. The case of the focal and reference group codes must match the case of the values listed for the DIF group variable. IF you use the wrong case, the program will not recognize the values in the group variable.
- You can run the analysis at this point, but you may want to change some of the default options.
- Binary item effect size – The default value is Common odds ratio option. This statistic ranges from 0 to positive infinity and has an expected value of unity. To use a more symmetric effect size, choose the ETS Delta option. The ETS statistic is a transformation of the common odds ratio that has values that range from about -4 to +4 and are centered about zero. Note that the polytomous item effect size is always the Standardized P-DIF statistic.
- Show frequency tables – Select this option to display frequency tables for all levels of the matching variable. Choosing this option will greatly increase the output.
- Score as zero – Select this option to score missing item responses as zero points. If not selected examinees missing an item response for an item will be omitted from the analysis of that item.
- If you would like to save item statistics in a new database table, click the Save button and type a name for the new table.
- Click the Run button to execute the analysis.