jMetrik FAQ

No. jMetrik only provides a view of the data. If you must enter data manually, use a program like OpenOffice Calc, http://www.openoffice.org/ (it’s a free download). Save the file as a CSV (comma delimited) file. Import the CSV file into jMetrik. The first row of data in the CSV file may contain variable names, but variable names are not required. You should have one row of data for each examinee and one column for each variable.

You can save all graphs created in jMetrik. After creating a graph, right click (or CTRL click on a Mac) the graph to display a context menu. One of the options is for saving the graph as a PNG file. (Other options allow you to customize various features of the graph.)

Windows Users
Microsoft increased security standards for programs downloaded from the internet. They now require programs to be signed with a code signing certificate and to have an established reputation. Installer files are now signed under the company name Psychomeasurement Systems, LLC. However, the new code signing certificate has not yet developed a reputation. Therefore, you may see the message “Windows Protected Your PC” or another message saying the program is unrecognized. If you click “More Info” you will be able to verify the company name and select an option to continue the download. After downloading the software, your antivirus software may issue additional warnings that the application has not developed a reputation. It is OK to select the options that allow you to continue with the installation. As more and more people download the code-signed version of the installer, the reputation will increase and these warning messages will no longer appear.

Mac Users
Apple has also increased security standards and requires programs to be signed with a code signing certificate. The installer files for Mac are not yet signed but they will be in the near future. Until the installer is signed, Mac users may see an error message when installing or a warning that the publisher is unknown. Read this FAQ for information about this error message and how to work around it. Apple does not have the additional requirement for an established reputation. You will not see any warnings messages once the installer file is signed.

There are several columns in the DIF analysis output. The first column “Item” is the name of the item. The second and third columns are the Mantel-Haesnzel chi-square and associated p-value. Note that jMetrik uses the Cochran-Mantel-Haenszel for stratified 2 x k tables. It is a generalization of the Mantel-Haenszel for stratified 2 x 2 tables and it works with binary and polytomous items. One difference between the Cochran-Mantel-Haesnzel for binary items and the Mantel-Haenszel is that the former does not use a correction for continuity. Therefore, with binary items the Mantel-Haenszel value reported by jMetrik may slightly differ from the Mantel-Haenszel reported by other programs.

The “Valid N” column lists the number of examinees involved in the Mantel-Haenszel statistic. Examinees that failed to provide a value for the DIF group code are deleted from the analysis. Also, any stratum table with only one examinee is not included in the analysis. As such, you could have several examinees eliminated from the analysis if you have several tables with only a single examinee. In this situation, try using the “Deciles” or “quntiles” options to preserve more data.

The “E.S. (95% C. I.)” columns list the effect size (E.S.) and 95% confidence interval for the effect size, respectively. For binary items, the effect size is the common odds ratio (COR). You can optionally convert this value to the ETS Delta metric. For polytomous items, the effect size is the standardized P-DIF statistic (sP-DIF).

The final column (“Class”) is the ETS DIF classification level. The possible classifications for binary items are A, B, and C, while the possible classification levels for polytomous items are AA, BB, and CC. A and AA items show little to no DIF. B and BB suggest moderate amounts of DIF. C and CC items suggest a large amount of DIF. These classifications are a function of statistical and practical significance. The rules for binary items are:

• A item: (a) Chi-square p-value > 0.05 or (b) the COR is strictly between 0.65 and 1.53.
• B item: not and A or C item
• C item: (a) COR < 0.53 AND the upper bound of the 95% confidence interval for the COR is less than 0.65, or (b) COR > 1.89 AND the lower bound for the 95% confidence interval for the COR is greater than 1.53.

Given that polytomous items use a different effect size, the classificaiton rules are different. The rules for polytomous items are are based on dividing sP-DIF (the value in the output) by the item score range to limit values to the interval -1 to 1. Then, take the absolute value to limit the interval to 0 to 1. Call this new value sP-DIF*. It is not displayed in the output. This change allows the rules developed for binary item P-DIF to be applied to polytomous items. The rules for polytomous items are:

• AA item: sP-DIF* < 0.05
• BB item: 0.05 >= sP-DIF* <0.10
• CC item: sP-DIF* >= 0.10

Each DIF classification also includes a sign. A “+” sign (without the quotes) indicates that the item favors the focal group. A “-” indicates that the item favors the reference group. More information about the DIf classification levels are available in the following articles.

Dorans, N. J., Schmitt, A. P., & Bleistein, C. A. (1992). The standardization approach to assessing comprehensive differential item functioning. Journal of Educational Measurement, 29, 309-319.

Potenza, M. T., & Dorans, N. J. (1995). DIF assessment for polytomously scored items: A framework for classification and evaluation, Applied Psychological Measurement, 19, 23-37.

Zwick, R., & Ercikan, K. (1989). Analysis of differential item functioning in the NAEP history assessment. Journal of Educational Measurement, 26, 55-66.

The maximum number of variables in each table is 1,012. However, you can have an unlimited number of tables. If you have a data file with more than 1,012 variables, consider dividing it into multiple files and import them as multiple tables.

There is no sample size limit in jMetrik. We have tested jMetrik with data files as large as 1,000,000 examinees using the default memory allocation of 256MB. If you have data files larger than 1,000,000 examinees, you may want to increase memory allocation to improve performance.

To ask a question, please join the jMetrik User Group at Google Groups, https://groups.google.com/forum/#!forum/jmetrik-user-group.

Newer Mac computers use Gatekeeper to prevent the installation of malicious software. The default Gatekeeper setting only allows the installation of software from the Mac App Store. There are two ways to work around this security setting.

The preferred solution on Mac OSX 10.7.3 is to (a) use Finder to locate the jMetrik installer file, (b) press the Control key and click the installer icon, (c) choose open from the shortcut menu, and (d) Click Open. Your computer may activate a virus scan automatically. Let it run to completion or cancel it. Your computer may also prompt you for your password one or more times during the installation process.

The alternative solution is a system wide change that may leave your computer vulnerable to malicious software download from the internet. For this solution go to Apple menu > System Preferences…> Security & Privacy > General tab. Under the header “Allow applications downloaded from:” choose the options “Anywhere.”

See the Gatekeeper webpage on the Apple website for more information on changing the setting. Note that this fix is a temporary solution and it leaves your Mac vulnerable to malicious programs you may accidentally download form the internet.