Factorizing Complex Discrete Data "with Finesse"

Environment

This version of Finesse is written in Java with the Java 1.7 SDK (java version 1.7.0_65). The Java code language level is 7 (it uses diamonds, for example). It was developed with IntelliJ IDEA on 64-bit ubuntu 14.04 LTS. Compilation on other operating systems "should" not cause any problems, but this was not tested.

Dependencies

The source for Finesse is in two directories under 'code' in this package. Most of the core logic is in the directory 'smfcore'. The source for the executable Finesse application is in 'smf'.

GSON is used for serializing/deserializing Java objects to/from JSON. Apache Commons CLI is used for easier parsing of command-line arguments. These dependencies are included in the 'jars' folder inside each respective code directory.

Compilation

In the following, $basedir refers to the absolute path to the base directory of this package on your system.

Compile the core library "smfcore" first with:

cd $basedir/code/smfcore/src/smfcore

javac -cp "$basedir/code/smfcore/jars/gson-2.3.1.jar" *.java

Then compile the "smf" (Finesse) application with:

cd $basedir/code/smf/src/smf

javac -cp "$basedir/code/smfcore/src/:$basedir/code/smf/jars/commons-cli-1.2.jar:." *.java

Usage

java -cp "$basedir/code/smfcore/src/:$basedir/code/smf/jars/commons-cli-1.2.jar:$basedir/smfcore/jars/gson-2.3.1.jar:$basedir/code/smf/src/" smf/Smf -k2 -s "$basedir/code/smfTestData/data.txt" -b "$basedir/code/smfTestData/basis.txt" -D "$basedir/code/smfTestData/usage.txt"

The parameters are:

• k: The decomposition rank (positive integer)
• s: The path to the input data file
• b: The path to the output basis file
• D: The path to the output usage file

File format

Below is the format of $basedir/code/smfTestData/data.txt, which corresponds to the example Boolean data set on the first page of the SMF paper. The first two rows are the dimensions n and m respectively. Then, the columns are listed sequentially. Each column begins with its type (Boolean in this case) followed by the n values for that column. There are m columns.

3
3
Boolean
1
0
1
Boolean
1
1
1
Boolean
0
1
1
                                    

With this input file, the basis file that Finesse generates ($basedir/code/smfTestData/basis.txt), which in turn corresponds to the basis matrix in the first Boolean example in the paper, is:

2
3
Boolean
1
0
Boolean
1
1
Boolean
0
1
                                    

The usage file that Finesse generates ($basedir/code/smfTestData/usage.txt), which in turn corresponds to the usage matrix in the first Boolean example in the paper, is:

3
2
Boolean
1
0
1
Boolean
0
1
1
                                    

(Note: Finesse is of course non-deterministic…the factors may be listed in a different order)

Other feature types are of course supported:

• The ternary feature has values 0 (false), 1 (unknown) and 2 (true). In the data matrix, use the string "Ternary" to specify that a column is ternary. Modify the dissimilarity measure in TernarySemiringFeature.java in the smfcore directory if need be (e.g. if you want missing-value dissimilarity). See also the TMF paper (Maurus, Plant, Ternary Matrix Factorization, ICDM '14).
• The ordinal feature has values 0 through (L-1), where L is the number of elements in the scale. In the data matrix, use the string "OrdinalBinU L" (where L is the integer number of elements in the scale) to specify that a column is ordinal and that the usage matrix should be binary (use "Ordinal L" if you want entries from the full scale in the usage matrix).
• The tree feature has set values. In the data matrix, use the JSON array syntax ["a","b","c"] for each value, where "a", "b" and "c" together represent the FULL collection of subtree nodes corresponding to that matrix entry. Use the string "Tree path" in the column header of the data matrix, where path is the absolute path to the master ontology (see below for more details).

For an example of a data file that uses all these features, check out e.g. the Yummly data in $basedir/experiments/data/yummly/smf_data_mains_summer_american.txt. That file shows how a tree feature needs the path to the corresponding master ontology.

Ontology files are in JSON format. The contents are a JSON serialization of an instance of the Tree class in the 'smfcore' code. That class has static methods ToJson and FromJson for serializing and deserializing. If you need to create your own master ontology, use the following approach in your own Java code that includes the 'smfcore' dependencies:

Tree myTree = new Tree("rootId");
// Code for adding (recursively perhaps) the required children to the tree.
String serializedTree = Tree.ToJson(myTree);
// Code for writing the serialized tree to a file.
                                    

The "personality testing" data sets all came from personality-testing.info. We reported the results for the following ten data sets (here presented in the order as shown in the paper):

• Experinces in Close Relationships Scale ("Relationships")
• Feminist Perspectives Scale ("Feminism")
• IPIP Assertiveness, Social confidence, Adventurousness, and Dominance scales ("Assertiveness")
• Cattell's 16 Personality Factors Test ("Personality 1")
• Holland Code (RIASEC) Test ("Occupation")
• Humor Styles Questionnaire ("Humor styles")
• Kentucky Inventory of Mindfulness Skills ("Mindfulness")
• Open Sex Role Inventory ("Masc./Fem.")
• IPIP HEXACO equivalent scales ("Personality 2")
• Multidimensional Sexual Self-Concept Questionnaire ("Sexual self")

Some of the original data sets included additional features (e.g. participant ID) which were not relevant. During preprocessing, we hence ensured that only ordinal features were included. Any observations with missing values were removed. The original and preprocessed versions of the data (as presented to each algorithm), as well as the codebooks, are included in the directory $basedir/experiments/data/psych.

The Yummly and IMDB data used for the experiments in the paper are in $basedir/experiments/data. Each directory there has the SMF data file, the corresponding master ontology file (JSON) and a file with the list of the recipe/film names. Note that if you wish to reproduce the examples in the paper, you'll need to update the path to the master ontology in the data file in each case.

The directory $basedir/experiments/results includes detailed results from experiments on the Yummly and IMDB data. Finesse's output basis and usage matrices are given for k=3 to k=16 in both cases.