# Factorizing Complex Discrete Data "with Finesse"

### Code, data, results and supplementary material

## Most importantly: use at your own risk!

From a software-engineering perspective, the implementations are not optimal. There are no unit tests and error-handling/reporting is weak. Take home message: do NOT use this implementation in any kind of production/live scenario. It should be used for experimental/research purposes only, and we cannot be held liable for any consequences of using this material in any way.

#### Code: Finesse Compilation and Usage

## Environment

This version of Finesse is written in Java with the Java 1.7 SDK (java version 1.7.0_65). The Java code language level is 7 (it uses diamonds, for example). It was developed with IntelliJ IDEA on 64-bit ubuntu 14.04 LTS. Compilation on other operating systems "should" not cause any problems, but this was not tested.

## Dependencies

The source for Finesse is in two directories under 'code' in this package. Most of the core logic is in the directory 'smfcore'. The source for the executable Finesse application is in 'smf'.

GSON is used for serializing/deserializing Java objects to/from JSON. Apache Commons CLI is used for easier parsing of command-line arguments. These dependencies are included in the 'jars' folder inside each respective code directory.

## Compilation

In the following, `$basedir` refers to the absolute path to the base directory of this package on your system.

Compile the core library "smfcore" first with:

    cd $basedir/code/smfcore/src/smfcore

    javac -cp "$basedir/code/smfcore/jars/gson-2.3.1.jar" *.java

Then compile the "smf" (Finesse) application with:

    cd $basedir/code/smf/src/smf

    javac -cp "$basedir/code/smfcore/src/:$basedir/code/smf/jars/commons-cli-1.2.jar:." *.java

## Usage

    java -cp "$basedir/code/smfcore/src/:$basedir/code/smf/jars/commons-cli-1.2.jar:$basedir/smfcore/jars/gson-2.3.1.jar:$basedir/code/smf/src/" smf/Smf -k2 -s "$basedir/code/smfTestData/data.txt" -b "$basedir/code/smfTestData/basis.txt" -D "$basedir/code/smfTestData/usage.txt"

The parameters are:

- k: The decomposition rank (positive integer)
- s: The path to the input data file
- b: The path to the output basis file
- D: The path to the output usage file

## File format

Below is the format of `$basedir/code/smfTestData/data.txt`, which corresponds to the example Boolean data set on the first page of the SMF paper. The first two rows are the dimensions n and m respectively. Then, the columns are listed sequentially. Each column begins with its type (Boolean in this case) followed by the n values for that column. There are m columns.

    3
    3
    Boolean
    1
    0
    1
    Boolean
    1
    1
    1
    Boolean
    0
    1
    1

With this input file, the basis file that Finesse generates (`$basedir/code/smfTestData/basis.txt`), which in turn corresponds to the basis matrix in the first Boolean example in the paper, is:

    2
    3
    Boolean
    1
    0
    Boolean
    1
    1
    Boolean
    0
    1

The usage file that Finesse generates (`$basedir/code/smfTestData/usage.txt`), which in turn corresponds to the usage matrix in the first Boolean example in the paper, is:

    3
    2
    Boolean
    1
    0
    1
    Boolean
    0
    1
    1

(Note: Finesse is of course non-deterministic…the factors may be listed in a different order)

Other feature types are of course supported:

- The ternary feature has values 0 (false), 1 (unknown) and 2 (true). In the data matrix, use the string "Ternary" to specify that a column is ternary. Modify the dissimilarity measure in TernarySemiringFeature.java in the smfcore directory if need be (e.g. if you want missing-value dissimilarity). See also the TMF paper (Maurus, Plant, Ternary Matrix Factorization, ICDM '14).
- The ordinal feature has values 0 through (L-1), where L is the number of elements in the scale. In the data matrix, use the string "OrdinalBinU L" (where L is the integer number of elements in the scale) to specify that a column is ordinal and that the usage matrix should be binary (use "Ordinal L" if you want entries from the full scale in the usage matrix).
- The tree feature has set values. In the data matrix, use the JSON array syntax ["a","b","c"] for each value, where "a", "b" and "c" together represent the FULL collection of subtree nodes corresponding to that matrix entry. Use the string "Tree path" in the column header of the data matrix, where path is the absolute path to the master ontology (see below for more details).

For an example of a data file that uses all these features, check out e.g. the Yummly data in `$basedir/experiments/data/yummly/smf_data_mains_summer_american.txt`. That file shows how a tree feature needs the path to the corresponding master ontology.

Ontology files are in JSON format. The contents are a JSON serialization of an instance of the `Tree` class in the 'smfcore' code. That class has static methods `ToJson` and `FromJson` for serializing and deserializing. If you need to create your own master ontology, use the following approach in your own Java code that includes the 'smfcore' dependencies:

    Tree myTree = new Tree("rootId");
    // Code for adding (recursively perhaps) the required children to the tree.
    String serializedTree = Tree.ToJson(myTree);
    // Code for writing the serialized tree to a file.

#### Code: Synthetic Data-Set Generator Compilation and Usage

## Compilation

First, compile the 'smfcore' code (it's a dependency) as explained above. Then you can compile the 'ordinalsynthgen' code with:

    cd $basedir/code/ordinalsynthgen/src/ordinalsynthgen
    javac -cp "$basedir/code/smfcore/src/:$basedir/code/ordinalsynthgen/jars/commons-cli-1.2.jar" *.java

## Usage

    java -cp "$basedir/code/smfcore/src/:$basedir/code/ordinalsynthgen/src/:$basedir/code/ordinalsynthgen/jars/commons-cli-1.2.jar" ordinalsynthgen/Ordinalsynthgen -n8000 -m100 -k16 -o10 -u0 -b -1.1 -s7 -a1 -l3 -f "$basedir/code/smfTestData/synthData.txt" -r1

See the source code in `$basedir/code/ordinalsynthgen/src/ordinalsynthgen/Ordinalsynthgen.java` for an explanation of these parameters. The example above uses the default parameters from the paper (i.e. approximately uniform distribution of entries in the resulting matrix). Executing the above will overwrite the file `$basedir/code/smfTestData/synthData.txt` with a new synthetically-generated ordinal matrix based on these parameters. This data file can then be used as an input to Finesse.

#### Experiments: Data Sets

The "personality testing" data sets all came from personality-testing.info. We reported the results for the following ten data sets (here presented in the order as shown in the paper):

- Experinces in Close Relationships Scale ("Relationships")
- Feminist Perspectives Scale ("Feminism")
- IPIP Assertiveness, Social confidence, Adventurousness, and Dominance scales ("Assertiveness")
- Cattell's 16 Personality Factors Test ("Personality 1")
- Holland Code (RIASEC) Test ("Occupation")
- Humor Styles Questionnaire ("Humor styles")
- Kentucky Inventory of Mindfulness Skills ("Mindfulness")
- Open Sex Role Inventory ("Masc./Fem.")
- IPIP HEXACO equivalent scales ("Personality 2")
- Multidimensional Sexual Self-Concept Questionnaire ("Sexual self")

Some of the original data sets included additional features (e.g. participant ID) which were not relevant. During preprocessing, we hence ensured that only ordinal features were included. Any observations with missing values were removed. The original and preprocessed versions of the data (as presented to each algorithm), as well as the codebooks, are included in the directory `$basedir/experiments/data/psych`.

The Yummly and IMDB data used for the experiments in the paper are in `$basedir/experiments/data`. Each directory there has the SMF data file, the corresponding master ontology file (JSON) and a file with the list of the recipe/film names. Note that if you wish to reproduce the examples in the paper, you'll need to update the path to the master ontology in the data file in each case.

#### Experiments: Additional Plots

The file `$basedir/experiments/plots/supplement.pdf` includes the following plots (as discussed in the paper):

- Ordinal Matrix Factorization experiments that include the results from GreEss on smaller data sets, 50x50.
- Boolean Matrix Factorization experiments on large data sets (comparison with Asso and PaNDa).
- Justification of the model-order selection (k=6) for the Yummly data.
- Justification of the model-order selection (k=9) for the IMDB data.

#### Experiments: Detailed Results

The directory `$basedir/experiments/results` includes detailed results from experiments on the Yummly and IMDB data. Finesse's output basis and usage matrices are given for k=3 to k=16 in both cases.

#### Code: C++ Version

## Environment

The C++ of Finesse is written in C++ against the C++11 standard. The g++ (SUSE Linux) 4.3.4 compiler version was used. For bitwise operations it uses Streaming SIMD Extensions 2 (SSE2) instructions intrinsics (emmintrin.h, smmintrin.h). It was developed with Netbeans 8 on 64-bit ubuntu 14.04 LTS. Compilation on other operating systems "should" not cause any problems as long as the same intrinsics are available, but this was not tested.

## Dependencies

There are no dependencies for the C++ version.

## Compilation

In the following, `$basedir` refers to the absolute path to the base directory of this package on your system.

Compile the program with:

    cd $basedir/code/semize_cpp

    make all

Note the various flags that are passed to the C++ compiler during make. Specifically, `-msse4.1` is used for SSE and `-std=gnu++0x` for the C++ standard we want. The flag `-fopenmp` is included by default to exploit concurrency as described in the paper (simply remove this flag in the makefile to disable this feature).

When 'make all' is complete, the executable, called `semize`, is available in `dist/Release/GNU-Linux-x86/`.

## Usage (from within `$basedir/code/semize_cpp`)

    ./dist/Release/GNU-Linux-x86/semize -k2 -s "$basedir/code/smfTestData_cpp/data.txt" -b "$basedir/code/smfTestData_cpp/basis.txt" -D "$basedir/code/smfTestData_cpp/usage.txt"

The parameters are:

- k: The decomposition rank (positive integer)
- s: The path to the input data file
- b: The path to the output basis file
- D: The path to the output usage file

## File format

Below is the format of `$basedir/code/smfTestData_cpp/data.txt`. The first two rows are the dimensions n and m respectively. Then, the number of values in the ordinal scale is listed (2 in this case...Boolean). Then the matrix is written row-wise (dense format, not sparse).

    3
    3
    2
    1 1 0
    0 1 1
    1 1 1

With this input file, the transposed basis file that Finesse generates (`$basedir/code/smfTestData_cpp/basis.txt`) is:

    3
    2
    2
    0 1
    1 1
    1 0

The usage file that Finesse generates (`$basedir/code/smfTestData_cpp/usage.txt`) is:

    3
    2
    2
    0 1
    1 0
    1 1

(Note: Finesse is of course non-deterministic…the factors may be listed in a different order)

Note that the C++ version is only for ordinal problems (simply a prototype to demonstrate the high-performance options)...it does not yet support arbitrary feature types (use the Java version for this). Note also that when using the synthetic data generation tool above to generate data matrices for use with the C++ version, use the `-r0` option to specify the "normal" format, which is what the C++ version expects.