Evaluation and Plotting

Learned Space Visualization

To view the learned state and play with the latent space of a trained model, you may use:

python -m enjoy.enjoy_latent --log-dir logs/nameOfTheDataset/nameOfTheModel

Create a Report

A report contains knn-mse, ground-truth correlation (GTC and GTC_mean) and information about each model present in a log folder.

Create a csv report file using:

python -m evaluation.gather_results -i logs/nameOfTheDataset/

Plot a Learned Representation

usage: representation_plot.py [-h] [-i INPUT_FILE] [--data-folder DATA_FOLDER]
                              [--color-episode] [--plot-against]
                              [--pretty-plot-against] [--correlation]
                              [--projection] [--print-corr]

Plotting script for representation

optional arguments:
  -h, --help            show this help message and exit
  -i INPUT_FILE, --input-file INPUT_FILE
                        Path to a npz file containing states and rewards
  --data-folder DATA_FOLDER
                        Path to a dataset folder, it will plot ground truth
                        states
  --color-episode       Color states per episodes instead of reward
  --plot-against        Plot against each dimension
  --pretty-plot-against
                        Plot against each dimension (diagonals are
                        distributions + cleaner look)
  --correlation         Plot correlation coeff against each dimension
  --projection          Plot 1D projection of predicted state on ground truth
  --print-corr          Only print correlation measurements

You can plot a learned representation with:

python -m plotting.representation_plot -i path/to/states_rewards.npz

You can also plot ground truth states with:

python -m plotting.representation_plot --data-folder path/to/datasetFolder/

To have a different color per episode, you have to pass --data-folder argument along with --color-episode.

Plotting each dimension of the state representation against another:

python -m plotting.representation_plot -i path/to/states_rewards.npz --plot-against

[Evaluation plot] Plotting the matrix of correlation with the ground truth states:

python -m plotting.representation_plot -i path/to/states_rewards.npz --data-folder path/to/datasetFolder/ --correlation

[Ground Truth Correlation] It measures the maximum correlation (in absolute value) in the learned representation for each dimension of the ground truth states. GTC_mean is the mean of the GTC vector. :

python -m plotting.representation_plot -i path/to/states_rewards.npz --data-folder path/to/datasetFolder/ --correlation --print-corr

Interactive Plot

You can have an interactive plot of a learned representation using:

python -m plotting.interactive_plot --data-folder path/to/datasetFolder/ -i path/to/states_rewards.npz

When you click on a state in the representation plot (left click for 2D, right click for 3D plots!), it shows the corresponding image along with the reward and the coordinates in the space.

Pass --multi-view as argument to visualize in case of multiple cameras.

You can also plot ground truth states when you don’t specify a npz file:

python -m plotting.interactive_plot --data-folder path/to/datasetFolder/

Create a KNN Plot and Compute KNN-MSE

Usage:

python evaluation/knn_images.py [-h] --log-folder LOG_FOLDER [--seed SEED]
                     [-k N_NEIGHBORS] [-n N_SAMPLES] [--n-to-plot N_TO_PLOT]
                     [--relative-pos] [--ground-truth] [--multi-view]

KNN plot and KNN MSE

optional arguments:
  -h, --help            show this help message and exit
  --log-folder LOG_FOLDER
                        Path to a log folder
  --seed SEED           random seed (default: 1)
  -k N_NEIGHBORS, --n-neighbors N_NEIGHBORS
                        Number of nearest neighbors (default: 5)
  -n N_SAMPLES, --n-samples N_SAMPLES
                        Number of test samples (default: 5)
  --n-to-plot N_TO_PLOT
                        Number of samples to plot (default: 5)
  --relative-pos        Use relative position as ground_truth
  --ground-truth        Compute KNN-MSE for ground truth
  --multi-view          To deal with multi view data format

Example:

python evaluation/knn_images.py --log-folder path/to/an/experiment/log/folder