edu.udo.cs.yale.operator.performance
Class PerformanceEvaluator

java.lang.Object
  edu.udo.cs.yale.operator.Operator
      edu.udo.cs.yale.operator.performance.PerformanceEvaluator

All Implemented Interfaces:

ConfigurationListener

public class PerformanceEvaluator
extends Operator

A performance evaluator is an operator that expects a test ExampleSet as input, whose elements have both true and predicted labels, and delivers as output a list of performance values according to a list of performance criteria that it calculates. If an input performance vector was already given, this is used for keeping the performance values.

All of the performance criteria can be switched on using boolean parameters. Their values can be queried by a ExperimentLogOperator using the same names. The main criterion is used for comparisons and need to be specified only for experiments where performance vectors are compared, e.g. feature selection experiments. If no other main criterion was selected the first criterion in the resulting performance vector will be assumed to be the main criterion.

Additional user-defined implementations of PerformanceCriterion can be specified by using the parameter list additional_performance_criteria. Each key/value pair in this list must specify a fully qualified classname (as the key), and a string parameter (as value) that is passed to the constructor. Please make sure that the class files are in the classpath (this is the case if the implementations are supplied by a plugin) and that they implement a one-argument constructor taking a string parameter. It must also be ensured that these classes extend MeasuredPerformance since the PerformanceEvaluator operator will only support these criteria. Please note that only the first three user defined criteria can be used as logging value with names "user1", ... , "user3".

The resulting performance vectors are usually compared with a standard performance comparator which only compares the fitness values of the main criterion. Other implementations than this simple comparator can be specified using the parameter comparator_class. This may for instance be useful if you want to compare performance vectors according to the weighted sum of the individual criteria. In order to implement your own comparator, simply subclass PerformanceComparator. Please note that for true multi-objective optimization usually another selection scheme is used instead of simply replacing the performance comparator.

Version:

$Id: PerformanceEvaluator.java,v 2.53 2006/04/15 02:56:07 martin_scholz Exp $

Author:

Simon Fischer, Ingo Mierswa

Field Summary

private static java.lang.String[]	allCriteriaDescriptions
private static java.lang.String[]	allCriteriaNames
private static java.util.Map<java.lang.String,java.lang.Class>	classnameMap Maps criteria names to classes.
private static java.lang.Class[]	CRITERIA_CLASSES The proper criteria to the names.
private static java.lang.String[]	CRITERIA_NAMES Names of the performance criteria which can be used in experiment configuration files.
private PerformanceVector	currentPerformanceVector The currently used performance vector.
private static int	NUMBER_OF_USER_CRITERIA The number of allowed user criteria.
private java.util.ArrayList<PerformanceCriterion>	userCriteria This list is needed for logging purposes of user defined criteria.

Constructor Summary

PerformanceEvaluator(OperatorDescription description)

Method Summary

private void	addPerformanceValue(java.lang.String name, java.lang.String description) Adds the performance criteria as plottable values, e.g. for the ExperimentLog operator.
IOObject[]	apply() Implement this method in subclasses.
private boolean	checkCriterionName(java.lang.String name) Returns true if the criterion with the given name should be added to the performance vector.
protected PerformanceVector	evaluate(ExampleSet testSet, PerformanceVector inputPerformance) Evaluates the given test set.
static void	evaluate(PerformanceEvaluator evaluator, ExampleSet testSet, PerformanceVector performanceCriteria, java.util.List<PerformanceCriterion> givenCriteria, boolean skipUndefinedLabels) Static version of evaluate(ExampleSet,PerformanceVector).
java.lang.Class[]	getInputClasses() Returns the classes that are needed as input.
InputDescription	getInputDescription(java.lang.Class cls) Shows a parameter keep_example_set with default value "false&quot.
java.lang.Class[]	getOutputClasses() Returns the classes that are guaranteed to be returned by apply() as additional output.
java.util.List<ParameterType>	getParameterTypes() Returns a list of ParameterTypes describing the parameters of this operator.
private PerformanceVector	initialisePerformanceVector(ExampleSet testSet, PerformanceVector performanceCriteria, java.util.List<PerformanceCriterion> givenCriteria) Creates a new performance vector if the given one is null.

Methods inherited from class edu.udo.cs.yale.operator.Operator

addError, addValue, addWarning, apply, checkDeprecations, checkIO, checkProperties, clearErrorList, cloneOperator, createExperimentTree, createExperimentTree, createFromXML, createMarkedExperimentTree, delete, experimentFinished, experimentStarts, getAddOnlyAdditionalOutput, getApplyCount, getDeliveredOutputClasses, getDeprecationInfo, getDesiredInputClasses, getErrorList, getExperiment, getInnerOperatorsXML, getInput, getInput, getInput, getIOContainerForInApplyLoopBreakpoint, getName, getNumberOfSteps, getOperatorClassName, getOperatorDescription, getParameter, getParameterAsBoolean, getParameterAsColor, getParameterAsDouble, getParameterAsFile, getParameterAsInt, getParameterAsString, getParameterList, getParameters, getParameterType, getParent, getStartTime, getStatus, getUserDescription, getValue, getValues, getXML, hasBreakpoint, hasBreakpoint, hasInput, inApplyLoop, isEnabled, isParameterSet, logMessage, performAdditionalChecks, register, remove, rename, resume, setBreakpoint, setEnabled, setExperiment, setInput, setListParameter, setOperatorParameters, setParameter, setParameters, setParent, setUserDescription, toString, writeXML

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

NUMBER_OF_USER_CRITERIA

private static final int NUMBER_OF_USER_CRITERIA

The number of allowed user criteria. This is necessary for plotting purposes.

See Also:

Constant Field Values

CRITERIA_NAMES

private static final java.lang.String[] CRITERIA_NAMES

Names of the performance criteria which can be used in experiment configuration files.

CRITERIA_CLASSES

private static final java.lang.Class[] CRITERIA_CLASSES

The proper criteria to the names.

allCriteriaNames

private static java.lang.String[] allCriteriaNames

allCriteriaDescriptions

private static java.lang.String[] allCriteriaDescriptions

classnameMap

private static java.util.Map<java.lang.String,java.lang.Class> classnameMap

Maps criteria names to classes.

userCriteria

private java.util.ArrayList<PerformanceCriterion> userCriteria

This list is needed for logging purposes of user defined criteria.

currentPerformanceVector

private PerformanceVector currentPerformanceVector

The currently used performance vector. This is be used for logging / plotting purposes.

Constructor Detail

PerformanceEvaluator

public PerformanceEvaluator(OperatorDescription description)

Method Detail

addPerformanceValue

private void addPerformanceValue(java.lang.String name,
                                 java.lang.String description)

Adds the performance criteria as plottable values, e.g. for the ExperimentLog operator.

initialisePerformanceVector

private PerformanceVector initialisePerformanceVector(ExampleSet testSet,
                                                      PerformanceVector performanceCriteria,
                                                      java.util.List<PerformanceCriterion> givenCriteria)
                                               throws OperatorException

Creates a new performance vector if the given one is null. Adds all criteria demanded by the user. If the criterion was already part of the performance vector before it will be overwritten.

Throws:

OperatorException

checkCriterionName

private boolean checkCriterionName(java.lang.String name)
                            throws UndefinedParameterError

Returns true if the criterion with the given name should be added to the performance vector. This is either the case

1. if the boolean parameter was selected by the user
2. if the given name is equal to the main criterion

Throws:

UndefinedParameterError

apply

public IOObject[] apply()
                 throws OperatorException

Description copied from class: Operator

Implement this method in subclasses.

Specified by:

apply in class Operator

Throws:

OperatorException

evaluate

protected PerformanceVector evaluate(ExampleSet testSet,
                                     PerformanceVector inputPerformance)
                              throws OperatorException

Evaluates the given test set. All PerformanceCriterion instances in the given PerformanceVector must be subclasses of MeasuredPerformance.

Throws:

OperatorException

evaluate

public static void evaluate(PerformanceEvaluator evaluator,
                            ExampleSet testSet,
                            PerformanceVector performanceCriteria,
                            java.util.List<PerformanceCriterion> givenCriteria,
                            boolean skipUndefinedLabels)
                     throws OperatorException

Static version of evaluate(ExampleSet,PerformanceVector). This method was introduced to enable testing of the method.

Parameters:

evaluator - Ususally this. May be null for testing. Only needed for exception.

Throws:

OperatorException

getInputDescription

public InputDescription getInputDescription(java.lang.Class cls)

Shows a parameter keep_example_set with default value "false&quot.

Overrides:

getInputDescription in class Operator

getInputClasses

public java.lang.Class[] getInputClasses()

Description copied from class: Operator

Returns the classes that are needed as input. May be null or an empty (no desired input). As default, all delivered input objects are consumed and must be also delivered as output in both Operator.getOutputClasses() and Operator.apply() if this is necessary. This default behavior can be changed by overriding Operator.getInputDescription(Class). Subclasses which implement this method should not make use of parameters since this method is invoked by getParameterTypes(). Therefore, parameters are not fully available at this point of time and this might lead to exceptions. Please use InputDescriptions instead.

Specified by:

getInputClasses in class Operator

getOutputClasses

public java.lang.Class[] getOutputClasses()

Description copied from class: Operator

Returns the classes that are guaranteed to be returned by apply() as additional output. Please note that input object which should not be consumed must also be defined by this method (e.g. for preprocessing operators). The default behavior for input consumation is defined by Operator.getInputDescription(Class) and can be changed by overwriting this method. Objects which are not consumed must not be defined as additional output in this method. May be null or an empy array (no additional output is produced).

Specified by:

getOutputClasses in class Operator

getParameterTypes

public java.util.List<ParameterType> getParameterTypes()

Description copied from class: Operator

Returns a list of ParameterTypes describing the parameters of this operator. The default implementation returns an empty list if no input objects can be retained and special parameters for those input objects which can be prevented from being consumed.

Overrides:

getParameterTypes in class Operator