edu.udo.cs.yale.operator.validation
Class ValidationChain

java.lang.Object
  edu.udo.cs.yale.operator.Operator
      edu.udo.cs.yale.operator.OperatorChain
          edu.udo.cs.yale.operator.validation.ValidationChain

All Implemented Interfaces:

ConfigurationListener

Direct Known Subclasses:

FixedSplitValidationChain, RandomSplitValidationChain, XValidation

public abstract class ValidationChain
extends OperatorChain

Abstract superclass of operator chains that split an ExampleSet into a training and test set and return a performance vector. The two inner operators must be a learner returning a Model and an operator or operator chain that can apply this model and returns a PerformanceVector. Hence the second inner operator usually is an operator chain containing a model applier and a performance evaluator.

Version:

$Id: ValidationChain.java,v 1.32 2006/10/01 12:09:24 ingomierswa Exp $

Author:

Ingo Mierswa, Simon Fischer

Field Summary

private PerformanceCriterion	lastPerformance
private IOContainer	learnResult

Constructor Summary

ValidationChain(OperatorDescription description)

Method Summary

IOObject[]	apply() Applies all inner operators.
abstract IOObject[]	estimatePerformance(ExampleSet inputSet) This is the main method of the validation chain and must be implemented to estimate a performance of inner operators on the given example set.
protected IOContainer	evaluate(ExampleSet testSet) Applies the applier and evaluator (= second encapsulated inner operator).
private Operator	getEvaluator() Returns the second encapsulated inner operator (or operator chain), i.e.
InnerOperatorCondition	getInnerOperatorCondition() Must return a condition of the IO behaviour of all desired inner operators.
java.lang.Class[]	getInputClasses() Returns the the classes this operator provides as output.
InputDescription	getInputDescription(java.lang.Class cls) The default implementation returns an input description that consumes the input IOObject without a user parameter.
private Operator	getLearner() Returns the first encapsulated inner operator (or operator chain), i.e.
int	getMaxNumberOfInnerOperators() Returns the maximum number of innner operators.
int	getMinNumberOfInnerOperators() Returns the minimum number of innner operators.
int	getNumberOfSteps() Returns getNumberOfValidationSteps() * getNumberOfChildrensSteps() + 1.
abstract int	getNumberOfValidationSteps() Returns the number of validation steps, i.e. the number of times the inner operators of this validation chain are performed.
java.lang.Class[]	getOutputClasses() Returns the the classes this operator expects as input.
java.util.List<ParameterType>	getParameterTypes() Returns a list of ParameterTypes describing the parameters of this operator.
protected IOContainer	learn(ExampleSet trainingSet) Applies the learner (= first encapsulated inner operator).
protected void	setResult(PerformanceCriterion pc) Can be used by subclasses to set the performance of the example set.

Methods inherited from class edu.udo.cs.yale.operator.OperatorChain

addAddListener, addOperator, addOperator, checkDeprecations, checkIO, checkNumberOfInnerOperators, checkProperties, clearErrorList, clearStepCounter, cloneOperator, countStep, createExperimentTree, delete, experimentFinished, experimentStarts, getAllInnerOperators, getCurrentStep, getIndexOfOperator, getInnerOperatorForName, getInnerOperatorsXML, getNumberOfAllOperators, getNumberOfChildrensSteps, getNumberOfOperators, getOperator, getOperatorFromAll, getOperators, isEnabled, performAdditionalChecks, removeAddListener, removeOperator, setEnabled, setExperiment, shouldReturnInnerOutput

Methods inherited from class edu.udo.cs.yale.operator.Operator

addError, addValue, addWarning, apply, createExperimentTree, createFromXML, createMarkedExperimentTree, getAddOnlyAdditionalOutput, getApplyCount, getDeliveredOutputClasses, getDeprecationInfo, getDesiredInputClasses, getErrorList, getExperiment, getInput, getInput, getInput, getIOContainerForInApplyLoopBreakpoint, getName, getOperatorClassName, getOperatorDescription, getParameter, getParameterAsBoolean, getParameterAsColor, getParameterAsDouble, getParameterAsFile, getParameterAsInt, getParameterAsString, getParameterList, getParameters, getParameterType, getParent, getStartTime, getStatus, getUserDescription, getValue, getValues, getXML, hasBreakpoint, hasBreakpoint, hasInput, inApplyLoop, isParameterSet, logMessage, register, remove, rename, resume, setBreakpoint, setInput, setListParameter, setOperatorParameters, setParameter, setParameters, setParent, setUserDescription, toString, writeXML

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

lastPerformance

private PerformanceCriterion lastPerformance

learnResult

private IOContainer learnResult

Constructor Detail

ValidationChain

public ValidationChain(OperatorDescription description)

Method Detail

getNumberOfValidationSteps

public abstract int getNumberOfValidationSteps()

Returns the number of validation steps, i.e. the number of times the inner operators of this validation chain are performed. For example this method should returns the number of iterations in case of a crossvalidation.

estimatePerformance

public abstract IOObject[] estimatePerformance(ExampleSet inputSet)
                                        throws OperatorException

This is the main method of the validation chain and must be implemented to estimate a performance of inner operators on the given example set. The implementation can make use of the provided helper methods in this class.

Throws:

OperatorException

getMaxNumberOfInnerOperators

public int getMaxNumberOfInnerOperators()

Returns the maximum number of innner operators.

Specified by:

getMaxNumberOfInnerOperators in class OperatorChain

getMinNumberOfInnerOperators

public int getMinNumberOfInnerOperators()

Returns the minimum number of innner operators.

Specified by:

getMinNumberOfInnerOperators in class OperatorChain

getInputDescription

public InputDescription getInputDescription(java.lang.Class cls)

Description copied from class: Operator

The default implementation returns an input description that consumes the input IOObject without a user parameter. Subclasses may override this method to allow other input handling behaviors.

Overrides:

getInputDescription in class Operator

getInputClasses

public java.lang.Class[] getInputClasses()

Returns the the classes this operator provides as output.

Specified by:

getInputClasses in class Operator

getOutputClasses

public java.lang.Class[] getOutputClasses()

Returns the the classes this operator expects as input.

Specified by:

getOutputClasses in class Operator

getInnerOperatorCondition

public InnerOperatorCondition getInnerOperatorCondition()

Description copied from class: OperatorChain

Must return a condition of the IO behaviour of all desired inner operators. If there are no "special" conditions and the chain works similar to a simple operator chain this method should at least return a SimpleChainInnerOperatorCondition. More than one condition should be combined with help of the class CombinedInnerOperatorCondition.

Specified by:

getInnerOperatorCondition in class OperatorChain

getLearner

private Operator getLearner()

Returns the first encapsulated inner operator (or operator chain), i.e. the learning operator (chain).

getEvaluator

private Operator getEvaluator()

Returns the second encapsulated inner operator (or operator chain), i.e. the application and evaluation operator (chain)

setResult

protected void setResult(PerformanceCriterion pc)

Can be used by subclasses to set the performance of the example set.

apply

public IOObject[] apply()
                 throws OperatorException

Description copied from class: OperatorChain

Applies all inner operators. The input to this operator becomes the input of the first inner operator. The latter's output is passed to the second inner operator and so on. Note to subclassers: If subclasses (for example wrappers) want to make use of this method remember to call exactly this method (super.apply()) and do not call super.apply(IOContainer) erroneously which will result in an infinite loop. Subclasses who override this method without invoking super.apply() should at least invoke the method OperatorChain.clearStepCounter().

Overrides:

apply in class OperatorChain

Returns:

the last inner operator's output or the input itself if the chain is empty.

Throws:

OperatorException

learn

protected IOContainer learn(ExampleSet trainingSet)
                     throws OperatorException

Applies the learner (= first encapsulated inner operator).

Throws:

OperatorException

evaluate

protected IOContainer evaluate(ExampleSet testSet)
                        throws OperatorException

Applies the applier and evaluator (= second encapsulated inner operator). In order to reuse possibly created predicted label attributes, we do the following: We compare the predicted label of testSet before and after applying the inner operator. If it changed, the predicted label is removed again. No outer operator could ever see it. The same applies for the confidence attributes in case of classification learning.

Throws:

OperatorException

getNumberOfSteps

public int getNumberOfSteps()

Returns getNumberOfValidationSteps() * getNumberOfChildrensSteps() + 1.

Specified by:

getNumberOfSteps in class OperatorChain

getParameterTypes

public java.util.List<ParameterType> getParameterTypes()

Description copied from class: Operator

Returns a list of ParameterTypes describing the parameters of this operator. The default implementation returns an empty list if no input objects can be retained and special parameters for those input objects which can be prevented from being consumed.

Overrides:

getParameterTypes in class Operator