Evaluation & Experiments¶
- class pynlpl.evaluation.AbstractExperiment(inputdata=None, **parameters)¶
- defaultparameters()¶
- delete()¶
- done(warn=True)¶
Is the subprocess done?
- duration()¶
- run()¶
- sample(size)¶
Return a sample of the input data
- score()¶
- start()¶
Start as a detached subprocess, immediately returning execution to caller.
- startcommand(command, cwd, stdout, stderr, *arguments, **parameters)¶
- wait()¶
- class pynlpl.evaluation.ClassEvaluation(goals=[], observations=[], missing={}, encoding='utf-8')¶
- accuracy(cls=None)¶
- append(goal, observation)¶
- auc(cls=None, macro=False)¶
- compute()¶
- confusionmatrix(casesensitive=True)¶
- fp_rate(cls=None, macro=False)¶
- fscore(cls=None, beta=1, macro=False)¶
- outputmetrics()¶
- precision(cls=None, macro=False)¶
- recall(cls=None, macro=False)¶
- specificity(cls=None, macro=False)¶
- tp_rate(cls=None, macro=False)¶
- class pynlpl.evaluation.ConfusionMatrix(tokens=None, casesensitive=True, dovalidation=True)¶
Confusion Matrix
- class pynlpl.evaluation.ExperimentPool(size)¶
- append(experiment)¶
- poll(haltonerror=True)¶
- run(haltonerror=True)¶
- start(experiment)¶
- class pynlpl.evaluation.OrdinalEvaluation(goals=[], observations=[], missing={}, encoding='utf-8')¶
- compute()¶
- mae(cls=None)¶
- rmse(cls=None)¶
- class pynlpl.evaluation.ParamSearch(experimentclass, inputdata, parameterscope, poolsize=1, constraintfunc=None, delete=True)¶
A simpler version of ParamSearch without Wrapped Progressive Sampling
- exception pynlpl.evaluation.ProcessFailed¶
- class pynlpl.evaluation.WPSParamSearch(experimentclass, inputdata, size, parameterscope, poolsize=1, sizefunc=None, prunefunc=None, constraintfunc=None, delete=True)¶
ParamSearch with support for Wrapped Progressive Sampling
- searchbest()¶
- test(i=None)¶
- pynlpl.evaluation.auc(x, y, reorder=False)¶
Compute Area Under the Curve (AUC) using the trapezoidal rule
This is a general fuction, given points on a curve. For computing the area under the ROC-curve, see
auc_score()
.- Parameters:
x (array, shape = [n]) – x coordinates.
y (array, shape = [n]) – y coordinates.
reorder (boolean, optional (default=False)) – If True, assume that the curve is ascending in the case of ties, as for an ROC curve. If the curve is non-ascending, the result will be wrong.
- Returns:
auc
- Return type:
float
Examples
>>> import numpy as np >>> from sklearn import metrics >>> y = np.array([1, 1, 2, 2]) >>> pred = np.array([0.1, 0.4, 0.35, 0.8]) >>> fpr, tpr, thresholds = metrics.roc_curve(y, pred, pos_label=2) >>> metrics.auc(fpr, tpr) 0.75
See also
auc_score
Computes the area under the ROC curve
- pynlpl.evaluation.filesampler(files, testsetsize=0.1, devsetsize=0, trainsetsize=0, outputdir='', encoding='utf-8')¶
Extract a training set, test set and optimally a development set from one file, or multiple interdependent files (such as a parallel corpus). It is assumed each line contains one instance (such as a word or sentence for example).
- pynlpl.evaluation.mae(absolute_error_values)¶
- pynlpl.evaluation.rmse(squared_error_values)¶