Benchmark
- class pygmtools.benchmark.Benchmark(name, sets, obj_resize=(256, 256), problem='2GM', filter='intersection', **args)[source]
The Benchmark module provides a unified data interface and an evaluating platform for different datasets.
- Parameters
name – str, dataset name, currently support
'PascalVOC'
,'WillowObject'
,'IMC_PT_SparseGM'
,'CUB2011'
,'SPair71k'
sets – str, problem set,
'train'
for training set and'test'
for test setobj_resize – tuple, (default:
(256, 256)
) resized object sizeproblem – str, (default:
'2GM'
) problem type,'2GM'
for 2-graph matching and'MGM'
for multi-graph matchingfilter – str, (default:
'intersection'
) filter of nodes,'intersection'
refers to retaining only common nodes;'inclusion'
is only for 2GM and refers to filtering only one graph to make its nodes a subset of the other graph, and'unfiltered'
refers to retaining all nodes in all graphsargs – keyword settings for specific dataset
- compute_img_num(classes)[source]
Compute number of images in specified classes.
- Parameters
classes – list of dataset classes
- Returns
list of numbers of images in each class
- compute_length(cls=None, num=2)[source]
Compute the length of image combinations in specified class.
- Parameters
cls – int or str, class of expected data. None for all classes
num – int, number of images in each image ID list; for example, 2 for two-graph matching problem
- Returns
length of combinations
- eval(prediction, classes, verbose=False, rm_gt_cache=True)[source]
Evaluate test results and compute matching accuracy and coverage.
- Parameters
prediction – list, prediction result, like
[{'ids': (id1, id2), 'cls': cls, 'permmat': np.array or scipy.sparse}, ...]
classes – list of evaluated classes
verbose – bool, whether to print the result
rm_gt_cache – bool, whether to remove ground truth cache
- Returns
evaluation result in each class and their averages, including p, r, f1 and their standard deviation and coverage
Note
If there are duplicate data pair in
prediction
, this function will only evaluate the first pair and expect that this pair is also the first fetched pair. Therefore, it is recommended thatprediction
is built in an ordered manner, and not shuffled.Note
Ground truth cache is saved when data pairs are fetched, and should be removed after evaluation. Make sure all data pairs are evaluated at once, i.e.,
prediction
should contain all fetched data pairs.
- eval_cls(prediction, cls, verbose=False)[source]
Evaluate test results and compute matching accuracy and coverage on one specified class.
- Parameters
prediction – list, prediction result on one class, like
[{'ids': (id1, id2), 'cls': cls, 'permmat': np.array or scipy.sparse}, ...]
cls – str, evaluated class
verbose – bool, whether to print the result
- Returns
evaluation result on the specified class, including p, r, f1 and their standard deviation and coverage
Note
If there are duplicate data pair in
prediction
, this function will only evaluate the first pair and expect that this pair is also the first fetched pair. Therefore, it is recommended thatprediction
is built in an ordered manner, and not shuffled. Same as the functioneval
.Note
This function will not automatically remove ground truth cache. However, you can still mannually call the class function
rm_gt_cache
to remove groud truth cache after evaluation.
- get_data(ids, test=False, shuffle=True)[source]
Fetch a data pair or pairs of data by image ID for training or test.
- Parameters
ids – list of image ID, usually in
train.json
ortest.json
test – bool, whether the fetched data is used for test; if true, this function will not return ground truth
shuffle – bool, whether to shuffle the order of keypoints
- Returns
data_list: list of data, like
[{'img': np.array, 'kpts': coordinates of kpts}, ...]
perm_mat_dict: ground truth, like
{(0,1):scipy.sparse, (0,2):scipy.sparse, ...}
,(0,1)
refers to data pair(ids[0],ids[1])
ids: list of image ID
- get_id_combination(cls=None, num=2)[source]
Get the combination of images and length of combinations in specified class.
- Parameters
cls – int or str, class of expected data. None for all classes
num – int, number of images in each image ID list; for example, 2 for 2GM
- Returns
id_combination_list: list of combinations of image ids
length: length of combinations
- rand_get_data(cls=None, num=2, test=False, shuffle=True)[source]
Randomly fetch data for training or test. Implemented by calling
get_data
function.- Parameters
cls – int or str, class of expected data. None for random class
num – int, number of images; for example, 2 for 2GM
test – bool, whether the fetched data is used for test; if true, this function will not return ground truth
shuffle – bool, whether to shuffle the order of keypoints
- Returns
data_list: list of data, like
[{'img': np.array, 'kpts': coordinates of kpts}, ...]
perm_mat_dict: ground truth, like
{(0,1):scipy.sparse, (0,2):scipy.sparse, ...}
,(0,1)
refers to data pair(ids[0],ids[1])
ids: list of image ID