Benchmark

class pygmtools.benchmark.Benchmark(name, sets, obj_resize=(256, 256), problem='2GM', filter='intersection', **args)[source]

The Benchmark module provides a unified data interface and an evaluating platform for different datasets.

Parameters
  • name – str, dataset name, currently support ‘PascalVOC’, ‘WillowObject’, ‘IMC_PT_SparseGM’, ‘CUB2011’, ‘SPair71k’

  • sets – str, problem set, ‘train’ for training set and ‘test’ for test set

  • obj_resize – tuple, resized object size

  • problem – str, problem type, ‘2GM’ for 2-graph matching and ‘MGM’ for multi-graph matching

  • filter – str, filter of nodes, ‘intersection’ refers to retaining only common nodes; ‘inclusion’ is only for 2GM and refers to filtering only one graph to make its nodes a subset of the other graph, and ‘unfiltered’ refers to retaining all nodes in all graphs

  • args – specific settings for dataset

compute_img_num(classes)[source]

Compute number of images in specified classes.

Parameters

classes – list of dataset classes

Returns

list of numbers of images in each class

compute_length(cls=None, num=2)[source]

Compute the length of image combinations in specified class.

Parameters
  • cls – int or str, class of expected data. None for all classes

  • num – int, number of images in each image ID list; for example, 2 for 2GM

Returns

length of combinations

eval(prediction, classes, verbose=False)[source]

Evaluate test results and compute matching accuracy and coverage.

Parameters
  • prediction – list, prediction result, like [{'ids': (id1, id2), 'cls': cls, 'permmat': np.array or scipy.sparse},…]

  • classes – list of evaluated classes

  • verbose – bool, whether to print the result

Returns

evaluation result in each class and their averages, including p, r, f1 and their standard deviation and coverage

eval_cls(prediction, cls, verbose=False)[source]

Evaluate test results and compute matching accuracy and coverage on one specified class.

Parameters
  • prediction – list, prediction result on one class, like [{'ids': (id1, id2), 'cls': cls, 'permmat': np.array or scipy.sparse},…]

  • cls – str, evaluated class

  • verbose – bool, whether to print the result

Returns

evaluation result on the specified class, including p, r, f1 and their standard deviation and coverage

get_data(ids, test=False, shuffle=True)[source]

Fetch a data pair or pairs of data by image ID for training or test.

Parameters
  • ids – list of image ID, usually in ‘train.json’ or ‘test.json’

  • test – bool, whether the fetched data is used for test; if true, this function will not return ground truth

  • shuffle – bool, whether to shuffle the order of keypoints

Returns

data_list: list of data, like [{‘img’: np.array, ’kpts’: coordinates of kpts}, …]

perm_mat_dict: ground truth, like {(0,1):scipy.sparse, (0,2):scipy.sparse, …}, (0,1) refers to data pair (ids[0],ids[1])

ids: list of image ID

get_id_combination(cls=None, num=2)[source]

Get the combination of images and length of combinations in specified class.

Parameters
  • cls – int or str, class of expected data. None for all classes

  • num – int, number of images in each image ID list; for example, 2 for 2GM

Returns

id_combination_list: list of combinations of image ids

length: length of combinations

rand_get_data(cls=None, num=2, test=False, shuffle=True)[source]

Randomly fetch data for training or test. Implemented by calling get_data function.

Parameters
  • cls – int or str, class of expected data. None for random class

  • num – int, number of images; for example, 2 for 2GM

  • test – bool, whether the fetched data is used for test; if true, this function will not return ground truth

  • shuffle – bool, whether to shuffle the order of keypoints

Returns

data_list: list of data, like [{‘img’: np.array, ’kpts’: coordinates of kpts}, …]

perm_mat_dict: ground truth, like {(0,1):scipy.sparse, (0,2):scipy.sparse, …}, (0,1) refers to data pair (ids[0],ids[1])

ids: list of image ID

rm_gt_cache(last_epoch=False)[source]

Remove ground truth cache.

Parameters

last_epoch – Boolean variable, whether this epoch is last epoch; if true, the directory of cache will also be removed.