Dask annotators
4 unresolved threads
4 unresolved threads
Merge request reports
Activity
changed milestone to %Bob 9.0.0
added 2 commits
added 1 commit
- 0a328a44 - [py] Ported the 'annotate' command to pipelines.
62 if key not in self.required_keys: 63 del kwargs['annotations'][key] 64 return kwargs['annotations'] 37 def transform(self, samples, **kwargs): 38 for sample in samples: 39 if 'annotations' not in kwargs or kwargs['annotations'] is None: 40 kwargs['annotations'] = {} 41 for annotator in self.annotators: 42 try: 43 sample = annotator([sample], **kwargs)[0] 44 except Exception: 45 logger.debug( 46 "The annotator `%s' failed to annotate!", annotator, 47 exc_info=True) 48 sample.annotations = None 49 if not sample.annotations: - bob/bio/base/annotator/dummy.py 0 → 100644
10 11 def __init__(self, **kwargs): 12 super(DummyAnnotator, self).__init__(**kwargs) 13 14 def transform(self, sample, **kwargs): 15 for s in sample: 16 logger.debug(f"Annotating sample: {s.key}") 17 s.annotations = { 18 "time": time.localtime(), 19 "rand": list(numpy.random.uniform(0,1,2)) 20 } 21 time.sleep(0.1) 22 return sample 23 24 25 annotator = DummyAnnotator() added 1 commit
- 0b4640c7 - [py] Remove debug DummyAnnotator added by mistake.
added 1 commit
- 7c54afe1 - Revert passing Sample objects to Transformers
- Resolved by Amir MOHAMMADI
assigned to @ydayer and unassigned @amohammadi
added 1 commit
- da6153af - [py] Apply changes to annotate_samples command
- Resolved by Amir MOHAMMADI
@ydayer when discussing #143 (closed) I realized we need an annotation wrapper that does:
sample.annotations = annotator(sample.data)
which can probably be done by changing the sample wrapper in bob.pipelines
Edited by Amir MOHAMMADI
added 1 commit
- c21b1b03 - [py] Change to new sample and checkpoint wrappers.
added 1 commit
- 854eb190 - annotate-samples now uses custom 'read' function.
35 35 self.only_required_keys = only_required_keys 36 36 37 def annotate(self, sample, **kwargs): 37 def transform(self, sample_batch, **kwargs): 38 38 if 'annotations' not in kwargs or kwargs['annotations'] is None: 39 39 kwargs['annotations'] = {} 40 for annotator in self.annotators: 41 try: 42 annotations = annotator(sample, **kwargs) 43 except Exception: 44 logger.debug( 45 "The annotator `%s' failed to annotate!", annotator, 46 exc_info=True) 40 all_annotations = [] 41 for sample in sample_batch: 42 annotations = kwargs['annotations'].copy() @ydayer annotations in kwargs would be a list of dictionaries so you can't copy it like this.
45 34 required=True, 46 35 cls=ResourceOption, 47 36 entry_point_group="bob.bio.annotator", 48 help="A callable that takes the database and a sample (biofile) " 49 "of the database and returns the annotations in a dictionary.", 37 help="A Transformer instance that takes a series of sample and returns " 38 "the modified samples with annotations as a dictionary.", returns annotations as dictionary
. You should probably mention here that the annotator must inherit from ... class.Edited by Amir MOHAMMADI
128 for group in groups: 129 references_samplesets.extend(database.references(group=group)) 130 probes_samplesets.extend(database.probes(group=group)) 131 132 # Unravels all samples in one list (no SampleSets) 133 samples = background_model_samples 134 samples.extend([ 135 sample 136 for r in references_samplesets 137 for sample in r.samples 138 ]) 139 samples.extend([ 140 sample 141 for p in probes_samplesets 142 for sample in p.samples 143 ]) 228 229 create_directories_safe(dirname(outpath)) 230 with open(outpath, "w") as f: 231 json.dump(annot, f, indent=1, allow_nan=False) 255 for s in samples 256 ] 257 # Splits the samples list into bags 258 dask_bags = to_dask_bags.transform(samples_obj) 259 260 logger.info(f"Saving annotations in {output_dir}") 261 logger.info(f"Annotating {len(samples_obj)} samples...") 262 annotator.transform(dask_bags).compute(scheduler=scheduler) 263 264 if dask_client is not None: 265 logger.info("Shutdown workers...") 266 dask_client.shutdown() @ydayer I will merge this. Let's address the concerns in another MR.
mentioned in commit 42322c0a
Please register or sign in to reply