Skip to content
Snippets Groups Projects

Dask annotators

Merged Amir MOHAMMADI requested to merge dask-annotators into master
4 unresolved threads

Merge request reports

Pipeline #45107 failed

Pipeline failed for 854eb190 on dask-annotators

Approval is optional

Merged by Amir MOHAMMADIAmir MOHAMMADI 4 years ago (Nov 10, 2020 3:33pm UTC)

Merge details

  • Changes merged into master with 42322c0a.
  • Deleted the source branch.

Pipeline #45326 failed

Pipeline failed for 42322c0a on master

Deployed to py‎pi‎ 4 years ago
Deployed to sta‎ble‎ 4 years ago
Canceled deployment to be‎ta‎

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
62 if key not in self.required_keys:
63 del kwargs['annotations'][key]
64 return kwargs['annotations']
37 def transform(self, samples, **kwargs):
38 for sample in samples:
39 if 'annotations' not in kwargs or kwargs['annotations'] is None:
40 kwargs['annotations'] = {}
41 for annotator in self.annotators:
42 try:
43 sample = annotator([sample], **kwargs)[0]
44 except Exception:
45 logger.debug(
46 "The annotator `%s' failed to annotate!", annotator,
47 exc_info=True)
48 sample.annotations = None
49 if not sample.annotations:
  • Amir MOHAMMADI
    Amir MOHAMMADI @amohammadi started a thread on commit 0a328a44
  • 10
    11 def __init__(self, **kwargs):
    12 super(DummyAnnotator, self).__init__(**kwargs)
    13
    14 def transform(self, sample, **kwargs):
    15 for s in sample:
    16 logger.debug(f"Annotating sample: {s.key}")
    17 s.annotations = {
    18 "time": time.localtime(),
    19 "rand": list(numpy.random.uniform(0,1,2))
    20 }
    21 time.sleep(0.1)
    22 return sample
    23
    24
    25 annotator = DummyAnnotator()
  • Yannick DAYER added 1 commit

    added 1 commit

    • 0b4640c7 - [py] Remove debug DummyAnnotator added by mistake.

    Compare with previous version

  • Yannick DAYER added 1 commit

    added 1 commit

    • 7c54afe1 - Revert passing Sample objects to Transformers

    Compare with previous version

  • Amir MOHAMMADI assigned to @ydayer and unassigned @amohammadi

    assigned to @ydayer and unassigned @amohammadi

  • Amir MOHAMMADI changed target branch from dask-pipelines to master

    changed target branch from dask-pipelines to master

  • Yannick DAYER added 1 commit

    added 1 commit

    • da6153af - [py] Apply changes to annotate_samples command

    Compare with previous version

  • Yannick DAYER unmarked as a Work In Progress

    unmarked as a Work In Progress

  • Yannick DAYER added 1 commit

    added 1 commit

    • fca57746 - [py] Adapt to annotations wrappers

    Compare with previous version

  • Yannick DAYER added 1 commit

    added 1 commit

    • c21b1b03 - [py] Change to new sample and checkpoint wrappers.

    Compare with previous version

  • Yannick DAYER added 1 commit

    added 1 commit

    • 854eb190 - annotate-samples now uses custom 'read' function.

    Compare with previous version

  • 35 35 self.only_required_keys = only_required_keys
    36 36
    37 def annotate(self, sample, **kwargs):
    37 def transform(self, sample_batch, **kwargs):
    38 38 if 'annotations' not in kwargs or kwargs['annotations'] is None:
    39 39 kwargs['annotations'] = {}
    40 for annotator in self.annotators:
    41 try:
    42 annotations = annotator(sample, **kwargs)
    43 except Exception:
    44 logger.debug(
    45 "The annotator `%s' failed to annotate!", annotator,
    46 exc_info=True)
    40 all_annotations = []
    41 for sample in sample_batch:
    42 annotations = kwargs['annotations'].copy()
  • 45 34 required=True,
    46 35 cls=ResourceOption,
    47 36 entry_point_group="bob.bio.annotator",
    48 help="A callable that takes the database and a sample (biofile) "
    49 "of the database and returns the annotations in a dictionary.",
    37 help="A Transformer instance that takes a series of sample and returns "
    38 "the modified samples with annotations as a dictionary.",
  • 128 for group in groups:
    129 references_samplesets.extend(database.references(group=group))
    130 probes_samplesets.extend(database.probes(group=group))
    131
    132 # Unravels all samples in one list (no SampleSets)
    133 samples = background_model_samples
    134 samples.extend([
    135 sample
    136 for r in references_samplesets
    137 for sample in r.samples
    138 ])
    139 samples.extend([
    140 sample
    141 for p in probes_samplesets
    142 for sample in p.samples
    143 ])
  • 228
    229 create_directories_safe(dirname(outpath))
    230 with open(outpath, "w") as f:
    231 json.dump(annot, f, indent=1, allow_nan=False)
    255 for s in samples
    256 ]
    257 # Splits the samples list into bags
    258 dask_bags = to_dask_bags.transform(samples_obj)
    259
    260 logger.info(f"Saving annotations in {output_dir}")
    261 logger.info(f"Annotating {len(samples_obj)} samples...")
    262 annotator.transform(dask_bags).compute(scheduler=scheduler)
    263
    264 if dask_client is not None:
    265 logger.info("Shutdown workers...")
    266 dask_client.shutdown()
  • @ydayer I will merge this. Let's address the concerns in another MR.

  • Amir MOHAMMADI mentioned in commit 42322c0a

    mentioned in commit 42322c0a

  • Please register or sign in to reply
    Loading