SGE defaults

Most of the modifications are changing scheduler defaults but temporally added biometric as a default for the SGE -P option.

I'll merge it because I need this quickly to be on the master for this afternoon.

bob/pipelines/distributed/sge.py

@@ -37,7 +37,7 @@ class SGEIdiapJob(Job):
         self,
         *args,
         queue=None,
-        project=None,
+        project="biometric",
         resource_spec=None,
         job_extra=None,
         config_name="sge",
@@ -278,6 +278,7 @@ class SGEMultipleQueuesCluster(JobQueueCluster):
             "io_big=TRUE," if "io_big" in job_spec and job_spec["io_big"] else ""
         )
 
+
         memory = _get_key_from_spec(job_spec, "memory")[:-1]
         new_resource_spec += (f"mem_free={memory},")
 
@@ -289,7 +290,7 @@ class SGEMultipleQueuesCluster(JobQueueCluster):
 
         return {
             "queue": queue,
-            "memory": "0",
+            "memory": _get_key_from_spec(job_spec, "memory"),
             "cores": 1,
             "processes": 1,
             "log_directory": self.log_directory,
@@ -445,9 +446,9 @@ class SchedulerResourceRestriction(Scheduler):
 
     def __init__(self, *args, **kwargs):
         super(SchedulerResourceRestriction, self).__init__(
-            idle_timeout=3600,
-            allowed_failures=500,
-            synchronize_worker_interval="240s",
+            idle_timeout=60,
+            allowed_failures=100,
+            synchronize_worker_interval="60s",
             *args,
             **kwargs,
         )

[Amir MOHAMMADI]

why not use bob.extension.rc for this?

[Tiago de Freitas Pereira]

RC should be optional

[Tiago de Freitas Pereira]

something like

project = rc["bob.pipelines.sge-project"] if rc["bob.pipelines.sge-project"] else None

[Samuel GAIST]

Python already provides something cleaner:

project = rc.get("bob.pipelines.sge-project")

By default get returns None but you can either put it explicitly or any other sensible defaults. That is unless you expect the content matching that key to be "falsy".

[Amir MOHAMMADI]

I think you can also name the key just sge.project as it may become relevant in other bob packages.