Commit b1ebd124 authored by Amir MOHAMMADI's avatar Amir MOHAMMADI

Merge branch 'hackathon2017' into 'master'

removed unnecessary commands starting with ./bin/

See merge request !4
parents e26d2f5a 62a4bcec
Pipeline #8997 passed with stages
in 29 minutes and 31 seconds
......@@ -8,33 +8,25 @@
The Command Line Interface
============================
The command line interface requires the package to be installed properly.
Fortunately, this is easy to be done, using the Buildout tools in the main directory of GridTK:
The command line interface requires the package to be installed properly (see the README file).
.. code-block:: sh
$ python boostrap.py
$ ./bin/buildout
These two commands will download all required dependencies and create a ``bin``
directory containing all the command line utilities that we will need in this
section. To verify the installation, you can call out nose tests:
When installation has finished, you should have all required dependencies and command line utilities in your environment. To verify the installation, you can call out nose tests:
.. code-block:: sh
$ ./bin/nosetests -sv
$ nosetests -sv
To build the package documentation, do:
.. code-block:: sh
$ ./bin/sphinx-build doc sphinx
$ sphinx-build doc sphinx
The Job Manager
===============
The most important utility is the Job Manager ``bin/jman``. This Job Manager
The most important utility is the Job Manager ``jman``. This Job Manager
can be used to:
* submit jobs
......@@ -55,13 +47,13 @@ Basic Job Manager Parameters
There are two versions of Job Managers: One that submits jobs to the SGE grid,
and one that submits jobs so that they are run in parallel on the local
machine. By default, the SGE manager is engaged. If you don't have access to
the SGE grid, or you want to submit locally, please issue the ``bin/jman
--local`` (or shortly ``bin/jman -l``) command.
the SGE grid, or you want to submit locally, please issue the ``jman
--local`` (or shortly ``jman -l``) command.
To keep track of the submitted jobs, an SQL3 database is written. This
database is by default called ``submitted.sql3`` and put in the current
directory, but this can be changed using the ``bin/jman --database``
(``bin/jman -d``) flag.
directory, but this can be changed using the ``jman --database``
(``jman -d``) flag.
Normally, the Job Manager acts silently, and only error messages are reported.
To make the Job Manager more verbose, you can use the ``--verbose`` (``-v``)
......@@ -72,32 +64,32 @@ option several times, to increase the verbosity level to 1) WARNING, 2) INFO,
Submitting Jobs
---------------
To submit a job, the ``bin/jman submit`` command is used.
To submit a job, the ``jman submit`` command is used.
The simplest way to submit a job to be run in the SGE grid is:
.. code-block:: sh
$ bin/jman -vv submit myscript.py
$ jman -vv submit myscript.py
This command will create an SQL3 database, submit the job to the grid and register it in the database.
To be more easily separable from other jobs in the database, you can give your job a name:
.. code-block:: sh
$ bin/jman -vv submit -n [name] myscript.py
$ jman -vv submit -n [name] myscript.py
If the job requires certain machine specifications, you can add these (please see the SGE manual for possible specifications of [key] and [value] pairs).
Please note the ``--`` option that separates specifications from the command:
.. code-block:: sh
$ bin/jman -vv submit -q [queue-name] -m [memory] --io-big -s [key1]=[value1] [key2]=[value2] -- myscript.py
$ jman -vv submit -q [queue-name] -m [memory] --io-big -s [key1]=[value1] [key2]=[value2] -- myscript.py
To have jobs run in parallel, you can submit a parametric job. Simply call:
.. code-block:: sh
$ bin/jman -vv submit -t 10 myscript.py
$ jman -vv submit -t 10 myscript.py
to run ``myscript.py`` 10 times in parallel. Each of the parallel jobs will
have a different environment variable called ``SGE_TASK_ID``, which will range
......@@ -111,7 +103,7 @@ started):
.. code-block:: sh
$ bin/jman -vv submit -x [job_id_1] [job_id_2] -- myscript.py
$ jman -vv submit -x [job_id_1] [job_id_2] -- myscript.py
In case the first job fails, it can automatically stop the depending jobs from
being executed. Just submit jobs with the ``--stop-on-failure`` option.
......@@ -127,7 +119,7 @@ This directory can be changed by specifying:
.. code-block:: sh
$ bin/jman -vv submit -l [log_dir]
$ jman -vv submit -l [log_dir]
.. note::
......@@ -147,7 +139,7 @@ local scheduler needs to be run. This is achieved by issuing the command:
.. code-block:: sh
$ bin/jman -vv run-scheduler -p [parallel_jobs] -s [sleep_time]
$ jman -vv run-scheduler -p [parallel_jobs] -s [sleep_time]
This will start the scheduler in the daemon mode. This will constantly monitor
the SQL3 database and execute jobs after submission, starting every
......@@ -173,7 +165,7 @@ line. To get the full command line, please use the ``-vv`` option:
.. code-block:: sh
$ bin/jman -vv list
$ jman -vv list
By default, array jobs are not listed, but the ``-a`` option changes this
behavior. Usually, it is a good idea to combine the ``-a`` option with ``-j``,
......@@ -181,7 +173,7 @@ which will list only the jobs of the given job id(s):
.. code-block:: sh
$ bin/jman -vv list -a -j [job_id_1] [job_id_2]
$ jman -vv list -a -j [job_id_1] [job_id_2]
Note that the ``-j`` option is in general relatively smart. You can use it to
select a range of job ids, e.g., ``-j 1-4 6-8``. In this case, please assert
......@@ -225,7 +217,7 @@ recorded and can be seen using the utilities. E.g.:
.. code-block:: sh
$ bin/jman -vv report -j [job_id] -a [array_id]
$ jman -vv report -j [job_id] -a [array_id]
will print the contents of the output and error log file from the job with the
desired ID (and only the array job with the given ID).
......@@ -238,12 +230,12 @@ Re-submitting the job
---------------------
After correcting your code you might want to submit the same command line
again. For this purpose, the ``bin/jman resubmit`` command exists. Simply
again. For this purpose, the ``jman resubmit`` command exists. Simply
specify the job id(s) that you want to resubmit:
.. code-block:: sh
$ bin/jman -vv resubmit -j [job_id_1] [job_id_2]
$ jman -vv resubmit -j [job_id_1] [job_id_2]
This will clean up the old log files (if you didn't specify the ``--keep-logs``
option) and re-submit the job. If the submission is done in the grid the job
......@@ -290,7 +282,7 @@ Cleaning up
-----------
After the job was successfully (or not) executed, you should clean up the
database using the ``bin/jman delete`` command. If not specified otherwise
database using the ``jman delete`` command. If not specified otherwise
(i.e., using the ``--keep-logs`` option), this command will delete all jobs
from the database and delete the log files (including the log directory in case
it is empty), and remove the database as well.
......@@ -301,7 +293,7 @@ only those jobs (and array jobs) with a certain status. E.g. use:
.. code-block:: sh
$ bin/jman -vv delete -s success -j 10-20
$ jman -vv delete -s success -j 10-20
to delete all jobs and the logs of all successfully finished jobs with job ids
from 10 to 20 from the database.
......@@ -313,11 +305,11 @@ Other command line tools
For convenience, we also provide additional command line tools, which are
mainly useful at Idiap. These tools are:
- ``bin/qstat.py``: writes the statuses of the jobs that are currently running
- ``qstat.py``: writes the statuses of the jobs that are currently running
in the SGE grid
- ``bin/qsub.py``: submit job to the SGE grid without logging them into the
- ``qsub.py``: submit job to the SGE grid without logging them into the
database
- ``bin/qdel.py``: delete job from the SGE grid without logging them into the
- ``qdel.py``: delete job from the SGE grid without logging them into the
database
- ``bin/grid``: executes the command in an grid environment (i.e., as if a
- ``grid``: executes the command in an grid environment (i.e., as if a
``SETSHELL grid`` command would have been issued before)
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment