Skip to content

Created 2 commands to process gitlab logs

Tiago de Freitas Pereira requested to merge process-log into master

These commands are useful if you want to process gitlab-ci logs.

For instance,

bdt gitlab get-pipelines bob/bob.nightlies

gets all the recent pipelines from bob/bob.nightlies. Something like this

Jobs from project bob/bob.nightlies
--------  -------------  --------  --------------------------------------------------------
Pipeline  Branch         Status    Jobs
34568     big-issue-106  failed    [178106, 178107, 178108]
34567     master         success   [178100, 178101, 178102, 178103, 178104, 178105]

Finally,

bdt gitlab process-pipelines bob/bob.nightlies 34567 --job-id 178101

summarizes execution time of a particular pipeline job. If nightlies is set, the stdout is split by project and we have an output like the one below.

Pipeline 34567, Job 178101
-----------------------------------------------  -------------------  -------
None                                             2019-10-20 19:19:37  0.5m
bob/bob.buildout@master (1/120                   2019-10-20 19:20:08  0.97m
bob/bob.extension@master (2/120                  2019-10-20 19:21:07  2.35m
bob/bob.blitz@master (3/120                      2019-10-20 19:23:28  11.7m
bob/bob.core@master (4/120                       2019-10-20 19:35:10  15.25m
bob/bob.io.base@master (5/120                    2019-10-20 19:50:25  16.57m
bob/bob.math@master (6/120                       2019-10-20 20:06:59  13.68m
bob/bob.measure@master (7/120                    2019-10-20 20:20:40  31.5m
bob/bob.io.image@master (8/120                   2019-10-20 20:52:10  20.9m
bob/bob.db.base@master (9/120                    2019-10-20 21:13:04  1.2m
bob/bob.io.video@master (10/120                  2019-10-20 21:14:16  18.03m
bob/bob.io.matlab@master (11/120                 2019-10-20 21:32:18  12.57m
bob/bob.io.audio@master (12/120                  2019-10-20 21:44:52  15.85m
bob/bob.sp@master (13/120                        2019-10-20 22:00:43  13.7m
bob/bob.ap@master (14/120                        2019-10-20 22:14:25  14.55m
bob/bob.ip.base@master (15/120                   2019-10-20 22:28:58  22.72m
bob/bob.ip.color@master (16/120                  2019-10-20 22:51:41  13.87m
bob/bob.ip.draw@master (17/120                   2019-10-20 23:05:33  10.55m
bob/bob.ip.gabor@master (18/120                  2019-10-20 23:16:06  37.2m
bob/bob.learn.activation@master (19/120          2019-10-20 23:53:18  15.4m
bob/bob.learn.libsvm@master (20/120              2019-10-21 00:08:42  16.27m
bob/bob.learn.linear@master (21/120              2019-10-21 00:24:58  19.0m
bob/bob.learn.mlp@master (22/120                 2019-10-21 00:43:58  16.87m
bob/bob.learn.boosting@master (23/120            2019-10-21 01:00:50  16.17m
bob/bob.db.iris@master (24/120                   2019-10-21 01:17:00  1.42m
bob/bob.learn.em@master (25/120                  2019-10-21 01:18:25  24.52m
bob/bob.db.wine@master (26/120                   2019-10-21 01:42:56  0.95m
bob/bob.db.mnist@master (27/120                  2019-10-21 01:43:53  1.05m
bob/bob.db.atnt@master (28/120                   2019-10-21 01:44:56  1.1m
bob/bob.ip.facedetect@master (29/120             2019-10-21 01:46:02  15.72m
bob/bob.ip.optflow.hornschunck@master (30/120    2019-10-21 02:01:46  13.18m
bob/bob.ip.optflow.liu@master (31/120            2019-10-21 02:14:57  15.88m
bob/bob.ip.flandmark@master (32/120              2019-10-21 02:30:50  31.0m
bob/gridtk@master (33/120                        2019-10-21 03:01:50  1.17m
bob/bob.ip.qualitymeasure@master (34/120         2019-10-21 03:03:00  17.63m
bob/bob.ip.skincolorfilter@master (35/120        2019-10-21 03:20:38  1.85m
bob/bob.ip.facelandmarks@master (36/120          2019-10-21 03:22:30  3.2m
bob/bob.ip.dlib@master (37/120                   2019-10-21 03:25:43  2.37m
bob/bob.ip.mtcnn@master (38/120                  2019-10-21 03:28:05  2.37m
bob/bob.db.arface@master (39/120                 2019-10-21 03:30:27  1.27m
bob/bob.db.asvspoof@master (40/120               2019-10-21 03:31:43  3.55m
bob/bob.db.asvspoof2017@master (41/120           2019-10-21 03:35:16  0.87m
bob/bob.db.atvskeystroke@master (42/120          2019-10-21 03:36:09  0.88m
bob/bob.db.avspoof@master (43/120                2019-10-21 03:37:02  2.65m
bob/bob.db.banca@master (44/120                  2019-10-21 03:39:41  1.67m
bob/bob.db.biosecure@master (45/120              2019-10-21 03:41:21  0.83m
bob/bob.db.biosecurid.face@master (46/120        2019-10-21 03:42:11  0.87m
bob/bob.db.casia_fasd@master (47/120             2019-10-21 03:43:04  0.97m
bob/bob.db.casme2@master (48/120                 2019-10-21 03:44:02  0.85m
bob/bob.db.caspeal@master (49/120                2019-10-21 03:44:53  0.93m
bob/bob.db.cohface@master (50/120                2019-10-21 03:45:49  1.57m
bob/bob.db.frgc@master (51/120                   2019-10-21 03:47:24  0.8m
bob/bob.db.gbu@master (52/120                    2019-10-21 03:48:12  0.98m
bob/bob.db.hci_tagging@master (53/120            2019-10-21 03:49:11  6.9m
bob/bob.db.ijba@master (54/120                   2019-10-21 03:56:07  3.17m
bob/bob.db.ijbc@master (55/120                   2019-10-21 03:59:18  14.1m
bob/bob.db.kboc16@master (56/120                 2019-10-21 04:13:29  0.9m
bob/bob.db.lfw@master (57/120                    2019-10-21 04:14:23  2.77m
bob/bob.db.livdet2013@master (58/120             2019-10-21 04:17:09  0.8m
bob/bob.db.mobio@master (59/120                  2019-10-21 04:17:57  1.7m
bob/bob.db.msu_mfsd_mod@master (60/120           2019-10-21 04:19:39  1.37m
bob/bob.db.multipie@master (61/120               2019-10-21 04:21:02  1.83m
bob/bob.db.nist_sre12@master (62/120             2019-10-21 04:22:52  7.33m
bob/bob.db.putvein@master (63/120                2019-10-21 04:30:13  0.97m
bob/bob.db.replay@master (64/120                 2019-10-21 04:31:11  1.2m
bob/bob.db.replaymobile@master (65/120           2019-10-21 04:32:23  1.48m
bob/bob.db.scface@master (66/120                 2019-10-21 04:33:52  0.93m
bob/bob.db.utfvp@master (67/120                  2019-10-21 04:34:48  3.92m
bob/bob.db.verafinger@master (68/120             2019-10-21 04:38:43  1.67m
bob/bob.db.fv3d@master (69/120                   2019-10-21 04:40:23  1.98m
bob/bob.db.hkpu@master (70/120                   2019-10-21 04:42:22  1.7m
bob/bob.db.thufvdt@master (71/120                2019-10-21 04:44:04  1.97m
bob/bob.db.mmcbnu6k@master (72/120               2019-10-21 04:46:02  2.27m
bob/bob.db.hmtvein@master (73/120                2019-10-21 04:48:18  1.72m
bob/bob.db.voicepa@master (74/120                2019-10-21 04:50:01  6.37m
bob/bob.db.xm2vts@master (75/120                 2019-10-21 04:56:23  1.03m
bob/bob.db.youtube@master (76/120                2019-10-21 04:57:26  1.8m
bob/bob.db.pericrosseye@master (77/120           2019-10-21 04:59:15  0.98m
bob/bob.db.maskattack@master (78/120             2019-10-21 05:00:14  1.18m
bob/bob.db.casiasurf@master (79/120              2019-10-21 05:01:25  1.48m
bob/bob.db.fargo@master (80/120                  2019-10-21 05:02:54  1.2m
bob/bob.bio.base@master (81/120                  2019-10-21 05:04:06  3.98m
bob/bob.bio.gmm@master (82/120                   2019-10-21 05:08:05  8.3m
bob/bob.bio.face@master (83/120                  2019-10-21 05:16:23  4.97m
bob/bob.bio.spear@master (84/120                 2019-10-21 05:21:21  2.88m
bob/bob.bio.video@master (85/120                 2019-10-21 05:24:14  2.35m
bob/bob.bio.vein@master (86/120                  2019-10-21 05:26:35  4.87m
bob/bob.db.voxforge@master (87/120               2019-10-21 05:31:27  1.42m
bob/bob.rppg.base@master (88/120                 2019-10-21 05:32:52  1.57m
bob/bob.pad.base@master (89/120                  2019-10-21 05:34:26  2.92m
bob/bob.pad.face@master (90/120                  2019-10-21 05:37:21  6.42m
bob/bob.pad.voice@master (91/120                 2019-10-21 05:43:46  1.73m
bob/bob.pad.vein@master (92/120                  2019-10-21 05:45:30  1.55m
bob/bob.fusion.base@master (93/120               2019-10-21 05:47:03  1.67m
bob/bob.db.oulunpu@master (94/120                2019-10-21 05:48:43  2.13m
bob/bob.db.uvad@master (95/120                   2019-10-21 05:50:51  1.78m
bob/bob.db.swan@master (96/120                   2019-10-21 05:52:39  2.33m
bob/bob.db.cuhk_cufs@master (97/120              2019-10-21 05:54:59  2.05m
bob/bob.db.cbsr_nir_vis_2@master (98/120         2019-10-21 05:57:02  1.93m
bob/bob.db.nivl@master (99/120                   2019-10-21 05:58:58  4.2m
bob/bob.db.pola_thermal@master (100/120          2019-10-21 06:03:10  2.85m
bob/bob.db.cuhk_cufsf@master (101/120            2019-10-21 06:06:01  1.4m
bob/bob.db.ldhf@master (102/120                  2019-10-21 06:07:25  1.43m
bob/bob.ip.tensorflow_extractor@master (103/120  2019-10-21 06:08:51  3.48m
bob/bob.ip.caffe_extractor@master (104/120       2019-10-21 06:12:20  2.13m
bob/bob.bio.caffe_face@master (105/120           2019-10-21 06:14:28  2.2m
bob/bob.learn.tensorflow@master (106/120         2019-10-21 06:16:40  134.88m
bob/bob.learn.pytorch@master (107/120            2019-10-21 08:31:33  10.98m
bob/bob.bio.face_ongoing@master (108/120         2019-10-21 08:42:32  2.27m
bob/bob.bio.htface@master (109/120               2019-10-21 08:44:48  105.23m
bob/bob.db.drive@master (110/120                 2019-10-21 10:30:02  1.02m
bob/bob.db.stare@master (111/120                 2019-10-21 10:31:03  0.83m
bob/bob.db.chasedb1@master (112/120              2019-10-21 10:31:53  0.82m
bob/bob.db.iostar@master (113/120                2019-10-21 10:32:42  0.83m
bob/bob.db.hrf@master (114/120                   2019-10-21 10:33:32  0.83m
bob/bob.db.rimoner3@master (115/120              2019-10-21 10:34:22  0.83m
bob/bob.db.drionsdb@master (116/120              2019-10-21 10:35:12  0.83m
bob/bob.db.refuge@master (117/120                2019-10-21 10:36:02  0.87m
bob/bob.db.drishtigs1@master (118/120            2019-10-21 10:36:55  0.82m
bob/bob.ip.binseg@master (119/120                2019-10-21 10:37:44  1.42m
bob/bob@master (120/120                          2019-10-21 10:39:09  284.9m
-----------------------------------------------  -------------------  -------

Merge request reports