Moving documentation building and deployment to readthedocs.org

This would also alleviate problems and dependence to "https://www.idiap.ch/software/bob/docs"

Milestone changed to %Refactoring 2016 and gitlab migration milestone

removed milestone

changed milestone to %May 2017 Hackathon

This is something to look into during the hackathon but not necessarily move to readthedocs.org but we definitely need to move out of pythonhosted.org @jdiefenbaugh might be interested in this.

Yup, I'll look into solutions to this (moving away from pythonhosted) during the hackathon. It'll probably take longer to completely implement than just two days, however.

assigned to @jdiefenbaugh

How docs.python.org does it: http://blog.python.org/2012/10/updates-to-docspythonorg.html

One interesting sphinx add-on https://robpol86.github.io/sphinxcontrib-versioning/v2.2.1/index.html

https://stackoverflow.com/questions/20939598/enabling-sidebar-on-python-sphinx-documentation-based-on-a-sphinx-bootstrap-them?rq=1

https://stackoverflow.com/questions/16072086/including-dynamic-content-in-documentation-generated-by-python-sphinx?rq=1

mentioned in issue bob#244 (closed)

Here's a little writeup of what I found during the hackathon. After a quick overview of the problems with the current documentation system, there's a "Possible Solutions" section listing the solutions I investigated for certain problems that had multiple feasible solutions. I wrote a possible plan of action at the end.

Problems

pythonhosted.org documentation-hosting deprecation
Lack of support for building~viewing documentation for non-master branches
No scheme for saving~viewing old versions of documentation

The documentation build system does not save old versions
The documentation pages do not have functionality for navigating to old/new versions

Lack of cross-package search support in webpages

Possible Solutions & Problems

`pythonhosted.org` deprecation

Moving to ReadTheDocs

Needs to build documentation on ReadTheDocs yet has ~~15min timeout for each entity~~project~repo
Lack of configuration support (versioning schemas, etc.) may either rule out ReadTheDocs entirely, or just put alot of work on us to conform to their standards

Moving to Gitlab Pages (Nhttps://about.gitlab.com/2016/12/24/were-bringing-gitlab-pages-to-community-edition/OTE: Pages support for Gitlab CE was added in Gitlab 8.16)

Same configuration issues as above
Only 1 Pages instance per repo could mean a complex build system to take into account other versions~branches

Hosting a private instance of ReadTheDocs

Lack of official support, barely documented

Moving fully to Idiap hosting (https://www.idiap.ch/software/bob/docs)

Further reliance on Idiap infrastructure
Serves only from Switzerland (I think?)

Versioning support in documentation webpages

Moving to ReadTheDocs (either through their hosting or our hosting)

See last section for issues with this approach

sphinx-versioning Sphinx extension

Needs to build every version of documentation every time the version list should be updated

Forking rtd_sphinx_theme to change versions.html functionality

Needs a fork maintainer

Custom sphinx extension using a custom HTML sidebar template

Will need a developer~maintainer

Custom Javascript injection script

Hack-ish

Better search functionality

Place/build docs in one folder, so sphinx search takes into account all documentation.

[generally not a good idea]

Sphinx extension (either via a Search Adapter or a sidebar template)

Would very most likely not be as performant as option 3, and would need a maintainer

Use a custom google search

Effectively outsourcing search
No offline support

Possible Plan of Action

Move away from pythonhosted.org to only use the Idiap apache server.

Change the resolution of the documentation server in the ci script.
Destroy the pythonhosted docs.

Update the CI & apache server currently serving docs to support multiple branches/versions. There will be slightly different functionality between the public documentation releases (stable versions of master branch) vs. private~internal releases (latest, stable, and tags for every currently-existing branch).

Have the docs job run for any branch, not just master (at least, changing the ci template).
Change the scheme for naming tarballs to be sent to the idiap apache server's incoming folder from ${CI_PROJECT_NAME}-${CI_BUILD_REF}.tar.bz2 (repo name & commit) to ${CI_PROJECT_NAME}-${CI_BUILD_REF_NAME}-${CI_BUILD_TAG}-${CI_BUILD_REF}.tar.bz2 (repo name, branch, tag, commit). The delimiter will probably need to be changed as well, as - is commonly found in branch names.
Update the apache server script that processes the above tarballs (from incoming/) to parse the new tarball name format and extract to repo/version/, not repo/.
Add latest/master & stable symlinks for each package pointing to the corresponding versions, and update the apache server script to keep these links up-to-date.
Update the apache server config to serve the documentation with the version in the URL (possibly an unnecessary change if the server constructs URLs via file paths).

Add the versioning toolbar to the existing documentation webpages.

Modify the apache server to maintain a list of versions for each package.
Write a small JS script to inject a version-selection toolbar based on the above list of versions.
Serve this JS script in the documentation webpages (either via server settings, adding during sphinx build, or server script).

Either change the web search to be, or add, a custom google search with a site specification (site:https://www.idiap.ch/software/bob/docs/).

Notes & Reasoning

Assuming I'd implement these changes - I don't have any experiment with working with sphinx build processes or extensions. Although the JS injection script is hack-ish, it would be far faster than me learning all that's needed to work with the sphinx build process & extensions.
We already have the Idiap server set up, so switching to it for all our docs would be the easiest/fastest way to get off of pythonhosted.org.
I couldn't find much info about the apache server the docs are served from, nor info about the scripts that handle incoming/, so references to changes to the script are unfortunately vague.

What do you guys think?

Update the apache server script that processes the above tarballs (from incoming/) to parse the new tarball name format and extract to repo/version/, not repo/.

This magical script used to do exactly that sometime in 2015.

The plan overall looks good (except the javascript hack) but I wonder what would the solution would look like using a sphinx extension.

Did you research how https://docs.python.org (http://blog.python.org/2012/10/updates-to-docspythonorg.html, https://bugs.python.org/issue8040) or how https://docs.djangoproject.com (https://github.com/django/djangoproject.com) or http://docs.sqlalchemy.org/en/latest/ does it? Can't we use one of them?

Have you seen this? https://github.com/TraceSoftwareInternational/HostMyDocs looks really cool and an easy solution to our problems.

Have you seen this? https://github.com/TraceSoftwareInternational/HostMyDocs looks really cool and an easy solution to our problems.

Personally I wouldn't feel comfortable betting on this:

This project was started last December
It had its last commit a couple months ago
It has obviously not attracted many people...it seems to almost be a dead project by now. I'd be afraid that we (I) would end up maintaining it (or a fork of it).
Most of the tech it uses are things we don't use in the biometrics group (Angular 2, Typescript, PHP, Docker).
It assumes that 1) we have root, 2) we have access to docker, and 3) that we have systemd (for a "production" build), which all may or may not be the case.
Just when @amohammadi was showing me, we found show-stopper bugs in the platform.
It doesn't meld with Sphinx/sphinx_rtd_theme, but just adds another layer on top of it.

I don't feel that it gives us alot when I consider all the above points. It solves our problem via adding an extra (!) sidebar that lets you choose a version based on the existing folders. Not only is adding an additional sidebar a lazy way to go, but I feel that adopting this project (backend, frontend, etc.) for just a versioning sidebar is too much, when a quick JS script (maybe 100ishLOC?) will do the same.

Did you research how https://docs.python.org (http://blog.python.org/2012/10/updates-to-docspythonorg.html, https://bugs.python.org/issue8040) or how https://docs.djangoproject.com (https://github.com/django/djangoproject.com) or http://docs.sqlalchemy.org/en/latest/ does it? Can't we use one of them?

The python docs change the sphinx theme and add a JS file.
Django uses Django~Jinja templating (custom website, not Sphinx).
SQLAlchemy has a custom website (see the homepage footer), but does use Sphinx as well...I'm assuming it's a custom sidebar. Clicking on a different version redirects you the the homepage of that version - it doesn't just change the version of the page.

As I said to @jdiefenbaugh too. I don't know how it is implemented internally but it tackles a couple of issues we are having.

The benefits:

Accepts docs in zip format. No need for an incoming folder and post processing.
Understands semantic versioning and highlights the latest version of each project.
Has an API to upload new documentation
Can show the docs without the sidebar which removes the clutter if required.
- Even if it stops working we can keep serving our docs with webdav if we don't expose the sidebar to the users.
It was less than a minute to setup using docker for me while I had no knowledge of it.

There are some shortcomings too:

There is no API to delete or replace docs. https://github.com/TraceSoftwareInternational/HostMyDocs/issues/25
The sidebar is hard to navigate: https://github.com/TraceSoftwareInternational/HostMyDocs/issues/24
Offers no central search option but at least lists all projects which I think even that is great. And we have no solution for a global search as of now.

Accepts docs in zip format. No need for an incoming folder and post processing.

We don't need an incoming folder, either, unless we wanted post-processing. We could just scp it from the CI to the directory.

Understands semantic versioning and highlights the latest version of each project.

Couldn't "understanding" semantic versioning just be comparing the version strings and sorting in order? v3.0.5 > v3.0.4 > v2.9.1 > v2.9.0 > v2.9.0b2 > v2.9.0b1 > v2.9.0a1 > v1 > v0.0.1.

Can show the docs without the sidebar which removes the clutter if required.

There shouldn't be any clutter - it should just integrate with the sphinx_rtd_theme.

Even if it stops working we can keep serving our docs with webdav if we don't expose the sidebar to the users.

This is an option no matter what we implement, right? It's not something that this project is giving to us.

It was less than a minute to setup using docker for me while I had no knowledge of it.

I guess this is good of them to have enough docs to get you up-and-running quickly, but I'm not sure if this is a plus for us compared to not having to use docker at all.

The Plan^TM Phase 1

Remove the apache cron script (that manages incoming/) and remove incoming/
Change the docs job to run on any branch (this will be managed at a higher level)
Use curl to send the built docs folder to the appropriate directories (see below), and to not send to pythonhosted.org

If the folder exists, make sure to delete the contents of the folder before copying the new files!
send to private for private packages, send to public for public packages

Update bob.extension to resolve documentation links using the Idiap server URL schema, not pythonhosted.org

New folder scheme

Currently, the relevant scheme is:

public
  docs
    stable
      bob.extension
    master
      bob.extension

It will become:

public
  docs
    bob (group name)
      bob.extension (reporsitory name)
        stable
        master (branch name)
        v2.1.0 (tag name)

Which folder(s) `curl` sends the built docs to, depending on the commit

Commit Type X Location	`<tag>/`	`master/`	`stable/`
Non-tagged commit		x
Tagged commit	x	x	x

Just an FYI. Since the last time I used hostmydocs and opened some issues on their repository for improvement, they have released a new version that addresses all bugs that I faced and implements all the features that I asked for: https://github.com/TraceSoftwareInternational/HostMyDocs/releases/tag/1.3.0

These are good news! @jdiefenbaugh could you check that again. They seem to be active on it.

Glad to see them being so responsive! I'll take a look after I'm done with the beat.web MR.

While it's great that they fixed some things, and they seem to be supporting this software for now, I still don't see the need to move to this system. We already made a plan that's much simpler and more flexible but still provides the same feature as HostMyDocs - versioning support.

Several of my issues with HostMyDocs are still valid. They seem focused on providing a language- and format- angostic documentation server, while we only use 1 language/format. We shouldn't have to support an entire system - a PHP backend, systemd service, docker container, and an Angular2 frontend - just to do what we do now. We shouldn't have to deal with a messy API to send docs. I'm totally open for people to try to sell HostMyDocs to me again, but I still think it's a poor product for us. And frankly, it seems less of a community solution to documentation hosting, and more of a company's very specific solution to their very specific problem that they wanted to host publicly.

That said, @amohammadi was talking about asking the devs to add global search support via ElasticSearch. That would at least be a feature that we have no immediate solution for, which would be nice.

@andre.anjos what do you think?

I agree we should drop this idea and concentrate on having the right folder structure only. The rest can be inferred through the web directory listing.

Ok - decisions are made. How to move forward with this?

mentioned in commit e9e01f42

mentioned in merge request !58 (merged)

New strategy is now in place and seems to be working fine.

There is one potential problem though: in case a new package is created, the DAV MKDIR command will fail since the directories preceeding the top-level doc directory will not exist. We have to implement it in a way it works recursively, from the root of the DAV upload (i.e. "private-upload" or "public-upload").

mentioned in merge request !59 (merged)

mentioned in merge request bob.extension!46 (merged)

mentioned in merge request bob.extension!47 (merged)

bob.extension is now on version v2.4.2 and accomodates all required changes to make this work properly. Together with changes committed to bob.admin, the only remaining modification is to go through all README.rst files and modify public/private documentation links to match the one in the current admin template.