[ci] Remove sphinx directory (from other builds?)

3 jobs for conda-test in 58 minutes and 28 seconds (queued for 4 seconds)