Ok I have configured some working CI here which uses small conda environments, caches things, and uses conda packages for our test process and upload process.
I have a couple of issues still:
upload of packages
Don't know how to upload the files like you did with wheels.
After a file is uploaded you need to run conda index to reflect the new changes.