Extend Test Infrastructure to run multiple E2E tests in parallel

[jlewi]

We need to extend the E2E test infrastructure to run multiple tests in parallel. The diagram below illustrates the expected flow

The sequence of events is:

• An event (e.g. GitHub pull request) triggers a prow job
• Prow job builds the artifacts (Docker images, helm packages, etc...)
• Create one or more K8s clusters
  * We potentially need multiple clusters to test different features (e.g. GPUs) or different versions of
  K8s
• Deploy the operator
• Run a bunch of tests
  * Each test will exercise different TfJob features; e.g. non-distributed vs. distributed)
• Cleanup

So the sequence of events is well represented by the DAG as illustrated by the diagram. I'd like to find a convenient way to express and execute the DAG; preferably without rolling our own workflow system.

Option 1 One Prow Job Per Test
We could create one prow job for each test and rely on prow to run the tests in parallel.

Disadvantages

• Integrating new jobs into prow is tedious; lots of config files need to be updated
• Can't reuse resources (build artifacts, clusters, etc...) across tests
• Seems like a prow anti pattern
  • Most prow jobs run a suite of tests

Option 2 Use Ginkgo

Advantages

• Fairly lightweight
• Used by K8s for testing

Ginkgo is a go framework. I'm increasingly leaning towards using Python not go for our CI/CT. Since the TensorFlow test programs will be written in Python, I think Python is a good choice for CT.

Option 3 Airflow

We can express the DAG illustrated above as an Airflow pipeline. The prow job can then just trigger the pipeline and upload artifacts to GCS as needed.

Advantages

• Don't need to learn a new tool for CT
  • We can treat our CT pipeline just like an ML pipeline (preprocessing, training, evaluation, etc...)
  • I'm assuming that Airflow will become one of the standard workflow tools in ML
• Python
• Integration with K8s is coming
  * This should make it easy to deploy/manage Airflow on K8s
  * Should make it easy to express the steps in the DAG as K8s Jobs and orchestrate them using Airflow

Disadvantages

• Heavyweight

[bhack]

For option 3 do you know the status of https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=71013666?

[bhack]

I know that there will be a meetup quite soon.

[jlewi]

@foxish could provide more information about status of Airflow on K8s.

[foxish]

We'll present a demo at kubernetes sig-apps soon as well. The branch everyone is committing to currently is https://github.com/bloomberg/airflow/tree/airflow-kubernetes-executor. Some discussion remains around credential management and storage, but mostly, the executor is in pretty good shape. I think the expectation is for the basic first PR upstreamed in the next month or so.

cc/ @dimberman

[bhack]

Nice I was following that branch some times ago

[bhack]

I think that option 3 is more on the edge but probably the most future-proof.

[jlewi]

CNCF has a project to do cross-cloud integration for its projects. This slide seems to indicate its based on GitLab.

[jlewi]

I took a look at GitLab CI/CD and I think I prefer the K8s + Airflow approach.

• It looks like GitLab has its own workflow system based on a simple declarative syntax (YAML)
• GitLab has its own abstractions for runners, processes, and containers.

Its not obvious to me that the cognitive load of using a different set of tools for CI/CD (compared to datascience/ml pipelines) is worth the slicker UI and better out of box support for some common CI/CD tasks.

I like the approach prow has taken of building a set of microservices and would rather continue in that direction rather than adopt a monolithic tool. Towards that end, I'd rather evolve a set of reusable scripts for common CI/CD tasks that can easily be reused with Airflow + K8s.

[jimexist]

had some internal efforts trying out option 3 except for that we went with luigi which was more mature at the time than Airflow (currently incubating). I would ditto the claim that airflow is more future-proof.