Topic on Talk:Platform Engineering Team/Event Platform Value Stream/Use case: Event Platform SDLC practices

Deployments should be automated and predicated upon code review.

vs.

We lack well defined development, staging and production environments.

This makes it sound like you'll need to deploy not just to standard wiki-production, but also to other environments, is that right?

What does manual deployment to those other environments look like at the moment?

For services destined for wiki-production we tend to use helm deploy—is that still what you're targeting? Or is something more complex needed?

Also, if you're deploying into other environments, are you still deploying artifacts from the Wikimedia docker registry?

This makes it sound like you'll need to deploy not just to standard wiki-production, but also to other environments, is that right?

That's correct. Right now we need to target systems other than wiki-production, for example Hadoop. You can find an example of manual deployment here: https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-stream-enrichment#deploy-on-yarn. I'm aware that other teams have bespoke deployment jobs that target that system (see reference to Airflow in the page), but to the best of my knowledge there is no off-the-shelf reusable solution.

The reason we target Hadoop is because of a mix of development and integration testing needs, that require access to data stored in Kafka and HDFS. We are not married to this platform though, and would be open to follow guidelines on alternative systems. Actually, having disjoint platforms (k8s vs yarn) for production and devel/test is a pain point we identified.

For services destined for wiki-production we tend to use helm deploy—is that still what you're targeting? Or is something more complex needed? Also, if you're deploying into other environments, are you still deploying artifacts from the Wikimedia docker registry?

For services destined to wiki-production we'll follow deployment-charts guidelines. We will be partnering with Search and SRE to streamline how we deploy shared stacks (e.g. apache flink). We are relying on images from the WIkimedia docker registry, and on a mix of Gitlab and Archiva for publishing jars and wheels.

Re environments: I'd like to think of dev / test / staging / prod in terms on needs and let you and SRE guide us on which resource pool better fits our needs (or help us build one).

Two pages that go a bit more in detail in the flink / hadoop side of things are:

Platform Engineering Team/Event Platform Value Stream/Use case: compute needs for streaming pipelines

Platform Engineering Team/Event Platform Value Stream/Assess what is required for the enrichment pipeline to run on k8s