service-utils

service-utils (repo) is a NodeJS utilities library built since 2024 to encourage standards for NodeJS services at the Wikimedia Foundation, replacing service-runner, service-template-node, servicelib-node, and service-scaffold-node. It has a similar API to service-runner for easy migration.

It was built by User:TChin (WMF) under the Data Engineering Team to support the Event Platform, and is now co-owned by SRE Service Operations. Production services have been using it at Wikimedia since late 2024.

If you are building a new NodeJS service to run at WMF, you should use service-utils from day one. If you already own a service powered by service-runner, you should consider migrating to service-utils at your earliest convenience.

Background/History

service-runner is a library built in 2015 to provide a common abstraction for all NodeJS services at the Wikimedia Foundation. At that time, all services were on bare-metal, and different teams used different libraries for logging and metrics. service-runner provided a way to build multiple services inside docker containers and spin them up in a worker cluster setup with restarts, rate limiting, etc. On top of that was service-template-node, which acted as a standard framework for all NodeJS services to derive from, providing OpenAPI support, distributed tracing, standard metrics, and some common helper functions.

As time went on, a few things happened.

1. The abilities of service-runner were superseded by Kubernetes. We do not need to roll our own cluster anymore, and all services that use service-runner now run with 0 workers.
2. service-runner has no maintainer, and uses libraries (and forks of libraries) deprecated 7+ years ago.
3. service-runner supported statsd and Prometheus for metrics reporting. We selected on Prometheus.
4. service-runner used bunyan and gelf-stream to export logs into logstash. We selected on a common ECS logging format, which bunyan doesn't support, and logs into stdout in Kubernetes are automatically exported into logstash now.
5. service-template-node became a burden, as any update to the template required manual updates to every service that used the template. We don't have clear insight into how many services use the template, and the structure of services across teams very quickly fell out of lockstep.
   1. An attempt was made in 2021 to fix this issue via the project servicelib-node, which abstracted all the logic into different libraries that are then used in a lightweight scaffolding called service-scaffold-node. This would allow easy updating of the logic without touching the scaffold. However, team restructuring and deprioritization killed the initiative.

Because of all of these issues, when teams decide to build new NodeJS services, they had to decide between using the standard but deprecated and ageing tooling, or put in the work to implement things themselves.

Concerns about the age of service-runner were raised every time services had to be upgraded to the latest NodeJS LTS version, and in 2024 to support the Event Platform's NodeJS services, service-utils was created.

Philosophy

The main goal of service-utils is to get services off of service-runner and by proxy service-template-node. It does not address the issue that service-template-node tried to solve of having a common, unified microservice structure. service-utils provides the tools, but leaves it up to the service implementor to use the tools correctly. For example, it provides helpers for distributed tracing, but does not automatically set it up for you. It supports ECS logging, but you have to make sure that your logs are formatted correctly.

In the future, a service-scaffold-node v2 could be built on top of it. The only thing service-utils doesn't (currently) do is provide an API wrapper for MediaWiki. However, since we don't create a lot of brand new NodeJS services anyways, it might be more effort than its worth.

Basic Setup

Also look at the README for additional documentation that might not be on this page

1. Install service-utils from Wikimedia's Data Engineering GitLab Package Registry:

   echo @wikimedia:registry=https://gitlab.wikimedia.org/api/v4/groups/189/-/packages/npm/ >> .npmrc
   npm i @wikimedia/service-utils
   

2. Create a config file service-utils.config.yaml. All keys are technically optional but at the bare minimum you should have a service_name:

   # service-utils.config.yaml
   service_name: foobar
   

3. Import service-utils in your codebase. service-utils exports a singleton through a getInstance() function. If you want to recreate the singleton (for example, with new config), you would need to call teardown() first.

   import { getInstance, teardown } from "@wikimedia/service-utils";
   const serviceUtils = await getInstance();
   
   // Somewhere else in the service
   serviceUtils.logger
     .log("info", "Test simple logger");
   
   // On service shutdown
   await teardown();
   

4. Run the NodeJS service normally. service-utils automatically looks for the config file in ${cwd}/service-utils.config

Configuration

Default Structure

If you have literally, absolutely nothing--a completely blank config file, this is the default config:

service_name: default_service

logging:
  level: info
  format: ecs
  stacktrace: true
  transports:
    - transport: Console # Case matters

metrics: false

Always be explicit with your config. The default config is inferred through various default values set throughout the codebase and can (and will) change at any time.

Complete Example

service-utils uses c12 to load and merge configuration.

# service-utils.config.yaml
service_name: foobar
# Default config
logging:
  level: info
  format: ecs
  stacktrace: true
  transports:
    - transport: Console # Case matters

# Environment-specific variables, merged with defaults
# These can be arbitrarily named and applied on ServiceUtils instantiation with
# the `envName` option like: loadConfig({ envName: 'development' })
$development:
  service_name: foobar dev
  logging:
    level: verbose
    format: simple

$production:
  service_name: foobar prod
  logging:
    level: info
  metrics:
    port: 9001

Custom Config

You can add any key to service-utils.config.yaml and it will be available within the singleton's config property. You can type this custom config using the generic when getting the instance.

Given:

# service-utils.config.yaml
service_name: some_service

foo: bar

In Typescript, you can type ServiceUtils like:

import { getInstance } from "@wikimedia/service-utils";

interface CustomConfig {
  foo: string;
}

const serviceUtils = await getInstance<CustomConfig>();
// This will be typed
console.log(serviceUtils.config.foo);

Migrating from service-runner

service-utils is incrementally adoptable, and is easily migratable from service-runner. It is compatible with service-runner's config structure, and also maintains the -c cli args.

Switching Logging from Bunyan to Winston

There are a few things that need to change code-wise:

• If you use Bunyan's createLogger to make loggers outside of service-* framework, you now need to change it to work with Winston.
• Winston does not have fatal or trace log levels. Move to error and verbose or debug.
• Winston takes the log message string first and then an object of metadata.

- bunyan.createLogger( { name: 'EventGate', src: true, level: 'info' } );
# Note that `name` != `service.name`, which is used in ECS format
+ winston.createLogger( { level: 'info', defaultMeta: { name: 'EventGate' } } );

- logger.trace({ event }, `Validating ${this.eventRepr(event)}...`);
+ logger.debug(`Validating ${this.eventRepr(event)}...`, { event });

Some things need to change structurally to support ECS logging. In particular, for those using service-template-node, the reqForLog method outputs a completely incorrect format. service-utils does naively adapt it into the correct format, but it's better to explicitly do it yourself. Look through your service for other areas that need to conform to the ECS common logging schema. If the logs don't fit the schema, you have to modify the schema. Wikimedia's version of ECS does not allow for arbitrary keys.

function reqForLog(req, whitelistRE) {

-	const ret = {
-		url: req.originalUrl,
-		headers: {},
-		method: req.method,
-		params: req.params,
-		query: req.query,
-		body: req.body,
-		remoteAddress: req.connection.remoteAddress,
-		remotePort: req.connection.remotePort
-	};

+	const ret = {
+		source: {
+			ip: req.connection.remoteAddress,
+			port: req.connection.remotePort
+		},
+		url: {
+			full: req.originalUrl,
+			// service-template-node dynamically defines routes
+			// which are captured with params[n]
+			// https://expressjs.com/en/4x/api.html#req.params
+			path: req?.params?.[ 0 ] ?? req?.path ?? '',
+			query: req.query
+		},
+		http: {
+			request: {
+				method: req.method,
+				headers: {},
+				body: {
+					content: req.body
+				},
+				params: req.params
+			}
+		}
+	};

	if (req.headers && whitelistRE) {
		Object.keys(req.headers).forEach((hdr) => {
			if (whitelistRE.test(hdr)) {
-				ret.headers[hdr] = req.headers[hdr];
+				ret.http.request.headers[hdr] = req.headers[hdr];
			}
		});
	}

	return ret;

}

The most important part is probably any x-request-id should go into http.request.id and the name of the service should go into service.name.

Migrating Metrics

TODO

Existing users

• Data Platform Engineering
  • eventgate
  • eventstreams
  • dataset-config-store-service
• Abstract Wikipedia team
  • function-orchestrator
  • function-evaluator

Links

• Service-runner
  • GitHub
• Service-template-node
  • GitHub
• Service-scaffold-node
• Service-utils GitLab
• Service-utils Phabricator Project
• Dasboard for production services using service-utils