Jump to content

XTools/Development

From mediawiki.org

Thank you for contributing to XTools! This guide gives a high-level walkthrough of everything you need to contribute to XTools in a local environment.

Prerequisites

[edit]

Optional

Running a development server

[edit]
  • Clone the repository: git clone https://github.com/x-tools/xtools.git && cd xtools
  • Install PHP dependencies: composer install
  • Run cp .env .env.local and fill in the missing values, using /Configuration as a guide.
    • If developing against the Wikimedia Toolforge replicas (recommended), set up your SSH tunnel and configure accordingly.
  • If you need to work on JS/CSS files:
    • npm install to install Node dependencies
    • Assets are generated with Symfony Encore. If you're working on JavaScript, CSS, or adding images, compile the assets with: npm run watch.
    • Before submitting your pull request, npm run build to build the production assets.[1]
  • Launch Symfony's built-in server: symfony serve

Assuming the configuration was done correctly and as desired, you should be up and running at http://localhost:8000

The Simple Counter is the simplest tool and should work as soon as you set up XTools. Test it by going to http://localhost:8000/sc and put in Jimbo Wales as the Username and en.wikipedia.org as the wiki. After submitting you should quickly get results.

The development server does not cache application data; any changes you make are visible after refreshing the page. However when you modify things in .env.local or otherwise aren't seeing changes you're making, you may need to clear the cache with symfony console cache:clear (or the shorter variant symfony console c:c).

The logs are in var/logs/dev.log. If things are acting up unexpectedly, you might try clearing the cache or restarting the server.

Developing against Toolforge replicas

[edit]

First make sure you have a Wikimedia developer account and access to Toolforge. Then you can set up the necessary tunnels and a shell session with:

ssh -L 4711:s1.web.db.svc.wikimedia.cloud:3306 -L 4712:s2.web.db.svc.wikimedia.cloud:3306 -L 4713:s3.web.db.svc.wikimedia.cloud:3306 -L 4714:s4.web.db.svc.wikimedia.cloud:3306 -L 4715:s5.web.db.svc.wikimedia.cloud:3306 -L 4716:s6.web.db.svc.wikimedia.cloud:3306 -L 4717:s7.web.db.svc.wikimedia.cloud:3306 -L 4718:s8.web.db.svc.wikimedia.cloud:3306 -L 4720:tools.db.svc.eqiad.wmflabs:3306 username@login.toolforge.org

Replace username with your Toolforge Unix shell username. Note our tunnel has to connect to each database slice. The ports used here should match up with the corresponding options in your .env.local, like so:

DATABASE_REPLICA_USER=<your-username>
DATABASE_REPLICA_PASSWORD=<your-password>
DATABASE_REPLICA_HOST_S1=127.0.0.1
DATABASE_REPLICA_PORT_S1=4711
DATABASE_REPLICA_HOST_S2=127.0.0.1
DATABASE_REPLICA_PORT_S2=4712
DATABASE_REPLICA_HOST_S3=127.0.0.1
DATABASE_REPLICA_PORT_S3=4713
DATABASE_REPLICA_HOST_S4=127.0.0.1
DATABASE_REPLICA_PORT_S4=4714
DATABASE_REPLICA_HOST_S5=127.0.0.1
DATABASE_REPLICA_PORT_S5=4715
DATABASE_REPLICA_HOST_S6=127.0.0.1
DATABASE_REPLICA_PORT_S6=4716
DATABASE_REPLICA_HOST_S7=127.0.0.1
DATABASE_REPLICA_PORT_S7=4717
DATABASE_REPLICA_HOST_S8=127.0.0.1
DATABASE_REPLICA_PORT_S8=4718
DATABASE_TOOLSDB_HOST=127.0.0.1
DATABASE_TOOLSDB_PORT=4720

Change the your-* bits to your own values, which you can find in your replica.my.cnf file in the home directory of your account on Toolforge.

OAuth

[edit]

If you need to test something that requires login, you need create an OAuth consumer for your localhost:

  1. Go to meta:Special:OAuthConsumerRegistration and select "Request a token for a new OAuth 1.0a consumer."
  2. Fill out the fields like so:
    1. Application name – something clearly indicating it's not for production, i.e. XTools (localhost)
    2. Consumer version – use 1.0, and bump this whenever you need to create a new consumer
    3. Application description – Something like "Testing for local environment"
    4. Select "This consumer is for use only by [my username]"
    5. Applicable project – *
    6. Types of grants being requested – assuming you want to make authentication read requests to the wikis, select "Request authorization for specific permissions", and don't set any additional grants other than the preselected "Basic rights".
    7. Allowed IP ranges – specify your IP if you want, or leave as-is.
    8. Allowed pages for editing – leave blank. The consumer should not be making edits.
    9. Public RSA key (optional) – leave blank.
    10. Check the confirmation checkbox and submit
  3. You will be given a consumer secret and a consumer key, which should be used as the OAUTH_KEY and OAUTH_SECRET in your .env.local file.

Note however you can simply supply a username for APP_LOGGED_IN_USER in your .env.local file and the "Login" will work without OAuth.

Caching

[edit]

XTools should probably take advantage of Doctrine's built-in caching mechanism, but it doesn't. Instead it is done in a somewhat manual fashion. This is only done in Repository classes, using this pattern:

public function doSomething($input) {
    $cacheKey = $this->getCacheKey( func_get_args(), 'component_method' );
    if ( $this->cache->hasItem( $cacheKey ) ) {
        return $this->cache->getItem( $cacheKey )->get();
    }
    $results = 'results of big query';
    return $this->setCache( $cacheKey, $results );
}

The cache key can be anything, so long as it is unique to the specific method. A third parameter can be passed to setCache to set the TTL, using the same syntax from the DateInterval class (e.g. P1D is one day, PT1H is one hour).

The above methods are just wrappers around a PSR-6 implementation, intended to reduce the repetition of similar lines of code.

Style guidelines

[edit]

Tests

[edit]

Tests are located in the tests/ directory, and match the src/ directory structure. They are built with PHPUnit. Repositories only handle fetching data and do not need to be tested. Add Controllers also interact with the database, and while tests are most welcomed for these, they will not run on the CI server (GitHub Actions) due to limitations. Instead, put all things that query the replicas behind a app.is_wmf check (example).

There are also tests for linting, file permissions, among other things.

Use composer test to run the full suite, or ./vendor/bin/phpunit tests/ to run just the unit tests.

Architecture

[edit]

Every "tool" that's part of XTools consist of a Controller (routing), Model (business logic), and a Repository (database queries). Here is a crude rundown of how the repo is organized:

  • assets/ – contains all CSS, JS, images and vendor scripts. The vendor scripts eventually should be moved to be NPM-based (T392531)
  • bin/ – the Symfony CLI is recommended, but if you don't use it, you can access the Symfony console from here i.e. ./bin/console cache:clear
  • config/ – where all static configuration lives. Usually you'll only need to be concerned with files in the root path, such as admin_stats.yaml (configuration for the XTools/Admin Stats tool and related tools) and assessments.yaml (configuration for xtools:api/project/assessments).
  • i18n/ – localization files
  • migrations/ – these manage the schema of the XTools internal usage tracking database. If changes are ever needed, they should only be made via new migration files. See the DoctrineMigrationsBundle docs for more information.
  • public/ – publicly viewable files including the entrypoint index.php. These files should never need to be changed directly unless you're upgrading Symfony. The public/build/ files get automatically updated by Symfony Encore.
  • src/ – the PHP code
    • Controller/ – Responsible for defining routes and API documentation, instantiating a Model and rendering the templates. Note that controllers have auto-wiring. To inject a dependency, just include a type-hint in the method signature.
    • EventSubscriber/ – houses the Symfony event listeners.
    • Exception/ – our custom exception classes. XtoolsHttpException can be used to redirect users, for example see XtoolsController::checkRestrictedApiEndpoint()
    • Helper/ – standalone services providing shared logic across multiple tools (dependencies are defined in config/services.yaml)
    • Model/ – All files here should extend the Model class. They contain the main logic for each tool, and have a corresponding Repository class that it calls to fetch data from the database or API. Public methods in models are exposed to the Twig templates, and this is the flow of execution for almost all requests apart from the XTools API. Note when searching for uses of methods, don't include the get portion. So for example, the AutoEdits Twig template calls AutoEdits::getEditCount() method with ac.editCount (where ac is a reference to the Model, passed to the template by the Controller).
    • Monolog/ – custom formatting for logging such as the error emails. Further configuration is made in config/{env}/monolog.yaml.
    • Repository/ – where all database queries and API requests should be made, using $this->executeProjectsQuery(), $this->executeQueryBuilder(), or $this->executeApiRequest() from the parent Repository class. Results should also be cached accordingly.
    • Twig/ – Custom Twig extensions, which make functionality available to Twig templates.
  • templates/ – where the Twig templates live. Templates are rendered by the Controller, which pass in a reference to the appropriate Model for that tool. The model methods then call the Repository methods, do any post-processing, and finally return the results for display in the template. Twig templates can be powerful and contain lots of logic, but they are hard to test. For this reason, it is best practice to keep as much logic as possible in the Model classes and out of the templates.
    • base.html.twig – the main layout template from which all other templates extend.
    • marcros/ – these are the Twig-implemented "functions" so to speak.
      • forms.html.twig – centralizes the input fields used on index.html.twig (the initial form) for each tool.
      • layout.html.twig – centralizes the HTML structure of result.html.twig pages (result pages).
      • pieChart.html.twig – egregious Twig-to-JS hackery to make pie charts with Chart.js
      • wiki.html.twig – functions for linking to the wiki in a consistent way.
  • tests/ – all the PHPUnit tests. Every Controller, Model or service (such as the Helper namespace) should have a corresponding test file.
  • var/ – the only thing of interest here are the logs in var/log/. Nothing else should be changed unless required for a Symfony upgrade.
  • vendor/ – git-ignored PHP dependencies

Releases

[edit]

Releases are made by tagging commits in the master branch. Before tagging a new release:

  • Update the APP_VERSION in the .env file (follow the Semantic Versioning guidelines)
  • Check the copyright year in README.md
  • Update RELEASE_NOTES.md with any notable new information for the end user.
  • Commit and push to the repo
  • Tag the release using the "Releases" feature in GitHub. Within 10 minutes, the changes should be live in production.
  • Update the version and updated parameters at XTools.

If an update is needed to the Wikimedia deployment of the PageInfo gadget, any mediawiki.org sysop can update XTools/ArticleInfo.js with the output from xtools:pageinfo-gadget.js.

Deployment

[edit]

Deployment is automated. There is a cron job on both https://xtools-dev.wmcloud.org and https://xtools.wmcloud.org that deploys the latest changes from GitHub to WMCloud every 10 minutes. XTools-dev uses the master branch, and XTools uses the latest tagged release.

The infrastructure side of things is documented at wikitech:Tool:XTools. XTools uses WMCloud and a Trove database server.

Glossary

[edit]

The following terms are used throughout the XTools codebase. These words may have different meanings in other MediaWiki-related software.

Anon / anonymous
Internal term referring to both temporary accounts and IPs. IP-specific methods will have "IP" in the name.
ArticleInfo
The old internal name for the Page History tool. The new internal name is PageInfo.
Length
In the context of edits, this is the size of the edit in bytes.
Model
A class holding the business logic for a tool.
Registered
Refers to users, edits or pages created by permanent registered accounts.
Repository
A class responsible for communication with the database, API, or external service.
Template
Refers to either the view (a Twig template), or the Template namespace.
Unregistered
Refers to users, edits or pages created by temporary accounts and IPs.

Additional help

[edit]
  • Email: tools.xtools@toolforge.org
  • Wikimedia Discord: #technical > XTools
  • IRC: ##wikimedia-xtools connect
  • MediaWiki talk page: Talk:XTools

Notes

[edit]
  1. ↑ We commit the assets to avoid having to build them on the production servers. Nothing in the public/build/ directory should be changed directly, including images. All such changes go in the assets/ directory then are compiled and/or copied to the build directory with Encore.
Retrieved from "https://www.mediawiki.org/w/index.php?title=XTools/Development&oldid=8130756"