Extension:GWToolset/Technical Design

Abstract
This section of the document answers questions about the project. What is the project? What is its purpose? What are the requirements?

Rationale
This section explains the value of the project, why we think it is of value, and how it fits into the bigger picture.

Process
Often cut into multiple sections, this describes how the feature is intended to work.

The current steps within the upload process are:
 * 1) Metadata detection
 * 2) Metadata mapping
 * 3) Batch preview
 * 4) Batch job creation

Metadata detection

 * 1) indicate which element within the metadata file represents a mediafile record.
 * 2) * a mediafile record contains metadata about the digital item such as author, date created, and a url to the mediafile.
 * 3) select a MediaWiki template that will display the mediafile metadata on the mediafile page.
 * 4) optionally select a previously saved metadata mapping that maps the metadata fields within the metadata file with the fields in the MediaWiki template.
 * 5) select the metadata file stored on your local hard drive.
 * 6) upload the metadata file.

The metadata file will be uploaded to a FileBackend store; a relative reference to the FileBackend store is placed in the subsequent HTML forms so that the extension can retrieve it as necessary.

Summary

 * a summary of the information provided in Metadata detection step.
 * a listing of all of the MediaWiki fields in the template selected in the Metadata detection step.
 * drop-down menus next to those fields that contain all of the metadata elements found in the metadata file.
 * a sample mediafile record with corresponding metadata information about the mediafile record.

Create a mapping

 * 1) create a mapping of the MediaWiki template fields to the metadata record elements by selecting the corresponding metadata record element from the drop-down next to the appropriate MediaWiki template field.
 * 2) * more than one metadata record element can be related to a MediaWiki template field.
 * 3) * a metadata record element can be related to many MediaWiki template fields.

Global categories

 * 1) optionally add global categories to the upload
 * 2) * global categories are applied to all mediafile records in the metadata file
 * 3) * more than one global category can be applied

Item specific categories

 * 1) optionally add item specific categories to the upload
 * 2) * these are applied to each mediafile record, but use item specific information.
 * for example, if the drop-down contains a mediafile field called author, the value for each individual record will be used.


 * 1) * the phrase allows you to prefix the mediafile metadata field with something like “created by” which could pair with a drop-down field author.

Summary

 * 1) optionally provide a summary message that gives an overview of why you are uploading this metadata file and all of its records.

Batch preview
Uploads and creates the first 3 mediafile pages based on those records found in the metadata file.
 * 1) you can preview the results of the mapping
 * 2) you can go back to the mapping step and make any necessary changes.

Batch job creation
If the Batch preview looks good, go ahead and create the batch job process. This step will create an initial UploadMetadataBatchJob that will cycle through all of the records found in the metadata file and create individual UploadMediaFileJobs that contain the mapping and specific record information as well as any categories that may have been added in the Mapping step.

The UploadMetadataBatchJob will continue to create another instance of itself as long as there are more metadata records to process. When it finishes cycling through all of the metadata records and has created the last UploadMediaFileJob, it will create a FileBackednCleanupJob that will delete the FileBackend metadata file that was originally uploaded in step 1.

The number of UploadMediaFileJobs created by each instance of UploadMetadataBatchJob is controlled by GWToolset\Config::$mediafile_job_throttle, default is 10.

There is also a throttle control on the total number of UploadMediaFileJobs allowed in the job queue; this is determined by GWToolset\Config::$mediafile_job_queue_max, default is 1000. If that limit is reached, UploadMetadataBatchJob will create another instance of itself to try and add its UploadMediaFileJobs on the next run. If delayedJobsEnabled is true, it will also set a jobReleaseTimestamp determined by GWToolset\Config::$metadata_job_delay, default is 5 minutes. This process will be attempted for a limited number of times, set by GWToolset\Config::$metadata_job_max_attempts, default is 5.

Gallery and Assets
These are images that are essential to understand the project. Mockups, screenshots, and icons fall into this category.
 * GWToolset demonstration screencast