Extension:OpenStackManager

Description of the project and author's support pages
For the full specifications and support of OpenStack managers, see OpenStack.org. You may also read these pages on the main author's blog (on ryandlane.com) related to this MediaWiki-based implementation of these OpenStack managers:
 * Building a test and development infrastructure using OpenStack
 * Why I chose Mediawiki for my OpenStack manager project

Contributing
Please help contribute by taking and fixing a bug. If you'd like to work in the same environment as the rest of us, talk to Ryan_Lane in #wikimedia-tech on freenode.

For now only Nova managers (for computing clouds) are specified and experimental; it is developed mostly in the context of the usage by the Wikimedia Foundation on its servers farm. The support of Swift managers (for storage clouds), and possibly other types of clouds, may be developed later.

Install LDAP, and optionally DNS and Puppet
For LDAP you can use your choice of other directory servers (likely excluding Active Directory). You'll need to add the puppet schema, the openssh-lpk schema, and the nova schema. I've been testing with OpenDJ. If you are more familiar with OpenLDAP, you may have an easier time with that. The schema files are included with the extension in both openldap and sun format.

For DNS you may be able to use any LDAP capable DNS server, but the extension has only been tested with PowerDNS with the LDAP backend in "strict" mode. It will not work with "tree" mode.

For puppet, you simply need to follow the puppetmaster installation instructions, and its LDAP configuration options.

Install OpenStack
Follow the instructions for a multinode install. Ensure you are using the trunk PPA. You will need a minimum of two hosts (can be VMs). You need to configure it to use MySQL and LDAP. You should be using version 2 of the LDAP schema (which is the default, if you are using trunk). Here's the configuration I'm using, with a controller node running MySQL, LDAP, nova-scheduler, nova-api, nova-network, and nova-scheduler, and 192.168.1.60 is the controller's IP:

The compute node should have the same configuration, but should only need the nova-compute and nova-volume services.

Install the LDAP Authentication plugin
You must first install and configure the LDAP authentication extension. It must be configured to use proxy authentication, to allow user creation, to allow password updates, to allow mailing of passwords, and should pull preferences. Here's an example configuration:

Install PHP prerequisites
The following PHP extensions are needed by the extension:


 * php5-uuid (OSSP)
 * php5-curl

Installing OpenStackManager
Download the trunk snapshot and untar into the extensions directory. Add the following to LocalSettings.php:

After installation, configure the extension using the options as shown below. You must currently use all options.

Initial setup
Initial setup can be awkward, due to the chicken and egg problem faced with wiki admin privileges and the LDAP plugin, and the need for an admin user for Nova. These are the steps I recommend:


 * 1) Fully configure Nova, and MediaWiki, the LDAP extension, and the OpenStackManager extension (minus the nova admin credentials)
 * 2) Create a user for yourself
 * 3) Enable local authentication using the following in LocalSettings.php: $wgLDAPUseLocal = true;
 * 4) Promote your user to Administrator and Bureaucrat
 * 5) Disable local authentication
 * 6) At this point, if you wish, you can
 * 7) Create a nova admin user
 * 8) Add the user to the "cloudadmins" global nova role
 * 9) * Do not give this user any special wiki group rights; in fact, you can block this user if you'd like
 * 10) Get the user's accesskey and secretkey from ldap: ldapsearch -x '(|(cn= )(uid=<username))'
 * 11) Add these to LocalSettings.php for the nova admin credentials; ensure that the accesskey is " : ", where is the default project name (this is annoyingly required by the EC2 API)
 * 12) Add your user to whichever roles you find appropriate, and add yourself to a project

This extension supports the ability to disable user creation by anyone other than Admins, if you choose to do so. If you would like to, set the following in LocalSettings.php:

LDAP
I've included some useful ldap scripts with the extension, under the scripts directory. Note that some of these are included because I'd like to improve them to the point where they are usable for this project, but will not currently work properly. The really useful ones are:


 * homedirectorymanager.py : Automatically creates home directories for ldap users, including adding their ssh keys to authorized_keys, and adding skel files
 * puppetsigner.py : Will automatically sign puppet ca requests for instances created, by ensuring they are valid LDAP entries
 * ldaplist : A utility made for easier display of information in your ldap database. A python reimplementation of the Solaris tool
 * modify-ldap-user: Can modify attributes or rename ldap users. Be careful with this, some actions may cause problems with the extension!
 * mail-instance-creator.py: Can be used to email users when an instance has finished creating itself, given a from address, a to address, the user's language code, and a wiki address. The script will pull a localized message in the user's language from the wiki, and send it to the user from the from address.

Upstarts
I've included some useful upstarts, found under the upstarts directory. Some may be WMF specific, but are still likely useful.


 * ttyS0.conf : Enables ttyS0 so that a web console can be used on an instance. This is useful when using ajaxterm.
 * runonce-fixpuppet.conf : Forces an initial puppet run, making it wait for a certificate, disables pluginsync, and ensures a consistent pid file is used. This may be a little WMF specific, but I found it makes an initial run of puppet more consistent.

Simple development spec
that this is mostly written from a Wikimedia Foundation perspective for now.

Nova manager

 * Software requirements
 * APIs:
 * AWS PHP API
 * MediaWiki extensions:
 * LDAP authentication
 * Semantic MediaWiki
 * Semantic Forms
 * Semantic Result Formats
 * External Data
 * Parser Functions
 * Dynamic Sidebar
 * Authentication:
 * LDAP using OpenDJ
 * Possibly also using OpenAM?
 * Console access:
 * Tomcat w/ Guacamole, or:
 * ajaxterm
 * DNS:
 * PowerDNS w/ LDAP backend
 * User/Project authentication/authorization
 * Pull Nova credentials from LDAP to use as proxy credentials
 * Nova users map directly to MediaWiki users
 * Nova projects map to MediaWiki groups, and MediaWiki namespaces
 * Project management
 * Special page for creating projects
 * Create project page on creation
 * Special page for managing projects
 * Add/delete users
 * Delete project
 * Only allow deletion if all pages in namespace are deleted!
 * Delete project page on deletion
 * Each project is a namespace. Only users in the ldap project group are given access
 * Make new right for vm management
 * Restrict renames to project members
 * Admin users can edit/manage all
 * How to handle editing of VM documentation? Only allow project members to do so? Allow writes to pages, but restrict edit access to manage interface to project members? Allow talk page modification only? Let the wiki sysadmin make these decisions and allow all?
 * How to create/delete namespaces dynamically? How to assign numbers to the namespaces, and have them be unique?
 * VM management
 * Special page for creating VMs
 * Create a documentation page on creation
 * Set userData['instance-name'] to fulltitle on instance creation, so that the name will be unique, and we can filter later instead of relying on instanceId
 * Add host node in LDAP with puppet configuration
 * On page rename, update the instance's userData name
 * Special page for resizing VMs
 * Special page for managing snapshots
 * Enable snapshot schedule
 * Restore from backup
 * Special page for console access
 * Special page for rebooting vm
 * Special page for rescue mode
 * Allow rescue mode by rebooting into a rescue disk and giving console access via guacamole
 * Instance is terminated on page deletion
 * Special page for changing public and private DNS
 * Special page for assigning, modifying, and removing security groups
 * Update VM page when complete
 * VM info
 * Add ExternalData code to pull info from OpenStack
 * User management
 * Special page for user information
 * Special page for importing keys
 * Special page for deleting keys
 * Users created in LDAP via MediaWiki
 * Schema required: openssh-lpk, nova
 * Objectclasses and attributes:
 * inetOrgPerson
 * cn
 * sn
 * person
 * posixAccount
 * uid
 * uidNumber (auto-generated)
 * gidNumber (auto-generated)
 * homeDirectory (/home/ )
 * shadowAccount
 * novaUser
 * accessKey (auto-generated)
 * secretKey (auto-generated)
 * isNovaAdmin (false)
 * ldapPublicKey
 * sshPublicKey (multi-attribute, populated via key manager)
 * IP Management
 * Special page for creating/deleting/assigning IPs
 * Update Property:Elastic IP on creation or deletion
 * Update VM pages on assignment
 * DNS management
 * PowerDNS with an LDAP backend
 * When adding instance, should also add DNS as well
 * Should be able to associate public and private addresses to public and private DNS
 * Security group management
 * Manage default security group
 * Add security groups
 * Delete security groups

Swift manager
TODO after Nova manager is complete.

Roadmap
See spec for now.

1.3

 * Added a reboot action for instances
 * Made compatibility changes for cactus and diablo nova releases
 * Made compatibility changes for MediaWiki 1.18
 * Added support for configurable naming attributes
 * Added support for adding objectclasses and attributes for users that are missing them
 * It's now possible for MediaWiki to no longer have to create users, only update select user attributes and objectclasses
 * Made a bunch of bugfixes regarding security groups
 * Added support for wildcard DNS entries
 * Bugfixes from John Du Hart, Sam Reed and Mark Hershberger
 * Added realm and instancename variables to puppet default variables, so that they can be used in puppet runs
 * Added support for wiki page creation for Projects
 * Added support for configuring default images for instances
 * Added support for creating server admin logs per project
 * Added support for default security group rules on project creation
 * Added dialog to project creation for adding members to projects and roles upon creation
 * Added support for managing puppet classes and variables through the interface, rather than LocalSettings.php
 * Made a usability change to instance creation: the default security group will always be selected by default
 * Added support for including the ssh key fingerprint of an instance in the "your instance is ready" emails
 * Made a usability change to only show actions to users if they can perform them
 * Lots of other minor bug fixes

1.2

 * Added support for managing sudo policies. Currently supports adding, modifying and deleting sudo policies with sudouser, sudohost, sudocommand, and sudooptions attributes. Requies wiki admin privileges to use the special page.
 * Changed host addition behavior to add project to the host entry's puppet variables by default.
 * Added support for adding/modifying/deleting instance information in Nova_Resource: wiki pages when creating/configuring/deleting instances. This can be disabled using the $wgOpenStackManagerCreateResourcePages configuration option.
 * Added support for creating/deleting/attaching/detaching volumes.
 * Changed the display of all tables to use the sortable CSS class.
 * Changed name of the VM namespace, to instead be Nova Resources. All Nova resources will be placed in this namespace.
 * Various MediaWiki 1.17 compatibility changes.
 * Added a default shell config option, so that a shell can be added to user entries.
 * Changed behavior of host addition to a location field on instance DNS entries.
 * Code cleanup for localization.
 * Fixed a number of spots the extension was throwing warnings.
 * Changed Special:NovaDomain to check for cloudadmins membership if roles are intersecting, as otherwise netadmins for specific projects could also manage DNS domains.
 * Fixed role check bug in Special:Domain, so that cloudadmins/netadmins can manage domains.
 * Fixed various bugs in project/role membership management.
 * Fixed a bug in public host management when an associated domain was removed that was also named for the dc attribute.
 * Changed logic of which images will be shown for image creation: only show images that are public, are available, are of type machine, and have an image name
 * Changed behavior of create action for Special:NovaInstance to show image names instead of image ids
 * Added an upstart that can be used to enable consoles in instances.
 * Added support for pulling instance type information from the Nova EC2 admin API.
 * Added a class to represent instance types.
 * Added support for the create action in Special:NovaInstance to list detailed information about instance types.
 * Added a global config option for the Nova EC2 admin API's endpoint.
 * Added support for project role members to add/remove other project role members.
 * Added a utility upstart that can be used to do some puppet configuration changes before it starts on initial boot. Should be removed during first puppet run.
 * Added support for getting console output from instances.
 * Changed actions td on instance list page to be a list.
 * Added script and localization messages for sending notification emails when instances are fully created
 * Added default puppet variables for all instances created; adding the following variables, which represent the instance creator's email address, (wiki) user name, and interface language:
 * instancecreator_email
 * instancecreator_username
 * instancecreator_lang
 * Changed scripts and upstarts in clouddata to use named values in the arrays, where the named values are the name of the file that will be created on the instance.
 * Changed behavior of instance termination to ensure any associated addresses are disassociated first.
 * Added a number of validation checks for nova resource creation and modification.
 * Changed creation of host entries to use instanceid instead of hostname for the dn, and changed creation to only use associated domain, not cname, since cnamerecords don't work when created that way.
 * Added a script to automatically sign puppet certificate requests for hosts that exist in LDAP.
 * Upgraded AWS SDK for PHP to 1.2.6 "Ifrit"

1.1

 * Added floating ip output to instance list
 * Removed cast calls from code pulling info from classes, and added them to the output of the classes
 * Added support for cloud-init via $wgOpenStackManagerInstanceUserData
 * Can currently add cloudconfig, scripts, and upstarts
 * cloudconfig is currently an array that is converted to YAML, whereas scripts and upstarts load from given files
 * Added schema directory with required schemas in openldap and sun format
 * Added some useful python ldap scripts (note that scripts like add-ldap-user will not currently create users that are valid for OpenStackManager)
 * A bunch of code clean up and code documentation (thanks Reedy!)
 * Added a check to ensure the uid attribute isn't already used when adding user accounts. Anyone sane is already doing this on their directory server, but it's better safe than sorry.
 * Added a bug fix for project searches
 * Added support for project members to add/remove other project members

1.0

 * Fixed a few possible XSS vectors
 * Added some missing localization in Special:NovaKey
 * Fixed a bug with key deletion

0.9

 * Finished adding support for project roles
 * Added support for global roles
 * Made some interface changes to Special:Project

0.8

 * Added initial support for managing global and project roles
 * Added availability zone and launch time display to instance list
 * Removed the rename option for now, as openstack doesn't currently support it
 * Added security group display to instance list

0.7

 * Fixed forms re-showing upon errors, when they should not re-show
 * Added Security Group support
 * Added support for Security Groups for instances, on instance creation

0.6

 * Finished adding localization support
 * Added support for allocating, release, associating, and deassociating floating IP addresses.
 * Refactored
 * Moved specials to /special
 * Added SpecialNova abstract class and subclasses all special pages beneath it
 * Added security
 * Special:NovaInstance actions require sysadmin role
 * Special:NovaDomain, Special:NovaAddress actions require netadmin role
 * Special:NovaProject is limited to wiki administrators
 * Roles are pulled from LDAP
 * Added an option to determine whether role checks for projects should require that users be in both the global and project role to be considered a member of the role. This is the current Nova behavior, until lp697936 is fixed.
 * Added support for managing public DNS entries via Special:NovaAddress
 * Public DNS entries can only be added to allocated floating IP addresses
 * A DNS entry can contain multiple A records, and multiple associated domain records (for aliases)
 * Restricted to netadmins

0.5

 * Added a small amount of locailization to Special:NovaInstance
 * Changed DNS configuration options for instances
 * Can now choose a domain only based upon location
 * Location and instance DNS is linked, as the DNS entry created will be a private DNS entry
 * Added a location field to the form for domains, so that domains can be location specific
 * Domains with no location attribute set will be public DNS domains; this likely needs to be made clear on the form
 * Fixed instance list for Special:NovaInstance; was previously only showing one instance, even if multiple instances existed
 * Added a OpenStackNovaHostJob, to add IP addresses to host entries in a deffered manner
 * This is due to a change in Nova; previously IP addresses were assigned on instance creation, now they are created on instance scheduling, so IP address information is not available immediately
 * Removed in-process caching for Nova API responses; was causing more trouble than it was worth. Should be re-added as memcache caches later
 * Modified the behavior of getInstance to get all instances from Nova, and return a specific instance, since this behavior changed in the API (which is likely a bug)
 * Added various extra error checking
 * Removed t1.micro as an instance type, as it isn't valid in Nova

0.4

 * Removed key-name as a field for instance creation
 * This should be added back in as a configurable option. We don't need key injection, but others may
 * Added support for PowerDNS with an LDAP backend
 * Added a special page for creating and deleting DNS domains
 * Added a class for hosts and domains

0.3

 * Added support for adding/removing ssh keys
 * Added support for adding/removing projects
 * Added support for configuration of extra namespaces using project from LDAP
 * Added instance name support via "DisplayName" property in OpenStack
 * Added a VM and VM_talk namespace; 498 and 499 respectively (500+ is for nova project namespaces)

0.2

 * Added actions to NovaInstance special page for instance creation, modification, and lists
 * Added security to Instance creation
 * Will ensure you are in the project you wish to create instances in
 * Will ensure your user account has Nova credentials
 * Added a NovaUser class to represent Nova user accounts
 * Functions added for getting credentials, getting project and role memberships, checking for project and role memberships, and checking for existence

0.1

 * Initial release
 * Very basic support for EC2 API
 * Can fetch images, instances, keys, availability zones, and instance types
 * Can create an instance
 * Has absolutely no error checking
 * Has no per-user security - uses admin for everything