Extension:OpenStackManager

Contributing
Please help contribute by taking and fixing a bug. If you'd like to work in the same environment as the rest of us, talk to Ryan_Lane in #wikimedia-tech on freenode.

Simple development spec
that this is mostly written from a Wikimedia Foundation perspective for now.

Nova manager

 * Software requirements
 * APIs:
 * AWS PHP API
 * MediaWiki extensions:
 * LDAP authentication
 * Semantic MediaWiki
 * Semantic Forms
 * Semantic Result Formats
 * External Data
 * Parser Functions
 * Dynamic Sidebar
 * Authentication:
 * LDAP using OpenDJ
 * Possibly also using OpenAM?
 * Console access:
 * Tomcat w/ Guacamole, or:
 * ajaxterm
 * DNS:
 * PowerDNS w/ LDAP backend
 * User/Project authentication/authorization
 * Pull Nova credentials from LDAP to use as proxy credentials
 * Nova users map directly to MediaWiki users
 * Nova projects map to MediaWiki groups, and MediaWiki namespaces
 * Project management
 * Special page for creating projects
 * Create project page on creation
 * Special page for managing projects
 * Add/delete users
 * Delete project
 * Only allow deletion if all pages in namespace are deleted!
 * Delete project page on deletion
 * Each project is a namespace. Only users in the ldap project group are given access
 * Make new right for vm management
 * Restrict renames to project members
 * Admin users can edit/manage all
 * How to handle editing of VM documentation? Only allow project members to do so? Allow writes to pages, but restrict edit access to manage interface to project members? Allow talk page modification only? Let the wiki sysadmin make these decisions and allow all?
 * How to create/delete namespaces dynamically? How to assign numbers to the namespaces, and have them be unique?
 * VM management
 * Special page for creating VMs
 * Create a documentation page on creation
 * Set userData['instance-name'] to fulltitle on instance creation, so that the name will be unique, and we can filter later instead of relying on instanceId
 * Add host node in LDAP with puppet configuration
 * On page rename, update the instance's userData name
 * Special page for resizing VMs
 * Special page for managing snapshots
 * Enable snapshot schedule
 * Restore from backup
 * Special page for console access
 * Special page for rebooting vm
 * Special page for rescue mode
 * Allow rescue mode by rebooting into a rescue disk and giving console access via guacamole
 * Instance is terminated on page deletion
 * Special page for changing public and private DNS
 * Special page for assigning, modifying, and removing security groups
 * Update VM page when complete
 * VM info
 * Add ExternalData code to pull info from OpenStack
 * User management
 * Special page for user information
 * Special page for importing keys
 * Special page for deleting keys
 * Users created in LDAP via MediaWiki
 * Schema required: openssh-lpk, nova
 * Objectclasses and attributes:
 * inetOrgPerson
 * cn
 * sn
 * person
 * posixAccount
 * uid
 * uidNumber (auto-generated)
 * gidNumber (auto-generated)
 * homeDirectory (/home/ )
 * shadowAccount
 * novaUser
 * accessKey (auto-generated)
 * secretKey (auto-generated)
 * isNovaAdmin (false)
 * ldapPublicKey
 * sshPublicKey (multi-attribute, populated via key manager)
 * IP Management
 * Special page for creating/deleting/assigning IPs
 * Update Property:Elastic IP on creation or deletion
 * Update VM pages on assignment
 * DNS management
 * PowerDNS with an LDAP backend
 * When adding instance, should also add DNS as well
 * Should be able to associate public and private addresses to public and private DNS
 * Security group management
 * Manage default security group
 * Add security groups
 * Delete security groups

Swift manager
TODO after Nova manager is complete.

Install LDAP, and optionally DNS and Puppet
For LDAP you can use your choice of other directory servers (likely excluding Active Directory). You'll need to add the puppet schema, the openssh-lpk schema, and the nova schema. I've been testing with OpenDJ. If you are more familiar with OpenLDAP, you may have an easier time with that.

For DNS you may be able to use any LDAP capable DNS server, but the extension has only been tested with PowerDNS with the LDAP backend in "strict" mode. It will not work with "tree" mode.

For puppet, you simply need to follow the puppetmaster installation instructions, and its LDAP configuration options.

Install OpenStack
Follow the instructions for a multinode install. Ensure you are using the trunk PPA. You will need a minimum of two hosts (can be VMs). You need to configure it to use MySQL and LDAP. You should be using version 2 of the LDAP schema (which is the default, if you are using trunk). Here's the configuration I'm using, with a controller node running MySQL, LDAP, nova-scheduler, nova-api, nova-network, and nova-scheduler, and 192.168.1.60 is the controller's IP:

The compute node should have the same configuration, but should only need the nova-compute and nova-volume services.

Install the LDAP Authentication plugin
You must first install and configure the LDAP authentication extension. It must be configured to use proxy authentication, to allow user creation, to allow password updates, to allow mailing of passwords, and should pull preferences. Here's an example configuration:

Install PHP prerequisites
The following PHP extensions are needed by the extension:


 * php5-uuid (OSSP)
 * php5-curl

Installing OpenStackManager
Download the trunk snapshot and untar into the extensions directory. Add the following to LocalSettings.php:

After installation, configure the extension using the options as shown below. You must currently use all options.

Roadmap
See spec for now.

1.1

 * Added floating ip output to instance list
 * Removed cast calls from code pulling info from classes, and added them to the output of the classes
 * Added support for cloud-init via $wgOpenStackManagerInstanceUserData
 * Can currently add cloudconfig, scripts, and upstarts
 * cloudconfig is currently an array that is converted to YAML, whereas scripts and upstarts load from given files
 * Added schema directory with required schemas in openldap and sun format
 * Added some useful python ldap scripts (note that scripts like add-ldap-user will not currently create users that are valid for OpenStackManager)
 * A bunch of code clean up and code documentation (thanks Reedy!)
 * Added a check to ensure the uid attribute isn't already used when adding user accounts. Anyone sane is already doing this on their directory server, but it's better safe than sorry.
 * Added a bug fix for project searches
 * Added support for project members to add/remove other project members
 * Added support for project role members to add/remove other project role members

1.0

 * Fixed a few possible XSS vectors
 * Added some missing localization in Special:NovaKey
 * Fixed a bug with key deletion

0.9

 * Finished adding support for project roles
 * Added support for global roles
 * Made some interface changes to Special:Project

0.8

 * Added initial support for managing global and project roles
 * Added availability zone and launch time display to instance list
 * Removed the rename option for now, as openstack doesn't currently support it
 * Added security group display to instance list

0.7

 * Fixed forms re-showing upon errors, when they should not re-show
 * Added Security Group support
 * Added support for Security Groups for instances, on instance creation

0.6

 * Finished adding localization support
 * Added support for allocating, release, associating, and deassociating floating IP addresses.
 * Refactored
 * Moved specials to /special
 * Added SpecialNova abstract class and subclasses all special pages beneath it
 * Added security
 * Special:NovaInstance actions require sysadmin role
 * Special:NovaDomain, Special:NovaAddress actions require netadmin role
 * Special:NovaProject is limited to wiki administrators
 * Roles are pulled from LDAP
 * Added an option to determine whether role checks for projects should require that users be in both the global and project role to be considered a member of the role. This is the current Nova behavior, until lp697936 is fixed.
 * Added support for managing public DNS entries via Special:NovaAddress
 * Public DNS entries can only be added to allocated floating IP addresses
 * A DNS entry can contain multiple A records, and multiple associated domain records (for aliases)
 * Restricted to netadmins

0.5

 * Added a small amount of locailization to Special:NovaInstance
 * Changed DNS configuration options for instances
 * Can now choose a domain only based upon location
 * Location and instance DNS is linked, as the DNS entry created will be a private DNS entry
 * Added a location field to the form for domains, so that domains can be location specific
 * Domains with no location attribute set will be public DNS domains; this likely needs to be made clear on the form
 * Fixed instance list for Special:NovaInstance; was previously only showing one instance, even if multiple instances existed
 * Added a OpenStackNovaHostJob, to add IP addresses to host entries in a deffered manner
 * This is due to a change in Nova; previously IP addresses were assigned on instance creation, now they are created on instance scheduling, so IP address information is not available immediately
 * Removed in-process caching for Nova API responses; was causing more trouble than it was worth. Should be re-added as memcache caches later
 * Modified the behavior of getInstance to get all instances from Nova, and return a specific instance, since this behavior changed in the API (which is likely a bug)
 * Added various extra error checking
 * Removed t1.micro as an instance type, as it isn't valid in Nova

0.4

 * Removed key-name as a field for instance creation
 * This should be added back in as a configurable option. We don't need key injection, but others may
 * Added support for PowerDNS with an LDAP backend
 * Added a special page for creating and deleting DNS domains
 * Added a class for hosts and domains

0.3

 * Added support for adding/removing ssh keys
 * Added support for adding/removing projects
 * Added support for configuration of extra namespaces using project from LDAP
 * Added instance name support via "DisplayName" property in OpenStack
 * Added a VM and VM_talk namespace; 498 and 499 respectively (500+ is for nova project namespaces)

0.2

 * Added actions to NovaInstance special page for instance creation, modification, and lists
 * Added security to Instance creation
 * Will ensure you are in the project you wish to create instances in
 * Will ensure your user account has Nova credentials
 * Added a NovaUser class to represent Nova user accounts
 * Functions added for getting credentials, getting project and role memberships, checking for project and role memberships, and checking for existence

0.1

 * Initial release
 * Very basic support for EC2 API
 * Can fetch images, instances, keys, availability zones, and instance types
 * Can create an instance
 * Has absolutely no error checking
 * Has no per-user security - uses admin for everything