Extension:OpenStackManager

Contributing
Please help contribute by taking and fixing a bug. If you'd like to work in the same environment as the rest of us, talk to Ryan_Lane in #wikimedia-tech on freenode.

Simple development spec
that this is mostly written from a Wikimedia Foundation perspective for now.

Nova manager

 * Software requirements
 * APIs:
 * AWS PHP API
 * MediaWiki extensions:
 * LDAP authentication
 * Semantic MediaWiki
 * Semantic Forms
 * Semantic Result Formats
 * External Data
 * Parser Functions
 * Dynamic Sidebar
 * Authentication:
 * LDAP using OpenDJ
 * Possibly using also using OpenAM?
 * Console access:
 * Tomcat w/ Guacamole, or:
 * ajaxterm
 * DNS:
 * PowerDNS w/ LDAP backend
 * User/Project authentication/authorization
 * Pull Nova credentials from LDAP to use as proxy credentials
 * Nova users map directly to MediaWiki users
 * Nova projects map to MediaWiki groups, and MediaWiki namespaces
 * Project management
 * Special page for creating projects
 * Create project page on creation
 * Special page for managing projects
 * Add/delete users
 * Delete project
 * Only allow deletion if all pages in namespace are deleted!
 * Delete project page on deletion
 * Each project is a namespace. Only users in the ldap project group are given access
 * Make new right for vm management
 * Restrict renames to project members
 * Admin users can edit/manage all
 * How to handle editing of VM documentation? Only allow project members to do so? Allow writes to pages, but restrict edit access to manage interface to project members? Allow talk page modification only? Let the wiki sysadmin make these decisions and allow all?
 * How to create/delete namespaces dynamically? How to assign numbers to the namespaces, and have them be unique?
 * VM management
 * Special page for creating VMs
 * Create a documentation page on creation
 * Set userData['instance-name'] to fulltitle on instance creation, so that the name will be unique, and we can filter later instead of relying on instanceId
 * Add host node in LDAP with puppet configuration
 * On page rename, update the instance's userData name
 * Special page for resizing VMs
 * Special page for managing snapshots
 * Enable snapshot schedule
 * Restore from backup
 * Special page for console access
 * Special page for rebooting vm
 * Special page for rescue mode
 * Allow rescue mode by rebooting into a rescue disk and giving console access via guacamole
 * Instance is terminated on page deletion
 * Special page for changing public and private DNS
 * Special page for assigning, modifying, and removing security groups
 * Update VM page when complete
 * VM info
 * Add ExternalData code to pull info from OpenStack
 * User management
 * Special page for user information
 * Special page for importing keys
 * Special page for deleting keys
 * Users created in LDAP via MediaWiki
 * Schema required: openssh-lpk, nova
 * Objectclasses and attributes:
 * inetOrgPerson
 * cn
 * sn
 * person
 * posixAccount
 * uid
 * uidNumber (auto-generated)
 * gidNumber (auto-generated)
 * homeDirectory (/home/ )
 * shadowAccount
 * novaUser
 * accessKey (auto-generated)
 * secretKey (auto-generated)
 * isNovaAdmin (false)
 * ldapPublicKey
 * sshPublicKey (multi-attribute, populated via key manager)
 * IP Management
 * Special page for creating/deleting/assigning IPs
 * Update Property:Elastic IP on creation or deletion
 * Update VM pages on assignment
 * DNS management
 * PowerDNS with an LDAP backend
 * When adding instance, should also add DNS as well
 * Should be able to associate public and private addresses to public and private DNS
 * Security group management
 * Manage default security group
 * Add security groups
 * Delete security groups

Swift manager
TODO after Nova manager is complete.

Install LDAP, and optionally DNS and Puppet
For LDAP you can use your choice of other directory servers (likely excluding Active Directory). You'll need to add the puppet schema, the openssh-lpk schema, and the nova schema. I've been testing with OpenDJ. If you are more familiar with OpenLDAP, you may have an easier time with that.

For DNS you may be able to use any LDAP capable DNS server, but the extension has only been tested with PowerDNS with the LDAP backend in "strict" mode. It will not work with "tree" mode.

For puppet, you simply need to follow the puppetmaster installation instructions, and its LDAP configuration options.

Install OpenStack
Follow the instructions for a multinode install. Ensure you are using the trunk PPA. You will need a minimum of two hosts (can be VMs). You need to configure it to use MySQL and LDAP. You should be using version 2 of the LDAP schema (which is the default, if you are using trunk). Here's the configuration I'm using, with a controller node running MySQL, LDAP, nova-scheduler, nova-api, nova-network, and nova-scheduler, and 192.168.1.60 is the controller's IP:

The compute node should have the same configuration, but should only need the nova-compute and nova-volume services.

Install the LDAP Authentication plugin
You must first install and configure the LDAP authentication extension. It must be configured to use proxy authentication, to allow user creation, to allow password updates, to allow mailing of passwords, and should pull preferences. Here's an example configuration:

Installing OpenStackManager
Download the trunk snapshot and untar into the extensions directory. Add the following to LocalSettings.php:

After installation, configure the extension using the options as shown below. You must currently use all options.

Roadmap
See spec for now.

0.5

 * Added a small amount of locailization to Special:NovaInstance
 * Changed DNS configuration options for instances
 * Can now choose a domain only based upon location
 * Location and instance DNS is linked, as the DNS entry created will be a private DNS entry
 * Added a location field to the form for domains, so that domains can be location specific
 * Domains with no location attribute set will be public DNS domains; this likely needs to be made clear on the form
 * Fixed instance list for Special:NovaInstance; was previously only showing one instance, even if multiple instances existed
 * Added a OpenStackNovaHostJob, to add IP addresses to host entries in a deffered manner
 * This is due to a change in Nova; previously IP addresses were assigned on instance creation, now they are created on instance scheduling, so IP address information is not available immediately
 * Removed in-process caching for Nova API responses; was causing more trouble than it was worth. Should be re-added as memcache caches later
 * Modified the behavior of getInstance to get all instances from Nova, and return a specific instance, since this behavior changed in the API (which is likely a bug)
 * Added various extra error checking
 * Removed t1.micro as an instance type, as it isn't valid in Nova

0.4

 * Removed key-name as a field for instance creation
 * This should be added back in as a configurable option. We don't need key injection, but others may
 * Added support for PowerDNS with an LDAP backend
 * Added a special page for creating and deleting DNS domains
 * Added a class for hosts and domains

0.3

 * Added support for adding/removing ssh keys
 * Added support for adding/removing projects
 * Added support for configuration of extra namespaces using project from LDAP
 * Added instance name support via "DisplayName" property in OpenStack
 * Added a VM and VM_talk namespace; 498 and 499 respectively (500+ is for nova project namespaces)

0.2

 * Added actions to NovaInstance special page for instance creation, modification, and lists
 * Added security to Instance creation
 * Will ensure you are in the project you wish to create instances in
 * Will ensure your user account has Nova credentials
 * Added a NovaUser class to represent Nova user accounts
 * Functions added for getting credentials, getting project and role memberships, checking for project and role memberships, and checking for existence

0.1

 * Initial release
 * Very basic support for EC2 API
 * Can fetch images, instances, keys, availability zones, and instance types
 * Can create an instance
 * Has absolutely no error checking
 * Has no per-user security - uses admin for everything