SwiftMedia using Rackspace Cloud Files

Jump to: navigation, search

Good morning Aaron,

I am working on a configuration using Rackspace CloudFiles rather than configure my own swift instance. We feel the CDN can provide what we need without having to manage the extra servers. I upgraded to 1.19 as you suggested and am researching the FileBackend object for this. I am, however, trying to locate a central information resource that explains in more detail the steps to integrate between FileBackend and cloudfiles. Do you have any resources that can better help me on this.

Also, rather than installing a full swift implementation, is there a way to install a limited swift configuration for the swiftproxy only?

Thanks! Justin

Jwestyp (talk)‎14:31, 10 May 2012

You just add "include( 'path/to/extensions/SwiftCloudFiles/SwiftCloudFiles.php' )" to LocalSettings.php, and SwiftFileBackend will use it.

Aaron (talk)‎16:27, 10 May 2012
 

You will also need to set $wgLocalFileRepo to zone URLs that map to the CDN urls. I may have to change FileRepo to make this work properly for the 'temp' zone (as well as for all other zones with just one setting).

Aaron (talk)‎16:50, 10 May 2012

I notice that much of the documentation for swift is written with an emphasis on Debian. Is there support for Cloud Files and Swift on Red Hat?

Jwestyp (talk)‎17:04, 10 May 2012

The cloudfiles package is just PHP, it should work for any system, provided the proper php extensions are enabled via php.ini (and compile flags)...namely fileinfo, curl, mb_string. Unless you are setting up your own swift cluster, you don't have to worry about that. It doesn't matter which OS flavor is running the swift nodes (all contact between cloudfiles and swift is done via an HTTP/HTTPS API).

Aaron (talk)‎19:18, 10 May 2012
 
 

I think to use rackspace CDN, I think SwiftFileBackend will need to be tweaked. The prepare() and secure() calls need to call container CDN functions.

Aaron (talk)‎18:24, 10 May 2012

Aaron,

Thanks so much for the excellent support. I will see what I can get from this. I will document as much of it as I can and I will provide it to you.

Justin

Jwestyp (talk)‎19:23, 10 May 2012
 

Aaron,

I have configured local uploads to my wiki and was trying to move to the files you updated yesterday for me.

Trying the files from your commit yesterday and am getting a blank page when clicking upload and this error from my httpd error log.

[Fri May 11 10:48:54 2012] [error] [client 76.31.142.155] PHP Fatal error: Call to undefined method FileBackend::isPathTraversalFree() in /var/www/html/mediawiki/includes/filerepo/FileRepo.php on line 1422, referer: http://*removed domain*/mediawiki/index.php/Special:Upload

My localsettings are: $wgFileExtensions = array('png','gif','jpg','jpeg','svg','psd','ai','doc','xls','mpp','pdf','ppt','tiff','bmp','docx', 'xlsx', 'pptx','ps','odt','ods','odp','odg','txt'); //What file extensions to allow to be uploaded $wgUploadDirectory = "{$IP}/media"; //Upload directory /mediawiki/media $wgUploadPath = "{$wgScriptPath}/media"; //Upload Path /mediawiki/media

The version in the default settings file from the commit is 1.20alpha and I am still on mediawiki 1.19.

Thanks Justin

Jwestyp (talk)‎16:01, 11 May 2012

Maybe you should just do a clone of the 'master' branch of MediaWiki from git and use that. That would assure that new CDN features and fixes can be applied with "git pull" and everything would be a consistent MW version (rather than picking out patches). Of course, you'd have to wait for things to get merged.

Aaron (talk)‎16:06, 11 May 2012

Aaron,

I did a clone of the master branch, added the commit files you created and then a clone of the extension for SwiftCloudFiles.

I assume I will need a filerepo array for this to work.

Do you have an example of the filerepo array settings? Do I still need the swiftproxy set up as well?

Thanks Justin

Jwestyp (talk)‎19:54, 11 May 2012

Go to http://noc.wikimedia.org/conf/highlight.php?file=CommonSettings.php and look for "OpenStack". Around there you can see some swift backend config and some file repo config. Of course our repo is still using the NFS ones, but you could imagine just swapping out the 'backend' => 'local-NFS', lines with the swift backend name. You can probably ignore 'shardViaHashLevels' and 'fileJournal'. Also, 'wikiId' can be ignored or set to some pretty and concise unique ID for the wiki. You won't need to deal with the 'shared' repo or backend either. So your setup will be much simpler.

Aaron (talk)‎23:14, 11 May 2012

Aaron,

Using the information from the link, I configured the SwiftBackend. However, I am getting this error: PHP Notice: Undefined index: directory in /var/www/html/mediawiki/includes/filerepo/backend/FileBackendGroup.php on line 83.

Ideas?

Thank you so much for all of your help!

Jwestyp (talk)‎14:40, 14 May 2012

Hmm, it might be easier to just email me the config. Here is some config from my testwiki:

# Backend/Repo stuff...
require( "$IP/../extensions/SwiftCloudFiles/SwiftCloudFiles.php" );
$wgFileBackends[] = array(
        'name'               => 'localSwift',
        'class'              => 'SwiftFileBackend',
        'wikiId'             => 'aaron-ubuntu-wiki',
        'lockManager'        => 'fsLockManager',
        'swiftAuthUrl'       => $wgSwiftConf['authUrl'],
        'swiftUser'          => $wgSwiftConf['user'],
        'swiftKey'           => $wgSwiftConf['key'],
        #'swiftProxyUser'     => 'test:anon',
        'shardViaHashLevels' => array(
                'local-public'  => array( 'levels' => 2, 'base' => 16, 'repeat' => true ),
                'local-thumb'   => array( 'levels' => 2, 'base' => 16, 'repeat' => true ),
                'local-deleted' => array( 'levels' => 2, 'base' => 36, 'repeat' => false )
        ),
        #'readOnly'           => 'Temporary planned maintenence for 5 min',
        'fileJournal'        => array( 'class' => 'DBFileJournal', 'wiki' => false ),
        'parallelize'        => 'implicit'
);
$wgLocalFileRepo = array(
        'class'             => 'LocalRepo',
        'name'              => 'local',
        'backend'           => 'localSwift',
        'scriptDirUrl'      => $wgScriptPath,
        'scriptExtension'   => $wgScriptExtension,
        'url'               => 'http://localhost/TrunkWiki/core/images/local',
        'hashLevels'        => 2,
        'transformVia404'   => true,
        'deletedHashLevels' => 2
);
Aaron (talk)‎16:35, 14 May 2012

Aaron,

I will do that. Can you email me at jwestyp@gmail.com and I will respond with it later. I am going into a meeting and will be back on it later.

Thanks Justin

Jwestyp (talk)‎16:52, 14 May 2012
 
 
 
 

also,

Local uploads work great and I have included the call to the extensions/SwiftCloudFiles/SwiftCloudFiles.php in my LocalSettings. I just am not sure where to go from there. The page for Extension:SwiftCloudFiles doesn't have any content and still points to the Rackspace-cloudfiles git repository.

Jwestyp (talk)‎20:16, 11 May 2012
 
 

CDN support added in https://gerrit.wikimedia.org/r/#/c/7688/. To use this, the 'zones' parameter (from https://gerrit.wikimedia.org/r/#/c/7199/) of FileRepo will have to be set for 'thumb', 'public', and 'temp' to the base CDN URL. Also see the comments in DefaultSettings.php.

Aaron (talk)‎23:18, 14 May 2012

Aaron,

I grabbed the files and put them in place. I also copied your config from above and get an error that wgSwiftConf is undefined when loading the Special:Upload page and I get a blank screen. The following error repeats several times:

Notice: Undefined variable: wgSwiftConf in /var/www/html/mediawiki/LocalSettings.php on line 157
[Tue May 15 15:44:30 2012] [error] [client 96.228.72.32] PHP Stack trace:
[Tue May 15 15:44:30 2012] [error] [client 96.228.72.32] PHP 1. {main}() /var/www/html/mediawiki/index.php:0
[Tue May 15 15:44:30 2012] [error] [client 96.228.72.32] PHP 2. require() /var/www/html/mediawiki/index.php:54
[Tue May 15 15:44:30 2012] [error] [client 96.228.72.32] PHP 3. require_once() /var/www/html/mediawiki/includes/WebStart.php:134

I then get these errors as well:

PHP Fatal error: Class 'FileBackendStoreOpHandle' not found in /var/www/html/mediawiki/includes/filerepo/backend/SwiftFileBackend.php on line 1179, referer: *url removed*/mediawiki/index.php/Main_Page
[Tue May 15 15:44:35 2012] [error] [client 96.228.72.32] PHP Stack trace:, referer:*url removed*/mediawiki/index.php/Main_Page
[Tue May 15 15:44:35 2012] [error] [client 96.228.72.32] PHP 1. {main}() /var/www/html/mediawiki/index.php:0, referer: *url removed*/mediawiki/index.php/Main_Page
[Tue May 15 15:44:35 2012] [error] [client 96.228.72.32] PHP 2. MediaWiki->run() /var/www/html/mediawiki/index.php:58, referer: *url removed*/mediawiki/index.php/Main_Page
[Tue May 15 15:44:35 2012] [error] [client 96.228.72.32] PHP 3. MediaWiki->main() /var/www/html/mediawiki/includes/Wiki.php:502, referer: *url removed*/mediawiki/index.php/Main_Page
[Tue May 15 15:44:35 2012] [error] [client 96.228.72.32] PHP 4. MediaWiki->performRequest() /var/www/html/mediawiki/includes/Wiki.php:592, referer: *url removed*/mediawiki/index.php/Main_Page
[Tue May 15 15:44:35 2012] [error] [client 96.228.72.32] PHP 5. SpecialPageFactory::executePath() /var/www/html/mediawiki/includes/Wiki.php:263, referer: *url removed*/mediawiki/index.php/Main_Page
[Tue May 15 15:44:35 2012] [error] [client 96.228.72.32] PHP 6. SpecialUpload->execute() /var/www/html/mediawiki/includes/SpecialPageFactory.php:493, referer: *url removed*/mediawiki/index.php/Main_Page
[Tue May 15 15:44:35 2012] [error] [client 96.228.72.32] PHP 7. SpecialUpload->getUploadForm() /var/www/html/mediawiki/includes/specials/SpecialUpload.php:181, referer: *url removed*/mediawiki/index.php/Main_Page
[Tue May 15 15:44:35 2012] [error] [client 96.228.72.32] PHP 8. SpecialUpload->getWatchCheck() /var/www/html/mediawiki/includes/specials/SpecialUpload.php:220, referer: *url removed*/mediawiki/index.php/Main_Page
[Tue May 15 15:44:35 2012] [error] [client 96.228.72.32] PHP 9. wfLocalFile() /var/www/html/mediawiki/includes/specials/SpecialUpload.php:515, referer: *url removed*/mediawiki/index.php/Main_Page
[Tue May 15 15:44:35 2012] [error] [client 96.228.72.32] PHP 10. RepoGroup->getLocalRepo() /var/www/html/mediawiki/includes/GlobalFunctions.php:3600, referer: *url removed*/mediawiki/index.php/Main_Page
[Tue May 15 15:44:35 2012] [error] [client 96.228.72.32] PHP 11. RepoGroup->getRepo() /var/www/html/mediawiki/includes/filerepo/RepoGroup.php:298, referer: *url removed*/mediawiki/index.php/Main_Page
[Tue May 15 15:44:35 2012] [error] [client 96.228.72.32] PHP 12. RepoGroup->initialiseRepos() /var/www/html/mediawiki/includes/filerepo/RepoGroup.php:266, referer: *url removed*/mediawiki/index.php/Main_Page
[Tue May 15 15:44:35 2012] [error] [client 96.228.72.32] PHP 13. RepoGroup->newRepo() /var/www/html/mediawiki/includes/filerepo/RepoGroup.php:336, referer: *url removed*/mediawiki/index.php/Main_Page
[Tue May 15 15:44:35 2012] [error] [client 96.228.72.32] PHP 14. FileRepo->__construct() /var/www/html/mediawiki/includes/filerepo/RepoGroup.php:348, referer: *url removed*/mediawiki/index.php/Main_Page
[Tue May 15 15:44:35 2012] [error] [client 96.228.72.32] PHP 15. FileBackendGroup->get() /var/www/html/mediawiki/includes/filerepo/FileRepo.php:79, referer: *url removed*/mediawiki/index.php/Main_Page
[Tue May 15 15:44:35 2012] [error] [client 96.228.72.32] PHP 16. AutoLoader::autoload() /var/www/html/mediawiki/includes/filerepo/FileRepo.php:0, referer: *url removed*/mediawiki/index.php/Main_Page
[Tue May 15 15:44:35 2012] [error] [client 96.228.72.32] PHP 17. require() /var/www/html/mediawiki/includes/AutoLoader.php:1032, referer: *url removed*/mediawiki/index.php/Main_Page

Here is my config with private info removed.

/*Swift Backend Config*/
require( "$IP/extensions/SwiftCloudFiles/SwiftCloudFiles.php" );
$wgFileBackends[] = array(
'name' => 'localSwift',
'class' => 'SwiftFileBackend',
'wikiId' => 'Development-Wiki',
'lockManager' => 'fsLockManager',
'swiftAuthUrl' => $wgSwiftConf['http://docs.rackspacecloud.com/auth/api/v1.1'],
'swiftUser' => $wgSwiftConf['removed'],
'swiftKey' => $wgSwiftConf['removed'],
#'swiftProxyUser' => 'test:anon',
'shardViaHashLevels' => array(
'local-public' => array( 'levels' => 2, 'base' => 16, 'repeat' => true ),
'local-thumb' => array( 'levels' => 2, 'base' => 16, 'repeat' => true ),
'local-deleted' => array( 'levels' => 2, 'base' => 36, 'repeat' => false )
),
#'readOnly' => 'Temporary planned maintenence for 5 min',
'fileJournal' => array( 'class' => 'DBFileJournal', 'wiki' => false ),
'parallelize' => 'implicit'
);
$wgLocalFileRepo = array(
'class' => 'LocalRepo',
'name' => 'local',
'backend' => 'localSwift',
'scriptDirUrl' => $wgScriptPath,
'scriptExtension' => $wgScriptExtension,
'url' => 'http://localhost/var/www/html/mediawiki/media',
'hashLevels' => 3,
'transformVia404' => true,
'deletedHashLevels' => 3
);
/*End Swift Backend Config*/

I searched the other files for a reference to wgSwiftConf and am not sure where the variable is defined.

Thanks Justin

Jwestyp (talk)‎15:51, 15 May 2012

Oh, $wgSwiftConf is just a variable I made up on my testwikis to separate out the private stuff. You can use any variable name that you want, but you'll have to define it yourself. You still want to have:

 'swiftAuthUrl'       => $wgSwiftConf['authUrl'],
 'swiftUser'          => $wgSwiftConf['user'],
 'swiftKey'           => $wgSwiftConf['key'],

...you just have to define $wgSwiftConf, like: $wgSwiftConf = array( 'authUrl' => 'STUFF', 'user' => 'STUFF', 'key' => 'STUFF' );

The FileBackendStoreOpHandle error makes me wonder if you are fully on master or not.

Aaron (talk)‎20:12, 15 May 2012

I defined the variable. It works fine.

The FileBackendStoreOpHandle error is unknown to me. What do you mean by fully master or not? This is a private server.

I find a reference to it in SwiftFileBackend.php but not anywhere else. Am I missing a file that defines FileBackendStoreOpHandle as a class?

Thanks Justin

Jwestyp (talk)‎20:53, 15 May 2012

I mean the 'master' branch in git. If you clone master and then copy paste some files, you can end up with this kind of breakage. It will be easier when all my patches are merged in master. Then you can just "git reset --hard" and then "git pull" them in.

Aaron (talk)‎01:13, 16 May 2012

Aaron,

I did a reset and pull, and lost the errors. I am no longer logging errors in the php error log. Now I am just getting this on the upload page when I click upload file:

Could not create directory "mwstore://localSwift/local-public/3/37/374".

Here is my configuration

#Swift Backend Config
require( "$IP/extensions/SwiftCloudFiles/SwiftCloudFiles.php" );
require( "/var/www/wikiprivate/wgswiftauth.php" );
$wgFileBackends[] = array(
        'name'               => 'localSwift',
        'class'              => 'SwiftFileBackend',
        'wikiId'             => 'Development-Wiki',
        'lockManager'        => 'fsLockManager',
        'swiftAuthUrl'       => $wgSwiftConf['authUrl'],
        'swiftUser'          => $wgSwiftConf['user'],
        'swiftKey'           => $wgSwiftConf['key'],        
        #'swiftProxyUser'     => 'test:anon',
        'shardViaHashLevels' => array(
                'local-public'  => array( 'levels' => 2, 'base' => 16, 'repeat' => true ),
                'local-thumb'   => array( 'levels' => 2, 'base' => 16, 'repeat' => true ),
                'local-deleted' => array( 'levels' => 2, 'base' => 36, 'repeat' => false )
        ),
        #'readOnly'           => 'Temporary planned maintenence for 5 min',
        'fileJournal'        => array( 'class' => 'DBFileJournal', 'wiki' => false ),
        'parallelize'        => 'implicit',
        'zones'                          => array( 'container' => 'apachewikifiles', 'url' => 'http://c14609052.r52.cf2.rackcdn.com' ),
        'url'                            => 'http://c14609052.r52.cf2.rackcdn.com'
);
$wgLocalFileRepo = array(
        'class'             => 'LocalRepo',
        'name'              => 'local',
        'backend'           => 'localSwift',
        'scriptDirUrl'      => $wgScriptPath,
        'scriptExtension'   => $wgScriptExtension,
        'url'               => 'http://localhost/var/www/html/mediawiki/media',
        'hashLevels'        => 3,
        'transformVia404'   => true,
        'deletedHashLevels' => 3
);
/*End Swift Backend Config*/


You mentioned earlier to define the url for the 'thumb', 'public', and 'temp' to the base CDN URL. I have my private info in the required file at the beginning of this configuration segment. How do I verify this is actually communicating with the cloud container?

On another note, is there a way to donate to your projects?

Thanks again. Justin

Jwestyp (talk)‎01:24, 16 May 2012

'zones' goes in the file repo config.

Aaron (talk)‎04:13, 9 March 2013
 
 
 
 
 
 
 
 
Retrieved from "http://www.mediawiki.org/wiki/Thread:User_talk:Aaron_Schulz/SwiftMedia_using_Rackspace_Cloud_Files#SwiftMedia_using_Rackspace_Cloud_Files_24854"