Deploying From Bitbucket to WordPress
Of all the projects I’ve worked in the last few years, there’s one that stands out as my favorite: I wrote a WordPress plugin called Great Eagle (Tolkien reference) that allows my team to install and update themes and plugins from our private Bitbucket repos, via the normal wp-admin updates UI.
This plugin has blasted our dev shop through the roof when it comes to development best practices, in ways we never expected or intended. It forces us to use proper version numbers because now we can’t deploy without them. It forces us to store our work in Bitbucket because now we can’t deploy without it. It forces us to use the command line en route to deploying our work (by which I simply mean, git push origin master), which then led to us using phpUnit. Now we can’t deploy unless our tests pass. We’ve arrived at the nirvana of test-driven development, all because we started with the unrelated step of deploying from git.
If this all sounds standard and obvious, great. I’d love a chance to learn from you. If this sounds like exotic rigmarole, guess what? This article is for you.
Disclaimer: My work in this plugin is heavily influenced by, and in some cases plagiarized from, the excellent GitHub Updater plugin, by Andy Fragen. The reason I wrote my own is because we have hundreds of themes and plugins in Bitbucket, and I was having some scale issues when I was auditioning GHU, which have since been addressed. I probably bailed too early, as that plugin has been under active and expert development for years. More than anything, we just wanted a version that was totally under our own maintenance. I’ll be featuring some gists from my plugin, but ultimately I recommend that users defer to GHU because it’s likely a better fit for most people, and also I don’t want to take any momentum from that awesome project.
Prerequisites
My examples demonstrate a multisite install, but that’s not particularly important. This works fine on a single-site install as well. I’m on WordPress version 4.8-alpha-39626 at the moment, but that’s not terribly important either.
Of chief importance is my assumption that all of the themes and plugins in your workplace are each stored in their own Bitbucket repo. This is quite an assumption! No joke: When embarking on this, we hired a company to manually create a repo for each of our themes and plugins. We were using SVN (poorly!) prior to this migration.
How does it work?
There are three(ish) steps:
1) Create a UI for the user to make an API request to Bitbucket and mirror all of our repository data into the WordPress database. Not all the data about each repo, really just the slug name, which we will use as a key for deeper queries.
An alternative would be to build this automatically whenever it’s empty, but for now, I’m happy to have complete control over when such a large series of API requests gets run.
2) Once we have a bit of information mirrored for all of our repos, we can offer a jQuery autocomplete to choose a few repos for data drill-down, where we make several more API calls for each of them, giving us access to deeper information like version number and download url.
Why not just gather all of those details for all repos right away? Because we have hundreds of repos and it takes several calls per repo to grab all of the pertinent information such as, say, the version number. It would probably take 15-30 minutes and over 1,000 API trips.
3) Once we have detailed information about the handful of repos we want to use at the moment, we can determine two important things about them. First, is it installed in WordPress? If not, it will appear in a UI for us to install it. Second, if it is installed, is it on the latest version? If not, it will appear in the normal wp-admin updates UI.
On the off-chance that a repo is not readable (maybe it lacks proper docblocks or naming conventions), it gets omitted from all of these steps. This has only happened to us with a small handful of poorly named plugins, but it can be annoying since changing the plugin folder and file names can deactivate the plugin.
Huh. How does it work, exactly?
Fair question. I’ll explain what the tricky parts were, and share some code from my plugin.
Building the list of repos
The maximum number of repos per API call is 100. That’s just how the Bitbucket API works. We have far more than that in our account, so we have to call Bitbucket in a loop:
<?php
/**
* Store a "shallow" list of repos.
*/
public function set_repo_list() {
...
// Let's get 100 per page, which is the maximum.
$max_pagelen = 100;
....
// Get the first page of repos.
$page = 1;
$call = new LXB_GE_Call( 'api', "repositories/$team", $max_pagelen, $page );
$get = $call -> get();
$out = $get['values'];
// Now we know how many there are in total.
$total = $get['size'];
// How many pages does that make for?
$num_pages = ceil( $total / $max_pagelen );
// Query each subsequent page. We already got the first one.
while( $page < $num_pages ) {
$page++;
$next_call = new LXB_GE_Call( 'api', "repositories/$team", $max_pagelen, $page );
$next_get = $next_call -> get();
$next_repos = $next_get['values'];
$out = array_merge( $out, $next_repos );
}
// Sort the list by most recently updated.
$out = $this -> sort( $out, 'updated_on' );
$this -> repo_list = $out;
}Determining the “main” plugin file
WordPress is very unopinionated when it comes to naming plugins. In most cases, a plugin folder does, in fact, contain exactly one plugin, and that plugin will have a “main” file of sorts, that contains a docblock to convey the plugin name, description, author, and most importantly, the version number. Because that file can be named anything, determining which file is the main plugin file is something of an open question. The approach I’ve taken is to assume that the plugin will conform to some naming conventions we try to use in our work.
<?php
function set_main_file_name() {
// Grab the slug name for this Bitbucket repo.
$slug = $this -> slug;
// Grab the list of file names in this repo.
$file_list = $this -> file_list;
// There's a good chance that there is a file with the same name as the repo.
if( in_array( "$slug.php", $file_list ) ) {
$main_file_name = "$slug.php";
// If not, there's a good chance there's a plugin.php file.
} elseif( in_array( 'plugin.php', $file_list ) ) {
$main_file_name = 'plugin.php';
// If not, it's probably a theme.
} elseif( in_array( 'style.css', $file_list ) && in_array( 'functions.php', $file_list ) ) {
$main_file_name = 'style.css';
// Else, oh well, couldn't find it.
} else {
$error = sprintf( esc_html__( 'Could not identify a main file for repo %s.', 'bucketpress' ), $slug );
$main_file_name = new BP_Error( __CLASS__, __FUNCTION__, __LINE__, func_get_args(), $error );
}
$this -> main_file_name = $main_file_name;
}Determining the version number
Given the main plugin or theme file, we can dig into the docblock in that file in order to determine the version number. Here’s how I do it:
<?php
/**
* Get the value for a docblock line.
*
* @param string $key The key for a docblock line.
* @return string The value for a docblock line.
*/
function get_value_from_docblock( $key ) {
// Grab the contents of the main file.
$main_file_body = $this -> main_file_body;
// Break the file into lines.
$lines = $this -> formatting -> get_lines_from_string( $main_file_body );
// Let's save ourselves some looping and assume the docblock is < 30 lines.
$max_lines = 30;
$i = 0;
foreach( $lines as $line ) {
$i++;
// If the line does not have the key, skip it.
if( ! stristr( $line, $key . ':' ) ) { continue; }
// We found the key!
break;
// Whoops, we made it to the end without finding the key.
if( $i == $max_lines ) { return FALSE; }
}
// Break the line into the key/value pair.
$key_value_pair = explode( ':', $line );
// Remove the key from the line.
array_shift( $key_value_pair );
// Convert the value back into a string.
$out = implode( ':', $line_arr );
$out = trim( $out );
return $out;
}While I’m at it, allow me to applaud php’s helpful version_compare() function, which can parse most common version syntaxes:
/**
* Determine if this asset needs to be updated.
*
* @return boolean Returns TRUE of the local version number
* is lower than the remote version number, else FALSE.
*/
function needs_update() {
$old_version = $this -> old_version;
$new_version = $this -> new_version;
$compare = version_compare( $old_version, $new_version );
if( $compare == -1 ) { return TRUE; }
return FALSE;
}Parsing the readme.txt
We actually don’t use the readme.txt for anything in our plugins, and therefore my Great Eagle plugin does not do much parsing of it either. However, if you wish to incorporate readme information, I’d recommend this library from Ryan McCue for parsing it.
The deal with private repos
Our repos all happen to be private – that’s just the way we do business at the moment. In order to query them, we have to filter in some creds. In this example, I’m doing so via basic auth:
<?php
/**
* Authenticate all of our calls to Bitbucket, so that we can access private repos.
*
* @param array $args The current args for http requests.
* @param string $url The url to which the current http request is going.
* @return array $args, filtered to include BB basic auth.
*/
public function authenticate_http( $args, $url ) {
// Find out the url to Bitbucket.
$call = new LXB_GE_Call( 'web', FALSE );
$bb_url = $call -> get_url();
// If we're not calling a Bitbucket download, don't bother.
if( ! stristr( $url, $bb_url ) ) { return $args; }
if( ! stristr( $url, '.zip' ) ) { return $args; }
// Okay, time to append basic auth to the args.
$creds = $this -> creds;
$args['headers']['Authorization'] = "Basic $creds";
return $args;
}I’m doing this via filtration, rather than passing args to wp_remote_get(), because I need WordPress to be prepared with these creds when it makes its calls during its normal theme and plugin update calls, which now happen to be going to Bitbucket.
It would be better to do Oauth instead of basic auth, but after quite a bit of research, I’ve concluded that there’s not a way to do so. The roadblock is because raw file content is actually not part of the Bitbucket API at this point, it’s just hosted on their website like any other static asset, such as this public test theme for example (it’s public for demo purposes, but again, if it were private, you could access it via basic auth). I do have this humble feature request to show for my efforts. As a security measure, I recommend using Bitbucket’s new application passwords feature to create an account specifically and only for scripted calls like this, where that app password only has read access. So, to be clear, with basic auth there is a universe (maybe this one) in which a packet-sniffing foe is reading our plugin files. I’m okay with that, at least for the moment.
Adding our repos to the update queue
If there’s one key to getting a foothold in this whole process, it’s found in the wp_update_plugins() function. That’s a huge function the core uses to loop through all of the installed plugins, determine which ones have an update available, and save the result to a transient. The key is that the transient is then exposed for filtering, which is exactly what my plugin does:
<?php
add_filter( 'pre_set_site_transient_update_plugins', array( $this, 'set_plugin_transient' ) );
/**
* Inject our updates into core's list of updates.
*
* @param array $transient The existing list of assets that need an update.
* @return The list of assets that need an update, filtered.
*/
public function set_plugin_transient( $transient ) {
if( ! is_array( $this -> assets_to_update ) ) { return $transient; }
foreach( $this -> assets_to_update as $asset ) {
if( empty( $asset -> transient_key ) ) { continue; }
if( ! $asset -> transient_content ) { continue; }
$transient -> response[ $asset -> transient_key ] = $asset -> transient_content;
}
return $transient;
}It took me forever to break into this, and it took me months and months to write this plugin. You should probably just use GHU instead. It’s pretty damn similar. That said, if you want to tweak some things and you don’t like running 3rd party plugins, maybe the above code will help you write your own.
So what’s the point, exactly?
The point is not so much how to build your own git deployer plugin, or which existing one you should use. You can figure that stuff out yourself. The really interesting thing is to look at what happened to us when we started deploying from git. Some of the side effects were profoundly surprising and positive.
So long, FTP
FTP stinks for so many reasons.
- FTP access is an attack vector.
- No easy way to track or revert changes.
- No easy way to allow multiple people to work on the same project at the same time.
- Human error. It pretty easy to mis-drag-n-drop, leading to a WSOD or worse.
- I never expected this, but it’s apparent when updating a plugin across many installs, that this git method is much faster than FTP.
With a git deployment system like the one I’m advocating and explaining in this article, you can go so far as to disable all FTP access to your production environment. Seriously: ou won’t need it.
Hello proper versioning
I recommend using a git deploy tool that uses docblocks in order to determine the version number, and uses the version number to determine if the theme or plugin is in need of an update. This forces your team to use proper version numbers, which is a nice first step down to the road from crankin’ out themes to maturely managing a long-living codebase.
I’m so stoked about unit testing now
If you’re not unit testing, you probably know you should be. With git deployment, it can be both automatic and required.
We use the command line to move our work from our local MAMP to Bitbucket, as in, git push origin master. Each of our plugins carries a Grunt task to execute our phpUnit tests upon git pre-commit, and if the tests fail, so does the commit.
We bind Grunt to our commit using GitHooks and we execute our unit tests via Exec. If the tests fail, so does the deployment.
There’s no way to sidestep the tests because there’s no way to sidestep git for deploying!
Rollbacks
There are no rollbacks per se with this method. Rather, you only roll forward. Whatever you want to fix or restore, get it in master, boost the version number, push, and deploy.
Staffing
This kind of maturation can have business-wide ramifications. Picture this: You have non-dev support folks on the front lines, trying to debug a problem for a client. In the past, they would have had to place this request in a dev ticket queue, while the customer waits hours or days for a resolution. Not anymore. Now, your front-line support agent can navigate to network admin and see that on this environment the plugin in question is outdated. They’re free to update the plugin right away via the normal wp-admin interface. The ticket is resolved by front-line support with no dev team involvement. Perhaps those front-line folks cost less than developers, or perhaps they carry a deep skill set in account management. Either way, you no longer have to open a dev ticket to deploy updates to your in-house plugins. Pivotal.
Rise of the machines
Before this process, we were very much an ordinary dev shop churning out themes and plugins for clients, cowboy-FTPing, not versioning our work. Why? Because we were lazy. Why? Because we were human. We’re no longer lazy because we are no longer human, at least when deploying. We’re a command line script and a series of API requests, and no matter how lazy we are, we have to follow proper deployment practices because we nuked the FTP creds for our developers! On top of all that, it’s a faster way to deploy, free from any click-n-drag misfires.
Can you get on board with this overnight? Okay, no. It’s a long and expensive process, and it might not be for you, but honestly it probably is. I think there are about 1,000 dev shops out there that should give careful consideration to this.
Deploying From Bitbucket to WordPress is a post from CSS-Tricks
