While we strive to be as efficient as possible, there are times when things don't go smoothly, causing stress and headaches for our user base and the community members who test and prepare our releases. This last week has been an example of some rough times for our releases, and though no single person or group is to blame for any of the issues that we've experienced with these last releases, they do highlight areas where our teams and workflows can improve.

2.5.26/3.2.6/3.3.5 Upgrade Bug

In yesterday's 2.5.26, 3.2.6, and 3.3.5 releases, a bug which impacts a user's ability to update their sites via the Joomla! Update component was introduced. The code powering this update system had been updated in response to a security vulnerability which had been reported just after last week's releases. As this code is derived from Akeeba Backup's Admin Tools, and used in both Joomla! and Akeeba products, a coordinated release was made by the two teams with this updated code. Shortly after yesterday's release, we were contacted by the Akeeba team and alerted to a potential upgrade issue introduced with the security fix, which after thorough testing was confirmed to affect both team's products. A patch was quickly supplied to us for inclusion into the next release, at which point I, acting as the Release Lead for the current Joomla! releases, decided we would push out as soon as possible. While preparing the next release, we needed to ensure that we would be able to provide proper documentation for all affected users as the upgrade path would not be the typical one-click update. We also elected to include a fallback upgrade mechanism for the update component itself so that if a situation such as this were to happen again, we would be able to provide a patch for the update component specifically. To ensure it's clear to all users, this bug in itself did not impede the normal functionality of a Joomla! site; the only impaired function is the update component itself and only the actual code which performs the updates. There is a fallback mechanism for updates if it does not work properly which would require users to update their sites via the Extension Manager, similar to how updates of Joomla! 1.6 and 1.7 were handled.

Regressions in 3.3 Releases

Another issue which has affected our recent 3.3 releases has been the introduction of several regressions into the code, which while relatively minor in many cases, caused disruptions on a site's general use. Two examples of recent regressions are in the e-mail cloaking plugin and component pagination. In the case of the e-mail cloaking plugin, a change had been made to the code which rendered the cloaked e-mail addresses in a <div> element which incorrectly modified the output of sites and was even seen on joomla.org sites. With the component pagination, the issue was introduced when a change in the database query logic was made in order to use the declared API more accurately, but the change was not adapted for in JModelList in the methods it uses to calculate the pagination. Though this bug could have been caught if it were applied to an environment with an appropriate amount of content, many community members who are active in our development workflow via the Joomla! Bug Squad are not testing issues against production style websites but rather bare installations which contain very little content or few extensions used in their typical site configurations.

Release Timing

Our typical release schedule aims to produce general maintenance releases on a six-to-eight week schedule. Unfortunately, this schedule is not always within our control and can vary greatly for various reasons, one of which involves security issues. As noted in the above linked security vulnerability, the security issue was not reported to Joomla! until after last week's releases had been published. Had the report come one or two days sooner, it is highly likely that those releases would have been delayed in order to include that fix. In the case of security issues, we aim to provide them to our community members in a timely manner and do our best to not inconvenience anyone with several quick releases in succession.

Lessons Learned

There is a lot to be taken from how things have gone with regards to the releases and the issues that have been included with them going forward. The Production Leadership Team will be looking for ways to improve our release procedures and testing coverage (through both user interaction and automated tests), but it is important to note that we cannot guarantee for a flawless system. Mistakes will happen, it's important to minimize them and learn from them when they do occur.