Webmasters Gallery

Average approval time for moderated packages is currently under 10 hours!

In my last post, I talked about things we were implementing or getting ready to implement to really help out with the process of moderation. Those things are:

• The validator – checks the quality of the package
• The verifier – tests the package install/uninstall and provides logs
• The cleaner – provides reminders and closes packages under review when they have gone stale.

The Cleanup Service

We’ve created a cleanup service, known as the cleaner that went into production recently.

• It looks for packages under review that have gone stale – defined as 20 or more days since last review and no progress
• Sends a notice/reminder that the package is waiting for the maintainer to fix something and that if another 15 days goes by with no progress, the package will automatically be rejected.
• 15 days later if no progress is made, it automatically rejects packages with a nice message about how to pick things back up later when the maintainer is ready.

Current Backlog

We’ve found that with all of this automation in place, the moderation backlog was quickly reduced and will continue to be manageable.

A visual comparison:

Note the improvements all around! The most important numbers to key in on are the first 3, they represent a waiting for reviewer to do something status. With the validator and verifier in place, moderation is much faster and more accurate, and the validator has increased package quality all around with its review!

The waiting for maintainer (927 in the picture above) represents the bulk of the total number of packages under moderation currently. These are packages that require an action on the part of the maintainer to actively move the package to approved. This is also where the clean up service comes in.

The cleaner sent 800+ reminders two days ago. If there is no response by early February on those packages, the waiting for maintainer status will drop significantly as those packages will automatically be rejected. Some of those packages have been waiting for maintainer action for over a year and are likely abandoned. If you are a maintainer and you have not been getting emails from the site, you should log in now and make sure your email address is receiving emails and that the messages are not going to your spam folder. A rejected package version is reversible, the moderators can put it back to submitted at any time when a maintainer is ready to work on moving the package towards approval again.

Statistics

This is where it really starts to get exciting.

Some statistics:

• Around 30 minutes after a package is submitted the validator runs.
• Within 1-2 hours the verifier has finished testing the package and posts results.
• Typical human review wait time after a package is deemed good is less than a day now.

We’re starting to build statistics on average time to approval for packages that go through moderation that will be visible on the site. Running some statistics by hand, we’ve approved 236 packages that have been created since January 1st, the average final good package (meaning that it was the last time someone submitted fixes to the package) to approval time has been 15 hours. There are some packages that drove that up due to fixing some things in our verifier and rerunning the tests. If I change to only looking at packages since those fixes have went in on the 10th, that is 104 packages with an average approval within 7 hours!