Webmasters Gallery

The following is a guest post by Noam Elboim, a developer at myheritage.com. Noam has dove head-first into Yeoman – from not knowing anything or understanding how it could help at work, to building a custom generator just for them. If you think a tool to help you scaffold out new projects to your liking could help you, check out Noam’s journey!

On my first day at MyHeritage, I remember this exchange with the web development team leader:

Team Lead: You have some experience working with Node.js modules like Yeoman, right?

Me: Yeah, I know a bit.

Team Lead: I think using Yeoman here would save us a bunch of time on creating new pages. Do you think you can manage something like that?

Well, that was my first day; I barely knew where my seat was. However, I remember clearly what I was thinking, and I didn’t understand how Yeoman could help here.

For me, and for many others, Yeoman was a tool to generate a full working application, from zero to hero, in no more than two seconds. It was not for adding parts to an existing application.

I was completely wrong. Allow me to share with you how we created a Node.js module at MyHeritage that extends the functionality of Yeoman to save us tons of time on everyday tasks.

The Problem

Creating new pages in MyHeritage used to be a challenging task. Each new page that was created demanded about two days of work to prepare everything, including asset management, a server-side component and a client-side bootstrap of an application. In short: a large number of files.

Most developers do not create new pages on a regular basis. Before the real work could even begin, a lot of time was required just to understand how to create the new page. Then, once the page was created and the work was “done”, there were still hours of time remaining to debug anything that didn’t work.

Two days was just way too much time to invest in a simple task like that.

A Ray of Hope

A couple of months after my conversation with our team lead, the toll of creating new pages had become significantly greater and we couldn’t continue down that path. We knew that an immediate solution was necessary and that it must be simple, easy to maintain, and most obviously, very fast to use.

First, we broke down the problems associated with creating a new page. The process required a lot of steps, including routing configurations and translations services, running terminal commands for creating certain files (like A/B tests or Sass files) and even running some Gulp tasks (like compiling Sass or creating sprites).

At this point we were certain that extending Yeoman would serve us best, but we still had to figure out how it could satisfy all of these needs while still maintaining its core simplicity and ease of use.

A Few Words on Yeoman

Yeoman helps you to kickstart new projects, prescribing best practices and tools to help you stay productive.

That is the first line on Yeoman’s website and it’s 100% right.

It’s a Node.js module with thousands of generators to choose from. All generators basically work the same — first the user is asked a set of questions in user interfaces that are based on Inquirer.js. Then, using the user’s input, Yeoman generates the proper new files.

Why Yeoman

We had a range of options from using a full working generator to building something from scratch. There are a couple of reasons we chose Yeoman:

First of all, simplicity. Yeoman features a clear API, a generator’s generator for getting started, and serious ease-of-use as a Node.JS module.

Second, maintainability. Yeoman is being used by thousands of people worldwide and it’s based on the Backbone.js Model which allows it to be extended in a way that is easy to understand.

Finally, speed. Simply put: speed is a given with Yeoman.

Our solution

We wanted to add another stage between the two halves of Yeoman’s flow to extend the data that we collect from the user through an ordered set of functions. For example, we may want the ability to get snake_case from CamelCase, or to capture the output of a command executed in the terminal. We require this stage to be configurable via a JSON file.

We break every function down to an object that includes the name of the function to execute, the arguments, and where to store the output. We use Node.JS so that those functions run asynchronously and return promises. When all promises resolve, we generate the files. Here is an example of how the Generator class uses the Yeoman-generator module, which is named “generators”:

var generators = require('yeoman-generator'),
    services, prompts, actions, preActions, mainActions,
    _this;
  
module.exports = Generator;

/**
 * Generator object for inheritance of Yeoman.Base with different services, prompts and actions files.
 * To use the generator in you Yeoman generator, create a new Generator instance with your local files and export the return value of Generator.getInstance()
 * @param externalService - local generator services.js file
 * @param externalPrompts - local generator prompts.json file
 * @param externalActions - local generator actions.json file
 * @constructor
 */
function Generator (externalService, externalPrompts, externalActions) {
  services = externalService;
  prompts = externalPrompts;
  actions = externalActions;
  preActions  = actions.pre;
  mainActions = actions.main;
}

/**
 * Get instance will create extension to the Yeoman Base with your local dependencies
 * @returns {Object} Yeoman Base extended object
 */
Generator.prototype.getInstance = function () {
  return generators.Base.extend({
    initializing: function () {
      _this = this;
      this.conflicter.force = true; // don't prompt when overriding files
    }, 
    prompting: function () { /* ... */  },
    writing:  function () { /* ... */  },
    end:  function () { /* ... */  }
  });
};

How to Build a Generator

Once the base is ready, creating a new generator is a piece of cake. We just need to create a new folder containing 4 files and a folder of templates in it. The name of the folder is the name of the generator. The 4 files are:

1. Prompts — a JSON file full of objects which map prompted questions to the stored data.

   [
     {
       "type": "input",
       "name": "name",
       "message": "Name your generator:",
       "default": "newGenerator"
     },
     {
       "type": "list",
       "name": "commonServices",
       "message": "Want some common services:",
       "choices": ["yes", "no"]
     }
   ]

   In this example, the first item will ask the user to “Name your generator:” with the default answer of “newGenerator”. It stores the answer in the data object by the field “name”.

   The second item asks a multi-choice question with the options “yes” and “no”. The default is the first choice, “yes”.

2. Services — a JS file with all the functions we need for extending the data from the user.

   var exec  = require('child_process').exec,
     chalk = require('chalk'),
     Q     = require('q');
   
   module.exports.generateSprite = generateSprite;
   
   /**
    * Run gulp sprites in our new sprite folder
    * @param {String} name - the name of the project
    * @return {Object | Promise}
    */
   function generateSprite (name) {
     return Q.Promise(function (resolve, reject) {
       console.info(chalk.green("Compiling new sprite files..."));
       if (name) {
         exec('gulp sprites --folder ' + name, function () {
           resolve();
         });
       }
       else reject("Missing name in generateSprite");
     });
   }

   This example explains a service called “generateSprite”, which will run a terminal command that is already configured for the project. This specific script creates a sprite folder. If the function manages to run the command successfully, it will resolve a promise that will mark the operation as successful. Templates can be generated based on the outcome.

3. Actions — a JSON file that maps of all the functions that we need to execute before generating files — the “pre” section — and another mapping of template files and their final location where the files will be generated, which is the “main” section.

   {
     "pre": [
       {
         "dependencies": ["name"],
         "arguments": ["name"],
         "output": "spriteFolder",
         "action": "generateSprite"
       }
     ],
     "main": [
       {
         "dependencies": ["name"],
         "optionalDependencies": ["spriteFolder"],
         "templatePath": "output_file.js",
         "destinationPath": "./sprites/<%= name %>output_file.js"
       }
     ]
   }

   In the “pre” section there is a task which will run only if the field “name” is defined or true (based on the requirements defined in the “dependencies” array of fields). The arguments to provide in the service function inside the “args” object by the “arguments” array. The output of the function is saved in the location specified by the “output” field. The name of the function is defined by the “action” field. In this implementation you can provide only one function, but it could easily support multiple functions by using an array in “action”.

   In the “main” section there is a task with similar functionality. If a field in the “dependencies” is missing or faulty, then the file will not be generated. However, this is not the case with “optionalDependencies” which are, in fact, optional. The fields exposed to the templating engine are defined in the “dependencies” and “optionalDependencies”.

   The “templatePath” has the path to the template file and the “destinationPath” is the destination for the file. “destinationPath” can use data from the “pre” section as we see here with the “name” field ().

   Notice that we use EJS templating language. Yeoman actually supports additional templating engines beside EJS, like handlebars.

4. Index — a JS file. Yeoman requires that each generator has an index file. The index links the 3 other files and inherits from the base. Basically, it creates an instance of the base with separated prompts, services and actions.

   var Generator   = require('../generator'),
       services    = require('./services'),
       prompts     = require('./prompts'),
       actions     = require('./actions');
       
   var generator = new Generator(services, prompts, actions);
       
   module.exports = generator.getInstance();

   As you can see in this example, it creates the instance and exports it to work with Yeoman.

5. Templates — easy as it sounds: just templates to create files.

   <h1>My first template in <%= name %>!</h1>
   <% if (spriteFolder) { %>
     I even managed to create sprites for it!
   <% } %>

   Note that the balance between extending the data and creating templates is completely dependant upon the kind of generator you would like to create. For example, a generator that only creates new files with no sophisticated data may not require extension. On the contrary, we might want a generator that just runs a set of system commands, which will have no templates.

   Both cases are possible, and even likely, for some uses. The ability to do both, or just either one, exhibits how flexible it is to use and create a generator for any need.

What We Have Gained

Since we solved our problem by creating the generator for new pages, we have been able to use Yeoman to create more generators, including one that creates a new part in our backend API and another that creates a full working Angular application (with unit tests and all).

As we use Yeoman more and more every day, the pros for us are quite obvious, but there are some cons that we have noticed and you should keep them in mind.

Pros

• Fast — Creating a new page now takes us about 1 minute.
• Best practices and conventions — Yeoman forces developers to use best practices and helps ensure that conventions are followed and learned by our team. A team can make sure every page made will use the same conventions, no matter which developer created it. Do you want to make sure every page uses some new and shiny service? Just add it in the “new page” generator and you are done.
• Cleaner code environment — Because we removed the ability for developers to copy/paste sections of code that they don’t completely understand, bugs tend to appear far less frequently and the unnecessary duplication of code is much less of a problem.

Cons

• Maintenance — This is the hard part. Best practices tend to evolve and change. You always have to make sure that the generator is in the forefront of the codebase, which requires investment. It’s not a “hit and run” project — you will have to update it as your codebase and technology evolve.
• Fit for all uses — This is more of a challenge than a con, but failure to implement a generator properly could lead to developers who don’t use the generator you have worked long and hard to build. It is important for the generator to be generic but still save the developer a significant amount of time on repetitive and frustrating tasks. For example, instead of implementing a function, put a TODO with some explanation.

In Conclusion

Our generators are very specific to our projects, so those aren’t open source, but there are lots of more widely useful generators you can use or reference as you build your own, like this one for Angular 2.

Using Yeoman really made our everyday work easier and better after we created generators for a lot of repetitive tasks. We even made a generators’ generator! But it also brought to our lives unexpected challenges that we cannot ignore. There is a balance between what should be automated and what needs to be done manually by a developer and it’s a matter of figuring this out for each project and need… But that’s a topic for another time.

How Using Yeoman Changed the Way We Work is a post from CSS-Tricks