Fast dependency installation (gem, brew, pip, npm)

At Wikia we write a lot of scripts and use a lot of tools to manage our App Farm. Anyone in the team should be able to use the scripts in the same way, so we created a little script that installs all the dependencies.

This file initially looked like that:

#!/bin/bash
sudo easy_install pip

sudo pip install invoke
sudo pip install termcolor
[..]
sudo gem install cocoapods -V
sudo gem install shenzhen -V
[...]

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
brew update

brew install git
brew install wget
[...]
sudo npm install chromedriver -g
sudo npm install ios-sim -g
[...]

This worked initially, but as the file grew, we had to rerun the script on every computer to download the missing dependencies and then the problems started to rise.

Main problems:

• Reinstalling of things you already have is taking a long time
• gem and npm reinstall all the packages by default. They don't check if packages are installed already
• If something failed to install, the information got lost in all the logs

So the script was rewritten in Python and learned some new tricks:

check_call(['which pip || sudo easy_install pip'], shell=True)
check_call(['which brew || ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"'],
           shell=True)

For commands installing binaries we can check if which has some output, meaning this binary is already installed.

def pip(package, pip_list=check_output(['pip', 'list'])):
    print "pip " + package
    package_with_space = package + " "
    if package_with_space not in pip_list:
        cmd = ['sudo', 'pip', 'install', package, '--upgrade']
        check_call(cmd)

Installing pip packages can use pip list and first check if the packages are installed. Using pip list as a default argument we only run it once and then check against it in subsequent calls.

def gem(package, gem_list=check_output(['gem', 'list'])):
    print "gem " + package
    package_with_space = package + " "
    if package_with_space not in gem_list:
        cmd = ['sudo', 'gem', 'install', package, '-V', '--no-ri', '--no-rdoc']
        check_call(cmd)

The same idea as with pip, but we're using gem list. We also add --no-ri and --no-rdoc as we don't care about documentation and such.

def brew(package):
    print "brew " + package
    cmd = ['brew', 'install', package]
    try:
        check_call(cmd)
    except CalledProcessError:
        pass

Brew checks if the package is already installed, but sometimes the error codes are inconsistent. That's why we try: except: and ignore the result.

def npm(package):
    print "npm " + package
    try:
        check_call(['which', package])
    except CalledProcessError:
        cmd = ['sudo', 'npm', 'install', package, '-g']
        check_call(cmd)

We only use npm for binaries, not JavaScript libraries, so we use the which check.

And the rest of the file looks pretty similar:

pip('invoke')
pip('termcolor')
[...]
gem('cocoapods')
gem('shenzhen')
[...]
brew('git')
brew('wget')
[...]
npm('chromedriver')
npm('ios-sim')
[...]

With this solution we only check what's needed and install that. We crash as early as possible to debug some installation.

The only problem is that we don't check version numbers, but we haven't had any problems with that yet. If any problem occurs we could add version checking to the script.