Deploying git branches from GitLab using Slack commands

tl;dr

Deploy a branch called ‘staging’ to a server called ‘staging’ with a Slack slash command:

/gitlab deploy staging to staging

Overview

If you’re using GitLab as your CI build system, and Slack for your team communications, you can invoke deployment commands directly from Slack.

Deploys are defined as jobs in specific environments in GitLab, so that different branches (or tags) can be deployed to different servers.

The format of the command is slightly clumsy, because of the way that the GitLab integrations work – and only a small number of commands are supported, ‘deploy’ being one of them. (See others here).

Define deploy jobs for branches

The deploy stage for a GitLab build is defined in the ‘.gitlab-ci.yml’ file. The file might look something like this (simplified):

stages:
  - test
  - deploy

test:
  stage: test
  environment: $CI_COMMIT_REF_NAME
  script:
    - ./bin/test.sh

deploy:
  stage: deploy
  environment: $CI_COMMIT_REF_NAME
  when: manual
  script:
    - ./bin/deploy_to_server.sh $CI_COMMIT_REF_NAME

Note that the stages both have an environment defined – this is what GitLab uses to create entries in the environments list for the project, which is where it keeps track of which commits have been deployed to which targets.

The environments here are named with the $CI_COMMIT_REF_NAME variable, which is set to the name of the branch that is building – so the build will create an environment for every branch, with jobs called ‘test’ and ‘deploy’.

You can see all the environments for a project – go to ‘CI/CD’ -> ‘Environments’.

The ‘test’ job will run automatically during the build, but the ‘deploy’ task won’t, because it’s labelled as ‘manual’. You could run the ‘deploy’ task from the GitLab UI (a button labelled ‘Manual’ on the Jobs page) once a successful ‘test’ job has finished.

That ‘deploy’ job is the one that we’re going to invoke from Slack.

More about invoking deploys

The format for invoking a deploy remotely will be

deploy <source-environment> to <target-environment>

The way that GitLab invokes a deploy is slightly convoluted – when triggered, GitLab will look for the ‘source-environment’ name in the environments which has had a successful build, and identify a manual deploy step in that same build with environment ‘target-environment’ – which it then invokes.

Setup GitLab/Slack integration

Now that the build has a deploy job that can be manually invoked, you can set up the integration with Slack.

Starting in GitLab, go to the ‘Slack slack commands’ integration page (Settings->Integrations->Slack slash commands).

That will give you instructions for configuring the slash command over in Slack, and a link to take you to the setup page in Slack:

You won’t have a ‘Token’ yet – that will be supplied by Slack once you’ve setup the slash command.

Keep this page open, ready to paste in the Token.

Click the “Add a slash command” link to go to the Slack setup page, and choose a name for the slash command (e.g. ‘/gitlab’):

After clicking ‘Add Slash Command Integration’, you’ll get the configuration page – paste in the URL for the command, taken from the page of instructions in GitLab:

Keep this page open, but take the ‘Token’ from here, and paste it into the ‘Token’ field on the configuration page for the integration, back in GitLab, and save the settings there.

Back in the Slash command setup, give it some Autocomplete Help Text, which will appear in Slack when someone starts typing with a ‘/’. Save the integration.

Make sure that your build has run successfully at least once for any branch that you’re trying to deploy, otherwise the environment won’t have been created yet, and the deploy job won’t be found.

Note

This is similar to my previous post about deploying using Jenkins.

In this case, though, GitLab provides more integrated support for Slack slash commands which trigger deploys, so the integration method is different..

Deploy straight from Slack using Jenkins and Capistrano

Deploy code by typing ‘/deploy_code_to_dev’ in a Slack channel.

Overview

Slack allows you to define slash commands, which will do a POST to a URL that you specify – so you can type ‘/deploy_code_to_dev’ in a Slack channel, and a POST request will trigger a build on Jenkins.

The Jenkins job is configured to run a Capistrano task as the build step – like ‘cap dev deploy’.

The job can be parameterised, so that the task name and target environment can be specified in the trigger URL.

Jenkins

First, install the “Slack Notification” plugins in Jenkins. This allows feedback from Jenkins into a Slack channel to let you know whether the task was successful.

Set up a build project in Jenkins, called something like ‘capistrano-build’. Flag it as parameterised, with ENVIRONMENT, TASK and PARAMS as parameters – these will be used in the capistrano call later.

Specify the git repo where your capistrano tasks are (note that we’re specifying the branch as $ENVIRONMENT, so that we can select a branch like dev, staging etc when we invoke the job):

Generate an authentication token to include with the call to the URL that triggers the build. I used a hex-encoded token to avoid any uncertainty about the encoding of URL parameters:

openssl rand -hex 32

Add a section for triggering the build remotely, using the token:

Set the build environment with a build number:

And configure the build step – this is where we’ll actually invoke the Capistrano command:

Now you can trigger the build by POSTing to the URL:

curl -X POST "https://build.knowmalaria.co.uk/job/capistrano-build/buildWithParameters?token=b069c41c2d82e9641602c198da09e8274f31032e1b9841e7528e39cf3c30d15d&ENVIRONMENT=dev&TASK=deploy"

Set the post-build actions to send some details to Slack – you’ll need to get the integration token from Slack

Slack

Now to trigger the build, we’ll add a Slash Command in Slack – these are user-defined commands that you can invoke by typing ‘/’ and your command name, and they can do some useful stuff like calling external URLs.

Add a new slash command, and configure it with a name like ‘/deploy_code_to_dev’ :

Set the Request URL to be the one that we used above in the curl request

https://build.knowmalaria.co.uk/job/capistrano-build/buildWithParameters?token=b069c41c2d82e9641602c198da09e8274f31032e1b9841e7528e39cf3c30d15d&ENVIRONMENT=dev&TASK=deploy

Now just type ‘/deploy_code_to_dev’ from a Slack channel, and the slash command will post to the Jenkins build trigger. Results from the build will be posted back into the Slack channel that you configured.

To call a task that needs some extra parameters passed in, you can set up a slack command like:

https://build.knowmalaria.co.uk/job/capistrano-build/buildWithParameters?token=b069c41c2d82e9641602c198da09e8274f31032e1b9841e7528e39cf3c30d15d&ENVIRONMENT=dev&TASK=remoterake:invoke&PARAMS=db:backup

Jenkins builds of GitLab branches stopped working due to security patch

There have been some security patches to Jenkins recently, which have stopped some plugins from working – in our case, the Gitlab Merge Request Builder Plugin.

Parameters that used to get passed in as part of a trigger to build a branch stopped being passed through.

Errors in the Jenkins build console output looked like this – you can see that ${gitlabSourceBranch} is not being replaced properly with the branch name:

git config remote.refs/remotes/origin/${gitlabSourceBranch}.url git@git.dev53.co.uk:specialproject/specialrepo.git # timeout=10

And the log file had lots of entries like this:

May 17, 2016 9:25:33 AM hudson.model.ParametersAction filter
WARNING: Skipped parameter `gitlabSourceBranch` as it is undefined on `knowmalaria-merge`. Set `-Dhudson.model.ParametersAction.keepUndefinedParameters`=true to allow undefined parameters to be injected as environment variables or `-Dhudson.model.ParametersAction.safeParameters=[comma-separated list]` to whitelist specific parameter names, even though it represents a security breach

To get the branches building again, we had to update the parameters that the Jenkins server is started with.

In /etc/init.d/jenkins, set up a list of the parameters that the build will need :


# allow parameters to be passed in to gitlab builds
ALLOW_GITLAB_PARAMETERS="-Dhudson.model.ParametersAction.safeParameters=gitlabMergeRequestIid,gitlabSourceRepository,gitlabMergeRequestId,gitlabTargetBranch,gitlabSourceBranch,gitlabDescription,gitlabSourceName"

and pass those parameters to the process at startup:


# --user in daemon doesn't prepare environment variables like HOME, USER, LOGNAME or USERNAME,
# so we let su do so for us now
$SU -l $JENKINS_USER --shell=/bin/bash -c "$DAEMON $DAEMON_ARGS -- $JAVA $JAVA_ARGS $ALLOW_GITLAB_PARAMETERS -jar $JENKINS_WAR $JENKINS_ARGS" || return 2

Then restart your Jenkins server:

sudo service jenkins restart

You may have a different list of parameters that need to be passed to the build – check the ‘parameters’ page for one of your previous builds.

See more description of the original security flaw here:

http://www.infoworld.com/article/3070093/security/jenkins-security-patches-could-break-plug-ins.html
https://wiki.jenkins-ci.org/display/SECURITY/Jenkins+Security+Advisory+2016-05-11

Search and replace with regex in Sublime Text

One of the things that we do a lot in our Ruby on Rails project is replace old-style Ruby hashes (“hash rockets”) with new style, more compact hashes.

So this

{ :key => "value" }

becomes this

{ key: "value" }

In Sublime Text, you can use a regex in find and replace – like this:

replace-rockets

Select the “regex” option (far left), and use the “…” button (middle right) to select “Add Current File” as the scope. Then use the “Replace” button (bottom left) – the prompt will tell you how many instances are going to be replaced.

Of course, if you’re feeling brave, you could do the same find and replace across ALL your files..

There’s some more about the new style here : http://blog.pluralsight.com/rip-ruby-hash-rocket-syntax

Two-Legged OAuth with the Google Drive API in Ruby

Google are discontinuing support for the Documents List API, and moving to the Drive API.

The old API supported authentication with a username and password, but that’s not allowed in the new API. Instead, you need to use OAuth for access.

If you want to have server-to-server authentication, without user interaction, you need a “two-legged” OAuth process, where a token is obtained with an encrypted request and then used for future service requests.

Continue reading “Two-Legged OAuth with the Google Drive API in Ruby”

from_sentence : the opposite of Rails to_sentence

I like the Rails to_sentence method on String class, which converts an array to a comma-separated sentence where the last element is joined by a connector word.

It makes it easy to take a list of names, for example, and make them human-readable, without having to fiddle with join words and last-item-is-special-case stuff:

['one', 'two'].to_sentence          # => "one and two"
['one', 'two', 'three'].to_sentence # => "one, two, and three"

Continue reading “from_sentence : the opposite of Rails to_sentence”