How-to: debug and trace problems in AWS CodeBuild

Your guide to making troubleshooting CodeBuild much easier!

Brought to you by Gert Leenders, AWS Cloud Solution Architect at DPG Media

Following an “AWS Unless” strategy within DPG Media, it was only a matter of time before moving our builds to AWS CodeBuild and AWS CodePipeline. Migrating our legacy pipelines to AWS CodeBuild turned out to be a pretty straightforward job. For that reason, development teams quickly adopted CodeBuild.

However, in general, there was one complaint about CodeBuild: troubleshooting problems in the buildspec.yml was hard, mainly because access to the remote session was nonexistent.

That was until last July — when AWS announced AWS Session Manager access for AWS CodeBuild. It’s strange, but it seems that the release of this awesome feature went unnoticed!? Maybe that’s because the press release seems to miss the right semantics? It’s definitely a hard post to find — even for Google — if you don’t use the right words. Fingers crossed that this article brings more attention to the feature :-)

By enabling remote access, AWS Session Manager finally brings debug capabilities to AWS CodeBuild. Besides Session Manager access, the new CodeBuild command codebuild-breakpoint is key to this new feature. The extra debug capabilities make troubleshooting CodeBuild less cumbersome. It also ends the nightmare in which a local build would succeed, but the remote Codebuild fails — putting you in a situation where a tedious trial and error is the only way out.

AWS CodeBuild In Practice

1. Add the permission to use AWS Session manager to AWS CodeBuild:

CodeBuildServiceRole:
  Type: AWS::IAM::Role
  Properties:
    AssumeRolePolicyDocument:
      Statement:
        - Effect: Allow
          Principal:
            Service:
              - codebuild.amazonaws.com
          Action:
            - sts:AssumeRole
    Path: /
    Policies:
      - PolicyName: log-access
        PolicyDocument:
          Statement:
            - Effect: Allow
              Action:
                - logs:CreateLogStream
                - logs:PutLogEvents
                - logs:CreateLogGroup
              Resource:
                - !Sub arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/codebuild/*
      - PolicyName: ssm-access
        PolicyDocument:
          Statement:
            - Effect: Allow
              Action:
                - ssmmessages:CreateControlChannel
                - ssmmessages:CreateDataChannel
                - ssmmessages:OpenControlChannel
                - ssmmessages:OpenDataChannel
              Resource: "*"
            - Effect: Allow
              Action:
                - s3:GetEncryptionConfiguration
                - s3:PutObject
              Resource:
                - arn:aws:s3:::your-log-bucket-name
                - arn:aws:s3:::your-log-bucket-name/*      ...

Remark: the CodeBuild policy mentioned above only needs S3 permissions if using Amazon S3 to store your logs. In case it’s missing you get an SSM Session with a stuck prompt without any further feedback. So, if using S3 to store your logs be sure to have that policy right.

2. Add a breakpoint to your buildspec.yml

3. Start a build for debugging

• Start a build using “Advanced Build Overrides”

• Under advanced settings choose “Enable Session Connection”

Start a remote session using the Web Console

Start a remote session using the CLI

1. Grab the Build ID (aka Build Run)

2. Get the sessionTarget using the Build Id

IMHO, currently, the documentation for batch-get-builds falls short. Getting thesessionTarget using the CLI could be tricky if you don’t use the right settings or if you’re not using a recent version of the CLI. Therefore I made a Pull Request to change the documentation to:

Copy the sessionTargetproperty value. Note:sessionTarget is only available if output is json or table. If output is set to text look for DEBUGSESSION instead. If the property is missing from the output then update your CLI to a more recent version.

3. Once you have copied thesessionTargetvalue you can start a new remote session using:

4. Debug your build :-)

You’re all set. To stop a debug session, just exeute $ codebuild-resume.

Until next time!

More info? Check also this link: https://docs.aws.amazon.com/codebuild/latest/userguide/session-manager.html