Sharing additional tags between steps

deployment
docker

#1

I have several services that depend on a hierarchy of Docker images hosted on AWS ECR. I hope to have the images built and pushed as part of my build process, with environment specific image tags. As you can probably see from my demo configuration below, the only workable approach I could think of requires a lot of verbose steps and services, along with environment specific Dockerfiles.

To put it another way, when pushing to the master branch, I want to build and push 2 images: aws_ecr_registry/base:production and aws_ecr_registry/server:production. I want the latter image to extend aws_ecr_registry/base:production. Likewise, when pushing to the dev branch, I want to build and push 2 images: aws_ecr_registry/base:dev and aws_ecr_registry/server:dev, and have the latter image extend the latest build of aws_ecr_registry/base:dev.

I noticed in the steps configuration documentation that it’s only possible to share the name, tag, exclude, and encrypted_dockercfg_path keys between steps. Is there any reason type, and dockercfg_service could not also be shared, reducing some of the repitition in the steps file?

https://documentation.codeship.com/pro/getting-started/steps/#using-codeship-stepsyml

And more generally, can anyone think of another approach that might be a better solution for this workflow?

deploy/Dockerfile-base

FROM python:3
COPY ./requirements.txt /app/
RUN pip install -r requirements.txt

deploy/Dockerfile-server-production

FROM aws_ecr_registry/base:production
COPY ./ /app/
WORKDIR /app
CMD ["python", "run_server.py"]

deploy/Dockerfile-server-dev

FROM aws_ecr_registry/base:dev
COPY ./ /app/
WORKDIR /app
CMD ["python", "run_server.py"]

codeship-services.yml

version: '2'
services:
  data:
    image: alpine
    volumes:
    - ./tmp/:/tmp/
  aws_dockercfg_generator:
    image: codeship/aws-ecr-dockercfg-generator
    add_docker: true
    encrypted_env_file: ./deploy/aws-credentials.env.encrypted
  awsdeployment:
    image: codeship/aws-deployment
    encrypted_env_file: ./deploy/aws-credentials.env.encrypted
    volumes_from:
      - data
  base:
    build:
      image: aws_ecr_registry/base
      dockerfile_path: ./deploy/Dockerfile-base
    cached: true
  server-production:
    build:
      image: aws_ecr_registry/server
      dockerfile_path: ./deploy/Dockerfile-server-production
    cached: true
  server-dev:
    build:
      image: aws_ecr_registry/server
      dockerfile_path: ./deploy/Dockerfile-server-dev
    cached: true
...

codeship-steps.yml

- name: Push images
  type: serial
  tag: "^(master|dev.*)$"
  steps:
  - name: Push base images
    type: serial
    steps:
    - type: push
      service: base
      tag: "master"
      image_name: aws_ecr_registry/base
      image_tag: "master"
      dockercfg_service: aws_dockercfg_generator
      registry: https://aws_ecr_registry
    - type: push
      service: base
      tag: "dev"
      image_name: aws_ecr_registry/base
      image_tag: "dev"
      dockercfg_service: aws_dockercfg_generator
      registry: https://aws_ecr_registry
  - name: Push final images
    type: serial
    steps:
    - type: push
      service: server-production
      tag: "master"
      image_name: aws_ecr_registry/server
      image_tag: "production"
      dockercfg_service: aws_dockercfg_generator
      registry: https://aws_ecr_registry
    - type: push
      service: server-dev
      tag: "dev"
      image_name: aws_ecr_registry/server
      image_tag: "dev"
      dockercfg_service: aws_dockercfg_generator
      registry: https://aws_ecr_registry

#2

Hi Sean,

Just to make sure i’m clear about what you’re asking for, you’re primarily looking to be able to define type, dockercfg_service, registry, and image_name on a step-group level like this:

  - name: Push base images
    type: 
      - serial
      - push
    dockercfg_service: aws_dockercfg_generator
    registry: https://aws_ecr_registry
    image_name: aws_ecr_registry/base
    steps:
    - service: base
      tag: "master"
      image_tag: "master"
    - service: base
      tag: "dev"
      image_tag: "dev"

or are you looking for a different way to deal with the dependency of the images?

Thanks,
Dennis


#3

Hi Sean,

As it turns out, there’s another option that should already work. It will make your file smaller, though there is a caveat around it. You can use yaml merging to have a “template” to merge into other sections of your file, but if you define your mappings at the top of the file as mentioned on the yaml.org page, they will get run as if they were separate steps.

I will see what we can do about making this an easier thing to use, but for now you could use something like the following to re-use parts of your yaml file (disclaimer: i haven’t tested this):

- name: Push images
  type: serial
  tag: "^(master|dev.*)$"
  steps:
  - name: Push base images
    type: serial
    steps:
    - &BASE
      type: push
      dockercfg_service: aws_dockercfg_generator
      registry: https://aws_ecr_registry
      service: base
      image_name: aws_ecr_registry/base
      tag: "master"
      image_tag: "master"
    - <<: *BASE
      tag: "dev"
      image_tag: "dev"
  - name: Push final images
    type: serial
    steps:
    - &SERVER
      <<: *BASE
      image_name: aws_ecr_registry/server
      service: server-production
      tag: "master"
      image_tag: "production"
    - <<: *SERVER
      service: server-dev
      tag: "dev"
      image_tag: "dev"

The idea is that you define a mapping using the &-sign followed by a name. You can later refer to that mapping by the name, and everything in the mapping will be merged into the specified location. If you supply new values for keys already in the mapping, the new values will overwrite those in the mapping.

Let me know if this is any help

Thanks,
Dennis


#4

Thanks @DennisNewel! You did indeed describe what I was trying to accomplish accurately. I’ll take a closer look at those YAML merges you recommended. I also just noticed this morning that the dockercfg_service key itself can already be shared between grouped steps, so that alone eliminates one big source of awkward duplication.

On a broader note, I wanted to just mention I have spotted a few little rough spots in the docs that might be worth addressing.

  • The services reference mentions that the dockerfile_path key is deprecated. (“Use dockerfile instead”). Later on that very same page, all of the snippets are using dockerfile_path.
  • Warnings are raised for volumes with absolute paths, even though they aren’t shared with the host:
volume:
   - /app/

#5

Hi Sean,

Good catch! we’re obviously a bit overzealous with the warnings. We’re going to change that.
Also good catch on the documentation; we’ll get that fixed too

Thanks,
Dennis