Recently I was using AWS LocalStack (which uses Python and Docker) to run various tests against DynamoDB. I was running into behaviour that seemed different from the ‘true’ AWS services, which kind of defeats the purpose of having localstack. in the first place.

The log files were a little cryptic to me as I am a more ‘hands on’ learner, so I decided to set up some debugging. It’s not as straightforward as doing the equivalent simple Java setup, so I thought I’d document what I did in case anyone else finds it useful.

I had to dig around various guides to find out what was going on – so I will distill it here.

Overview of approach

From all the readings I could find the general approach is a follows:

• Initiate the connection from your host. (This is the opposite of the most common Java approach, where you expose a port on your server and then try and connect from your host).
• Ensure that your docker image has a python module capable of debugging. (I used pydevd-pycharm).
• On the container you run, update the file you wish to debug with pycharm instrumentation code.

After a lot of experimentation I could get this to work, but there were several annoyances:

• Changing every file ad-hoc that you might want to debug (as most guides including this one suggest) is too time consuming to be useful. I’m after an oversight of code within a framework, after all.
• In order to pick up the changes to include pydevd-pycharm, you need to launch a new python process.
• It appears that for my container to call back to my host, I needed to use my host IP (as specified by en0) rather than my host name. If another developer wants to use this approach, they need to use their IP.

All of this lends itself to a setup script, which can then be used to manipulate my localstack environment BEFORE the container runs.

Overriding the Docker Entrypoint

Here’s a snippet from a local docker compose file that made use of localstack as follows:

version: '3'
services:
  aws-localstack:
    image: localstack/localstack:0.0@sha256:2740b5509173e0efbd509bdd949217f42c97e1ab1f5b354430fdf659c2b9a152
    entrypoint: '/local-stack-debug.sh'
    ports:
      - '4569:4569'
      - '4572:4572'
      - '4575:4575'
      - '4576:4576'
      - '4566:4566'
      - '4583:4583'
      - '4592:4592'
    environment:
      - SERVICES=s3,dynamodb,dynamodbstreams,lambda,sns,sqs,ssm,sts
      - DATA_DIR=/tmp/localstack/data
      - DEFAULT_REGION=ap-southeast-2
      - LAMBDA_EXECUTOR=docker
    DEBUG_HOSTNAME_CONNECTION_IP=${DEBUG_HOSTNAME_CONNECTION_IP}
      - DEBUG=1
    volumes:

My approach is for local-stack-debug.sh to setup a debug environment and then pass through to the original entrypoint to bootstrap localstack.

Identifying the version of localstack you to debug

DockerHub lets us match tags against the sha256 hash of your localstack version. You are using a tagged version, right? So our hash of 2740b5509173e0efbd509bdd949217f42c97e1ab1f5b354430fdf659c2b9a152, matches localstack 0.11.5

Download the specific version of LocalStack

Next I download the source for the version I wish to debug locally. I tend to use Python virtualenv so I don’t break my global python setup if I do something wrong.

python3 -m venv localstack-0-11-5
source localstack-0-11-5/bin/activate
pip3 install localstack==0.11.5

Create my entrypoint script

The local-stack- debug.sh script I use is as follows:

#!/bin/bash

PYCHARM_VERSION=203.3645.40
DEBUG_PORT=6676

function apply_remote_debug_before_launch() {
  echo "Spin up container with Python remote debugging"

  #This step is important so that you install pydevd-pycharm into the existing PYTHONPATH
  source .venv/bin/activate

  #Ignore any warning about the incorrect version of pydevd-pycharm, I found that a newer version was needed to make bootstrapping work
  pip install pydevd-pycharm==$PYCHARM_VERSION

  # Apply debug settings to the top level of each package you're interested in.
  # If you're only interested in a subset of localstack code you can append to  __init__.py in a package further down the hierarchy
  echo "import pydevd_pycharm" >>/opt/code/localstack/localstack/__init__.py
  echo "pydevd_pycharm.settrace('$DEBUG_HOSTNAME_CONNECTION_IP', port=$DEBUG_PORT, stdoutToServer=True, stderrToServer=True)" >>/opt/code/localstack/localstack/__init__.py
  echo "Kick off service with bootstrap added"
}

if [ -n "$DEBUG_HOSTNAME_CONNECTION_IP" ]; then
  echo "Debug host ip provided, will setup debug on localstack using $DEBUG_HOSTNAME_CONNECTION_IP before launch"
  apply_remote_debug_before_launch
else
  echo "Debug params not provided, will go straight to localstack docker entrypoint"
fi

/usr/local/bin/docker-entrypoint.sh

• I provide an arbitrary port and expose my own ip address as a environment variable (so it’s portable for other developers). In my case (and until I am leased a new ip, that address is 10.10.6.85).
• Next, I download pydev-pycharm and install that into a virtualenv so as not to break the system wide path of Python.
• Now I inject the ‘settrace’ code into the top-level __init__.py of the packages that I wish to debug. In other words, anything in /opt/code/localstack/localstack or below will be eligible for breakpoints to be set.
• I put a pass through block in so that developers who have not set this variable don’t run these steps. Localstack will proceed to its original entrypoint. If they have set it, they will run the above steps and THEN proceed to the original entrypoint.

Setup my Debug Configuration

With my entrypoint setup now I need to initiate the connection from the host. My localhost is the IP address I mentioned earlier, the port matches the one declared in local-stack-debug.sh, the path mappings are just like when you download a source jar in IntelliJ or Eclipse.

Your local path mapping is wherever you set up your virtualenv for localstack earlier, ie. /path/to/virtualenv/lib/python3.8/site-packages/localstack. The remote side is where localstack bootstraps to.

Launch your debug connection

I start the connection before I spin up localstack.

Next we spin up localstack, we’ll see pydev-pycharm being downloaded.

and at some point we’ll get prompted to download the source for the __init__.py file that we’ve edited. This means the debug connection has worked.

Set breakpoints and get debugging

With this code setup, you can go about debugging. I’m able to finally see what’s going on with DynamoDB. Happy debugging!

Conclusion

I hope you found this article useful for your AWS development. Please feel free to give feedback as ever.