This blog post contains a set of files and bash scripts for creating a new python project in a virtual environment. It installs a command line tool for the new app, and defines a minimal Dockerfile, and Makefile, which can be used for building the image. If you find this helpful, or you think it’s whack, let me know in the comments!

Step 1.

Create the following files.

touch $HOME/Templates/python_class.py
touch $HOME/Templates/python_cli.py
touch $HOME/Templates/python_main.py
touch $HOME/Templates/python_makefile
touch $HOME/Templates/python_setup.py
touch $HOME/Templates/python_test_context.py
touch $HOME/Templates/python_unittest.py
touch $HOME/Templates/python.dockerfile

We will go through the code of each of these files.

python_setup.py

# -*- coding: utf-8 -*-
import setuptools


with open("./README.md", "r") as fh:
    long_description = fh.read()

# Replace with your own project name

setuptools.setup(
    name="app",
    version="0.0.1",
    author="Your Name",
    author_email="youremail@somedomain.com",
    description="This project is just great!",
    long_description=long_description,
    long_description_content_type="text/markdown",
    url="https://danjcook.com/git",
    packages=setuptools.find_packages(),
    # include_package_data=True,
    install_requires=[
        "click==7.1.1",
        "python-dotenv==0.11.0"
    ],
    entry_points="""
        [console_scripts]
        app=main:cli
    """,
    classifiers=[
        "Programming Language :: Python :: 3",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
    ],
    python_requires='>=3.7',
)

We are going to install a command line tool for this project, so that you can pass parameters and arguments to the program easily. We will use the excellent module, Click. We will also define our secrets in a .env file, and use the python-dotenv module for securely accessing our secrets at runtime.

python_class.py

# -*- coding: utf-8 -*-
"""Module description

This module does great things.
"""
import os

# Implementation constants
SCRIPT_DIRNAME, SCRIPT_FILENAME = os.path.split(os.path.abspath(__file__))
PROJECT_ROOT_DIR = os.path.dirname(SCRIPT_DIRNAME)

# Classes, methods, functions, and variables
class AppName():
    def __init__(self):
        pass
    def func(self):
        pass

You will probably have a set of classes in your project that do different things. This will help to keep your project organized and understandable. Here, we define the first class for our project and create a new file for it.

python_cli.py

import os
import sys
import click

from app import app

class AppNameCLI(object):
    def __init__(self):
        self.app = app.AppName()

@click.group()
@click.pass_context
def cli(ctx, settings_file):
    ctx.obj = AppNameCLI()

@cli.command()
# @click.argument('arg1')
# @click.argument('arg2')
# @click.option('--dryrun', is_flag=True)
@click.pass_context
def app_func(ctx):#, arg1, arg2, dryrun):
    """Description:
    \b

    Hello world
    """
    server = ctx.obj.app.func()

if __name__ == '__main__':
    cli()

We will access our class that does the processing from the command line. Here, we import and do the work.

python_main.py

# -*- coding: utf-8 -*-
"""Module description

This module does great things.

Example:
    $ python3 great_things.py positional_argument --keyword_argument 1

Style guide: https://www.python.org/dev/peps/pep-0008/
"""

import argparse
import os

from app import app

# Implementation constants go ...
# here

if __name__ == '__main__':

    parser = argparse.ArgumentParser(
        description=""
    )
    parser.add_argument("--optional",
        metavar="opt",
        type=str,
        default="Add an option",
        help="Add an optional argument"
    )

    args = parser.parse_args()

    app = app.AppName()
    app.func()

This code is for legacy that does not want to transition to the use of the Click module. It is not necessary, but it can be helpful for quick one-off projects.

python_makefile

PODMAN := $(shell command -v podman 2>/dev/null)
DOCKER := $(shell command -v docker 2>/dev/null)
ifeq ($(PODMAN), /usr/bin/podman)
CONTAINER_ENGINE := $(PODMAN)
else
CONTAINER_ENGINE := $(DOCKER)
endif

.PHONY: help
help: ## The default task is help.
	@awk 'BEGIN {FS = ":.*?## "} /^[a-zA-Z_-]+:.*?## / {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}' $(MAKEFILE_LIST)
.DEFAULT_GOAL := help

build_image: help_deploy## Build the image
	 "$(CONTAINER_ENGINE)" build -f ./Dockerfile -t appname
	 "$(CONTAINER_ENGINE)" image prune --filter label=stage=appname_base_image

prune_baseimage: ## Remove the intermediate image
	 "$(CONTAINER_ENGINE)" image prune --filter label=stage=appname_base_image

run_test: ## Run app for a help message after image build
	 "$(CONTAINER_ENGINE)" run app

ship_image: ## Save image as tar, scp to and load the image on host machine
	"$(CONTAINER_ENGINE)" save > appname.tar appname
	scp appname.tar server:~/appname/
	ssh server "cd ~/appname && docker load -i appname.tar"

distribute: build_image ship_image ## build image, save tar, scp, load

help_deploy: ## Build image and run tests in container
	@echo "--------------------------------------------------------------------"
	@echo "Usage:"
	@echo "podman run \\"
	@echo "    appname <options> <commands> <options>"
	@echo "--------------------------------------------------------------------"

No project is complete without continuous integration and delivery built in. This Makefile will define some simple concepts for building, testing, and distributing your project.

python_test_context.py

# -*- coding: utf-8 -*-
#pylint: skip-file
"""
This script is a testing utility used to append the '../app' directory
(where the primary .py executables live) to the system path so that they can be
imported during the tests.
"""
import os
import sys


SCRIPT_DIRNAME, SCRIPT_FILENAME = os.path.split(os.path.abspath(__file__))
PROJECT_ROOT_DIR = os.path.dirname(SCRIPT_DIRNAME)
APP_DIR = os.path.join(PROJECT_ROOT_DIR)

sys.path.insert(0, APP_DIR)

from app import AppName

Of course, we need to test, but to do so, we also need to define our context.

python_unittest.py

# -*- coding: utf-8 -*-
"""
This module tests functions
"""
import unittest

from context import *


class MyTestClass(unittest.TestCase):
    """
    A simple class for testing
    """

    def setUp(self):
        self.do_tests = True
        self.app = AppName()

    def test_success(self):
        """Test this function!"""
        do_tests = self.do_tests
        self.assertEqual(do_tests, True)


if __name__ == '__main__':
    unittest.main()

Here, we define our unittest class.

python.dockerfile

# base image
FROM python:3.8.0-slim as base_image
LABEL stage=appname_base_image
RUN apt-get update \
  && apt-get install gcc -y \
  && apt-get clean

WORKDIR app
COPY ./README.md /app/README.md
COPY ./requirements.txt /app/requirements.txt
COPY ./app /app
COPY ./setup.py /app/setup.py

RUN pip install --upgrade pip
RUN pip install --user --editable .

# production image
FROM python:3.8.0-slim as app_pod
COPY --from=base_image /root/.local /root/.local
COPY --from=base_image /app /app
WORKDIR app
ENV PATH=/root/.local/bin:$PATH
ENTRYPOINT ["app"]

For containerizing our project, we define a simple Dockerfile.

Step 2.

Create a new bash function in a file, new-py-project.sh.

Copy and paste this code:

#!/bin/bash

import Hello

if [ -d ./$1 ]; then
  echo "That dir already exists!"
  exit 1
else
  mkdir ./$1
  cd ./$1
fi
if [ ! -f ./pyvenv.cfg ]; then
  python -m venv ./
  mkdir ./app
  mkdir ./app/tests
  cp $HOME/Templates/python_main.py ./simple-main.py
  cp $HOME/Templates/python_cli.py ./main.py
  cp $HOME/Templates/python_class.py ./app/app.py
  cp $HOME/Templates/python_unittest.py ./app/tests/test.py
  cp $HOME/Templates/python_test_context.py ./app/tests/context.py
  cp $HOME/Templates/python.dockerfile ./Dockerfile
  cp $HOME/Templates/python_setup.py ./setup.py
  cp $HOME/Templates/python_makefile ./Makefile
  cp $HOME/Templates/python_gitignore ./.gitignore
  touch ./requirements.txt
  touch README.md
  echo Click >> ./requirements.txt
  echo python-dotenv >> ./requirements.txt
  source ./bin/activate
  pip install --upgrade pip
  pip install -r requirements.txt
  pip install --editable .
  app
  deactivate
  rm -rf __pycache__
  git init
  git add .
  git status
  echo "Created new python project in $1"
fi

Step 3.

In your .bash_aliases file, put the following line,

alias python='python3'
alias pip='pip3'
alias newpyproj='$HOME/Utilities/new-python-project.sh'
alias entervenv='source bin/activate'

Now, source your .bash_aliases file, and test out your automation pipeline:

newpyproj my-awesome-app

When you cd into my-awesome-app dir, you should be able to run the app command once you enter into the venv with your new entervenv alias.

Now, don’t waste any more time. Start developing!