Building Command Line Tools with argparse

The smallest example

import argparse

parser = argparse.ArgumentParser(description="Count the lines in a file.")
parser.add_argument("path", help="the file to read")

args = parser.parse_args()
print(args.path)

$ python count.py notes.txt
notes.txt

$ python count.py
usage: count.py [-h] path
count.py: error: the following arguments are required: path
                                                        (exit code 2)

$ python count.py --help
usage: count.py [-h] path

Count the lines in a file.

positional arguments:
  path        the file to read

options:
  -h, --help  show this help message and exit

Two lines produced argument validation, a usage message, a help page and the correct exit code. That is the whole argument for using it.

Positional and optional arguments

import argparse

parser = argparse.ArgumentParser()

parser.add_argument("source")                       # required, by position
parser.add_argument("destination")                  # required, by position
parser.add_argument("-o", "--output")               # optional, by name
parser.add_argument("-n", "--number", type=int)     # converted to an int
parser.add_argument("-v", "--verbose", action="store_true")   # a flag

args = parser.parse_args(["a.txt", "b.txt", "-n", "5", "--verbose"])
print(args)
print(args.source, args.number, args.verbose)

Namespace(source='a.txt', destination='b.txt', output=None, number=5, verbose=True)

A name without dashes is positional and required. A name with dashes is optional. The attribute name comes from the long option, with dashes turned into underscores.

Types and defaults

import argparse
from pathlib import Path

parser = argparse.ArgumentParser()

parser.add_argument("--count", type=int, default=10)
parser.add_argument("--rate", type=float, default=1.0)
parser.add_argument("--path", type=Path)                 # any callable works
parser.add_argument("--mode", choices=["fast", "slow", "auto"], default="auto")
parser.add_argument("--tag", action="append", default=[])   # repeatable
parser.add_argument("--files", nargs="+")                   # one or more
parser.add_argument("--extra", nargs="*", default=[])       # zero or more
parser.add_argument("--pair", nargs=2, metavar=("KEY", "VALUE"))

args = parser.parse_args([
    "--count", "5", "--mode", "fast",
    "--tag", "a", "--tag", "b",
    "--files", "x.txt", "y.txt",
    "--pair", "host", "localhost",
])
print(args)

$ python tool.py --mode turbo
tool.py: error: argument --mode: invalid choice: 'turbo'
       (choose from 'fast', 'slow', 'auto')

$ python tool.py --count abc
tool.py: error: argument --count: invalid int value: 'abc'

type and choices do the validation, and produce a clear message and exit code 2 without a line of your own code.

Custom validation

import argparse
from pathlib import Path


def existing_file(value):
    path = Path(value)
    if not path.is_file():
        raise argparse.ArgumentTypeError(f"{value} is not a file")
    return path


def positive_int(value):
    number = int(value)
    if number <= 0:
        raise argparse.ArgumentTypeError(f"{value} must be positive")
    return number


def percentage(value):
    number = float(value)
    if not 0 <= number <= 100:
        raise argparse.ArgumentTypeError(f"{value} must be between 0 and 100")
    return number


parser = argparse.ArgumentParser()
parser.add_argument("input", type=existing_file)
parser.add_argument("--workers", type=positive_int, default=4)
parser.add_argument("--threshold", type=percentage, default=50.0)

A type is any callable that takes a string and returns a value, raising ArgumentTypeError on bad input. This puts every validation rule in one place, before your program starts.

Actions

import argparse

parser = argparse.ArgumentParser()
parser.add_argument("-v", "--verbose", action="count", default=0)
parser.add_argument("--version", action="version", version="%(prog)s 2.1.0")

args = parser.parse_args(["-vvv"])
print(args.verbose)                # 3

LEVELS = {0: "WARNING", 1: "INFO", 2: "DEBUG"}
print("log level:", LEVELS.get(args.verbose, "DEBUG"))

Mutually exclusive options

import argparse

parser = argparse.ArgumentParser()

group = parser.add_mutually_exclusive_group()
group.add_argument("--quiet", action="store_true")
group.add_argument("--verbose", action="store_true")

print(parser.parse_args(["--quiet"]))
# parser.parse_args(["--quiet", "--verbose"])
# error: argument --verbose: not allowed with argument --quiet


required = parser.add_mutually_exclusive_group(required=True)
required.add_argument("--from-file")
required.add_argument("--from-url")

Argument groups in the help page

import argparse

parser = argparse.ArgumentParser(description="Process some data.")

io_group = parser.add_argument_group("input and output")
io_group.add_argument("input")
io_group.add_argument("-o", "--output", default="-")

tuning = parser.add_argument_group("tuning")
tuning.add_argument("--workers", type=int, default=4)
tuning.add_argument("--batch-size", type=int, default=100)

parser.print_help()

Subcommands

import argparse
import sys


def cmd_add(args):
    print(f"adding {args.title!r} with tags {args.tags}")
    return 0


def cmd_list(args):
    print(f"listing up to {args.limit} notes, sorted by {args.sort}")
    return 0


def cmd_delete(args):
    if not args.force:
        print("refusing to delete without --force", file=sys.stderr)
        return 1
    print(f"deleting {args.note_id}")
    return 0


def build_parser():
    parser = argparse.ArgumentParser(prog="notes", description="A note manager.")
    parser.add_argument("--version", action="version", version="%(prog)s 1.0")
    parser.add_argument("-v", "--verbose", action="store_true")

    sub = parser.add_subparsers(dest="command", required=True,
                                metavar="COMMAND")

    add = sub.add_parser("add", help="add a note")
    add.add_argument("title")
    add.add_argument("--tags", nargs="*", default=[])
    add.set_defaults(func=cmd_add)

    listing = sub.add_parser("list", help="list notes")
    listing.add_argument("--limit", type=int, default=20)
    listing.add_argument("--sort", choices=["date", "title"], default="date")
    listing.set_defaults(func=cmd_list)

    delete = sub.add_parser("delete", help="delete a note")
    delete.add_argument("note_id")
    delete.add_argument("--force", action="store_true")
    delete.set_defaults(func=cmd_delete)

    return parser


def main(argv=None):
    parser = build_parser()
    args = parser.parse_args(argv)
    return args.func(args)


print(main(["add", "Regex notes", "--tags", "python", "text"]))
print(main(["list", "--limit", "5", "--sort", "title"]))
print(main(["delete", "n-1"]))

set_defaults(func=...) attaches the handler to each subcommand, so main is three lines regardless of how many commands there are. This is the standard shape for a tool with several verbs.

Files as arguments

import argparse
import sys

parser = argparse.ArgumentParser()
parser.add_argument("input", nargs="?",
                    type=argparse.FileType("r", encoding="utf-8"),
                    default=sys.stdin)
parser.add_argument("-o", "--output",
                    type=argparse.FileType("w", encoding="utf-8"),
                    default=sys.stdout)

# args = parser.parse_args()
# for line in args.input:
#     args.output.write(line.upper())

FileType opens the file and reports a clear error if it cannot. Using - as the file name means standard input or output. Note that these files are not closed automatically, so for anything long lived take a path and open it yourself.

Reading defaults from a file

import argparse

parser = argparse.ArgumentParser(fromfile_prefix_chars="@")
parser.add_argument("--workers", type=int, default=1)
parser.add_argument("--mode", default="auto")

# With a file options.txt containing one argument per line:
#   --workers
#   8
#   --mode
#   fast
#
# $ python tool.py @options.txt

A complete tool

"""filestats.py - report statistics about text files."""

import argparse
import sys
from pathlib import Path


def existing_file(value):
    path = Path(value)
    if not path.is_file():
        raise argparse.ArgumentTypeError(f"{value} is not a readable file")
    return path


def analyse(path, encoding):
    lines = words = chars = blank = 0
    longest = 0
    with open(path, encoding=encoding, errors="replace") as handle:
        for line in handle:
            lines += 1
            stripped = line.strip()
            if not stripped:
                blank += 1
            words += len(stripped.split())
            chars += len(line)
            longest = max(longest, len(line.rstrip("\n")))
    return {
        "file": path.name,
        "lines": lines,
        "blank": blank,
        "words": words,
        "chars": chars,
        "longest": longest,
    }


def build_parser():
    parser = argparse.ArgumentParser(
        prog="filestats",
        description=__doc__,
        epilog="Example: filestats.py *.txt --sort words --top 5",
        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
    )
    parser.add_argument("files", nargs="+", type=existing_file,
                        help="files to analyse")
    parser.add_argument("--encoding", default="utf-8",
                        help="text encoding to assume")
    parser.add_argument("--sort", choices=["file", "lines", "words", "chars"],
                        default="file", help="column to sort by")
    parser.add_argument("--top", type=int, help="show only the first N rows")
    parser.add_argument("--no-total", action="store_true",
                        help="do not print a total row")

    output = parser.add_mutually_exclusive_group()
    output.add_argument("--csv", action="store_true", help="output as CSV")
    output.add_argument("--quiet", action="store_true",
                        help="print only the totals")
    return parser


def main(argv=None):
    parser = build_parser()
    args = parser.parse_args(argv)

    rows = [analyse(path, args.encoding) for path in args.files]
    rows.sort(key=lambda r: r[args.sort], reverse=args.sort != "file")
    if args.top:
        rows = rows[: args.top]

    totals = {
        key: sum(r[key] for r in rows)
        for key in ("lines", "blank", "words", "chars")
    }

    if args.quiet:
        print(f"{totals['lines']} {totals['words']} {totals['chars']}")
        return 0

    if args.csv:
        print("file,lines,blank,words,chars,longest")
        for row in rows:
            print(",".join(str(row[k]) for k in
                           ("file", "lines", "blank", "words", "chars", "longest")))
        return 0

    header = f"{'file':<24}{'lines':>8}{'blank':>8}{'words':>9}{'chars':>10}"
    print(header)
    print("-" * len(header))
    for row in rows:
        print(f"{row['file']:<24}{row['lines']:>8}{row['blank']:>8}"
              f"{row['words']:>9}{row['chars']:>10}")

    if not args.no_total and len(rows) > 1:
        print("-" * len(header))
        print(f"{'total':<24}{totals['lines']:>8}{totals['blank']:>8}"
              f"{totals['words']:>9}{totals['chars']:>10}")

    return 0


if __name__ == "__main__":
    try:
        sys.exit(main())
    except KeyboardInterrupt:
        sys.exit(130)

Testing a parser

import unittest
import argparse


class TestParser(unittest.TestCase):
    def setUp(self):
        self.parser = build_parser()

    def test_defaults(self):
        args = self.parser.parse_args(["a.txt"])
        self.assertEqual(args.sort, "file")
        self.assertEqual(args.encoding, "utf-8")

    def test_invalid_choice(self):
        with self.assertRaises(SystemExit):
            self.parser.parse_args(["a.txt", "--sort", "nonsense"])

    def test_mutually_exclusive(self):
        with self.assertRaises(SystemExit):
            self.parser.parse_args(["a.txt", "--csv", "--quiet"])

Because parse_args accepts a list, the parser can be tested without running the program. Invalid input raises SystemExit, which is what assertRaises catches.

Common mistakes

• Forgetting type=int and comparing a string to a number.
• Expecting --batch-size to become args.batch-size; it is args.batch_size.
• Using action="store_true" together with type, which is meaningless.
• Not setting required=True on subparsers, so running with no command does nothing.
• Validating inside the program when a type callable would report it earlier and better.
• Writing your own --help.
• Leaving FileType handles unclosed in a long running program.

Best practices

• Build the parser in a function so it can be tested.
• Use type callables for every validation rule.
• Use choices for fixed option sets.
• Use subcommands with set_defaults(func=...) once there is more than one verb.
• Give every argument a help string, and use ArgumentDefaultsHelpFormatter.
• Let argparse handle usage errors and exit codes.

Practice

1. Write a tool taking a file, an output format and a verbosity flag.
2. Add a custom type that accepts only an existing directory.
3. Build a tool with three subcommands, each with its own options.
4. Add a mutually exclusive group and demonstrate the error message.
5. Write unit tests for a parser covering defaults, an invalid choice and a missing required argument.

Conclusion

Describe the interface and argparse builds the parser, the validation, the error messages, the help page and the exit codes. Put validation in type callables, use subcommands with set_defaults, and never write a help message by hand.