Added pytest helper, coverage report generator (#4)

Resolves Issue #2 by adding support for and documentation regarding Pytest and test coverage through Bazel.

6 files changed, 161 insertions, 10 deletions

diff --git a/.gitignore b/.gitignore
index 1377554..1ee84da 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1 +1 @@
-*.swp
+*.sw*
diff --git a/BUILD b/BUILD
index dbb524c..3ae07ae 100644
--- a/BUILD
+++ b/BUILD
@@ -1,6 +1,7 @@
 exports_files([
     "._dummy_.py",
     "pywrapper.sh",
+    "coverage_report.sh",
 ])
 
 sh_library(
@@ -8,3 +9,9 @@ sh_library(
     srcs = ["pywrapper.sh"],
     visibility = ["//:__subpackages__"],
 )
+
+py_library(
+    name = "pytest_helper",
+    srcs = ["pytest_helper.py"],
+    visibility = ["//visibility:public"],
+)
diff --git a/README.md b/README.md
index 6a4036e..a48de6b 100644
--- a/README.md
+++ b/README.md
@@ -60,19 +60,71 @@ bazel_python_interpreter(
 )
 ```
 
+#### Pytest and Coverage
+We have support for running Pytest tests with `bazel test //...` as well as
+preliminary support for extracting coverage reports.
+
+For Pytest support, first add `pytest` to your `requirements.txt` file and
+declare your test files with `py_test` rules depending on `pytest_helper`:
+```python
+py_test(
+    name = "test_name",
+    size = "small",
+    srcs = ["test_name.py"],
+    deps = [
+        ...
+        "@bazel_python//:pytest_helper",
+    ],
+)
+```
+and then structure your test file as follows:
+```python
+# ... first import system, third-party libraries here ...
+try:
+    from external.bazel_python.pytest_helper import main
+    IN_BAZEL = True
+except ImportError:
+    IN_BAZEL = False
+# ... then import your project libraries here ...
+
+# ... your tests go here ...
+
+if IN_BAZEL:
+    main(__name__, __file__)
+```
+
+If you will only be running under Bazel (i.e., do not need to support 'raw'
+`python3 -m pytest` calls), you can leave out the `try`/`except` and `IN_BAZEL`
+checks.
+
+To get coverage support, you can add `coverage` to your `requirements.txt` file
+then use the `bazel_python_coverage_report` macro:
+```python
+bazel_python_coverage_report(
+    name = "coverage_report",
+    test_paths = ["*"],
+    code_paths = ["*.py"],
+)
+```
+Here `test_paths` should be essentially a list of `py_test` targets which can
+produce `coverage` outputs. `code_paths` should be a list of Python files in
+the repository for which you want to compute coverage. To use it, after running
+`bazel test //...` you should be able to run `bazel run coverage_report` to
+produce an `htmlcov` directory with the coverage report.
+
 ## Known Issues
-### Missing Modules
+#### Missing Modules
 If you get errors about missing modules (e.g., `pytest not found`), please
 triple-check that you have installed OpenSSL libraries. On Ubuntu this looks
 like `apt install libssl-dev`.
 
-### Breaking The Sandbox
+#### Breaking The Sandbox
 Even if you don't use these `bazel_python` rules, you may notice that
 `py_binary` rules can include Python libraries that are not explicitly depended
 on. This is due to the fact that Bazel creates its sandbox using symbolic
 links, and Python will _follow symlinks_ when looking for a package.
 
-### Bazel-Provided Python Packages
+#### Bazel-Provided Python Packages
 Many Bazel packages come "helpfully" pre-packaged with relevant Python code,
 which Bazel will then add to the `PYTHONPATH`. For example, when you depend on
 a Python GRPC-Protobuf rule, it will automatically add a copy of the GRPC
@@ -89,7 +141,7 @@ Note this might cause problems if the path to the current repository contains
 `/com_github_grpc_grpc/`. We are on the lookout for a better solution
 long-term.
 
-### Non-Hermetic Builds
+#### Non-Hermetic Builds
 Although this process ensures everyone is using the same _version_ of Python,
 it does not make assurances about the _configuration_ of each of those Python
 instances. For example, someone who ran the `setup_python.sh` script with
@@ -97,14 +149,14 @@ instances. For example, someone who ran the `setup_python.sh` script with
 check the output of `setup_python.sh` to see which optional modules were not
 installed.
 
-### Duplicates in `~/.bazelrc`
+#### Duplicates in `~/.bazelrc`
 After building Python, `setup_python.sh` will append to your `~/.bazelrc` file
 a pointer to the path to the python parent directory provided. If you
 call `setup_python.sh` multiple times (e.g. to install multiple versions or
 re-install a single version), then multiple copies of that will be added to
 `~/.bazelrc`. These duplicates can be removed safely.
 
-### `:` Characters in Path
+#### `:` Characters in Path
 Python's venv hard-codes a number of paths in a way that Bazel violates by
 moving everything around all the time. We resolve this by replacing those
 hard-coded paths with a relative one that should work at run time in the Bazel
@@ -114,17 +166,17 @@ internal sandbox directory has a `:` character, our find and replace will
 fail.* If you notice errors that are otherwise unexplained, it may be worth
 double-checking that you don't have paths with question marks in them.
 
-### Installs Twice
+#### Installs Twice
 For some reason, Bazel seems to enjoy running the pip-installation script
 twice, an extra time with the note "for host." I'm not entirely sure why this
 is, but it doesn't seem to cause any problems other than slowing down the first
 build.
 
-### Custom Name
+#### Custom Name
 Need to support custom directory naming in pywrapper.
 
 ## Tips
-### Using Python in a Genrule
+#### Using Python in a Genrule
 To use the interpreter in a genrule, depend on it in the tools and make sure to
 source the venv before calling `python3`:
 ```python
diff --git a/bazel_python.bzl b/bazel_python.bzl
index 53e67b0..990aec7 100644
--- a/bazel_python.bzl
+++ b/bazel_python.bzl
@@ -118,3 +118,35 @@ bazel_python_venv = rule(
         "run_after_pip": attr.string(),
     },
 )
+
+def bazel_python_coverage_report(name, test_paths, code_paths):
+    """Adds a rule to build the coverage report.
+
+    @name is the name of the target which, when run, creates the coverage
+        report.
+    @test_paths should be a list of the py_test targets for which coverage
+        has been run. Bash wildcards are supported.
+    @code_paths should point to the Python code for which you want to compute
+        the coverage.
+    """
+    test_paths = " ".join([
+        "bazel-out/*/testlogs/" + test_path + "/test.outputs/outputs.zip"
+        for test_path in test_paths])
+    code_paths = " ".join([code_path for code_path in code_paths])
+    # For generating the coverage report.
+    native.sh_binary(
+        name = name,
+        srcs = ["@bazel_python//:coverage_report.sh"],
+        deps = [":_dummy_coverage_report"],
+        args = [test_paths, code_paths],
+    )
+
+    # This is only to get bazel_python_venv as a data dependency for
+    # coverage_report above. For some reason, this doesn't work if we directly put
+    # it on the sh_binary. This is a known issue:
+    # https://github.com/bazelbuild/bazel/issues/1147#issuecomment-428698802
+    native.sh_library(
+        name = "_dummy_coverage_report",
+        srcs = ["@bazel_python//:coverage_report.sh"],
+        data = ["//:bazel_python_venv"],
+    )
diff --git a/coverage_report.sh b/coverage_report.sh
new file mode 100755
index 0000000..b4471be
--- /dev/null
+++ b/coverage_report.sh
@@ -0,0 +1,29 @@
+COVTEMP=$PWD/coverage_tmp
+rm -rf $COVTEMP
+mkdir $COVTEMP
+
+source bazel_python_venv_installed/bin/activate
+
+# Go to the main workspace directory and run the coverage-report.
+pushd $BUILD_WORKSPACE_DIRECTORY
+
+# We find all .cov files, which should be generated by pytest_helper.py
+cov_zips=$(ls $1)
+i=1
+for cov_zip in $cov_zips
+do
+    echo $cov_zip
+    unzip -p $cov_zip coverage.cov > $COVTEMP/$i.cov
+    i=$((i+1))
+done
+
+# Remove old files
+rm -rf .coverage htmlcov
+
+# Then we build a new .coverage as well as export to HTML
+python3 -m coverage combine $COVTEMP/*.cov
+python3 -m coverage html $2
+
+# Remove temporaries and go back to where Bazel started us.
+rm -r $COVTEMP
+popd
diff --git a/pytest_helper.py b/pytest_helper.py
new file mode 100644
index 0000000..591e793
--- /dev/null
+++ b/pytest_helper.py
@@ -0,0 +1,31 @@
+"""Helper methods for using Pytest within Bazel."""
+import sys
+import numpy as np
+import pytest
+try:
+    import coverage
+    COVERAGE = True
+except ImportError:
+    COVERAGE = False
+import os
+
+if COVERAGE:
+    # We need to do this here, otherwise it won't catch method/class declarations.
+    # Also, helpers imports should be before all other local imports.
+    cov_file = "%s/coverage.cov" % os.environ["TEST_UNDECLARED_OUTPUTS_DIR"]
+    cov = coverage.Coverage(data_file=cov_file)
+    cov.start()
+
+def main(script_name, file_name):
+    """Test runner that supports Bazel test and the coverage_report.sh script.
+
+    Tests should import this module before importing any other local scripts,
+    then call main(__name__, __file__) after declaring their tests.
+    """
+    if script_name != "__main__":
+        return
+    exit_code = pytest.main([file_name, "-s"])
+    if COVERAGE:
+        cov.stop()
+        cov.save()
+    sys.exit(exit_code)