- Improved ci/Dockerfile.micro further by
- Installing QEMU from packages
- Optimizing the docker definition by combining multiple layers into single layer to reduce the footprint.
BUG=Clean-up
This update improves the `tflm-ci` Docker image and CI stability using it:
- Pinned CI for Deterministic Builds: All GitHub workflows now use the pinned `tflm-ci:0.6.3` image. This replaces the latest tag to ensure all CI runs are deterministic.
- Upgraded Base Image: The Dockerfile now uses Debian bookworm instead of bullseye.
- Updated Dependencies:
- `bazelisk` is updated to version 1.27.0.
- `yapf` is updated to 0.40.2 for consistency with [this](eacd4c404d/third_party/python_requirements.txt (L1170)).
- Improved Documentation: The Dockerfile comment is expanded to include instructions on how to upload new image versions to ghcr.io.
BUG=Clean-up
- Upgraded the Bazel BuildTool to the latest version (8.2.1).
- Updated the `tflm-ci` Docker image and pushed the new tag (0.6.1) to the `ghcr.io/tflm-bot/tflm-ci registry`.
- Updated the WORKSPACE to load `rules_cc` and `rules_shell` explicitly
- Ran `buildifier` to ensure all BUILD files to have all the fixes.
BUG=Clean-up
The Wave pypi package was recently updated, and the new version caused
failures in our CI for pip installing it. From what I can tell, this
package is no longer being used, so this PR simply removes it as a build
dependency.
BUG=b/329887147
* pointers to why we are installing from source (including a TODO)
* added additional packages to install to the dockerfile (copied from http://doc/1lP_b6NogecZ7HfLKm3bqUE81mTNvNDj8L6OIgbfqNdI and tested locally on my machine).
Note that the dockerfile changes are only tested locally and not as part of the dockerfile.
BUG=documentation
BUG=1891
Update Dockerfile.micro to install QEMU. This allows for CI to run QEMU based tests.
Verification:
1) Build container `docker build -f ci/Dockerfile.micro -t tflm-ci . `
2) Verify `qemu-arm` exists as `/usr/local/qemu/qemu-arm`
3) Run the container with the tflm directory mounted as a volume
`docker run -v $(pwd):/tmp/tflm -it tflm-ci bash`
4. Inside the tflm root in the container, run the qemu ci script.
`tensorflow/lite/micro/tools/ci_build/test_cortex_m_qemu.sh`
Additionally:
1. update to use `bullseye`
2. change how we `install clang` to make it easier to maintain and hopefully avoid occasional errors with the debian package installation
BUG=http://b/275010053
Fix problems with the Python extension module build:
- Explicitly build against downloaded NumPy headers instead of relying on the build OS implicitly via the compiler's built-in paths, fixing #1283.
- Use tagged, verified downloads of pybind11_bazel and pybind to silence noise in the bazel output and improve tamper resistance, reproducibility, and caching.
The underlying issue was that the extension module depends on header files from numpy but that dependency was not captured in the BUILD rules.
Upstream Tensorflow's solution is too complicated to import. They manage dependencies on Python runtimes and libraries via much more extensive system of [custom rules](https://github.com/tensorflow/tensorflow/tree/master/third_party/py), designed to meet their many additional requirements.
Background:
- [pybind/pybind11](https://github.com/pybind/pybind11) is a C++ library that helps in creating our extension's interface to CPython.
- [pybind/pybind11_bazel](https://github.com/pybind/pybind11_bazel) provides Bazel rules for building pybind-based extensions, but doesn't cover the unusual case of using C headers provided in another Python module (in our case, NumPy).
- Bazel's own [bazelbuild/rules_python](https://github.com/bazelbuild/rules_python) provides rules for fetching Python dependencies from PyPI for use when *running* Python code via Bazel, but doesn't directly help with the unusual case of *building* code using headers provided by Python packages from PyPI (again, in our case, NumPy).
- The standard way, in Bazel, to build against third-party code is to create an [external repository](https://bazel.build/docs/external) by writing and calling [repository rules](https://bazel.build/extending/repo).
This PR supplies the C headers from NumPy to the build of our extension by adding a repository (via the new repository rule `tflm_py_cc_headers`) that reuses the PyPI package downloaded by rules_python and wraps it in a `cc_library` target on which the build of the extension can depend.
BUG=fixes #1283
* First pass working with float input/output
* Numpy buffer input works. Got a sine graph for hello world output.
Copyright notice
* Remove static references
* Output tensors work
* Output tensors work
* bazel test works
bazel test tensorflow/lite/micro/tools:interpreter_test
--test_output=all
* Successfully built extension with Python setuptools
cd tensorflow/lite/micro/tools/interpreter_pypi
python -m build
pip install dist/example_tflm_interpreter_psho-0.0.3-cp39-cp39-linux_x86_64.whl --force-reinstall
python tests/test-interpreter.py
TestPyPi:
pip uninstall example-tflm-interpreter-psho
python -m twine upload --repository testpypi dist/example-tflm-interpreter-psho-0.0.3.tar.gz
pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple example-tflm-interpreter-psho==0.0.3
python tests/test-interpreter.py
* Refactored some
* Cleaned for production and tested with conv model
* Fixed formatting
* Fixed numpy header issue for Docker. Need to fix double registration issue that's somehow only present in Docker.
* Test fix with changes in ci.yml
* Added sudo
* Added nomsan and noasan tags to python test
* Added -layering-check, used py_library instead of py_binary to generate conv model, added micro namespace
* Removed extra debug code
* Added new tests
* Changed reinterpret_steal to cast
* Addressed PR comments
* size log refactor (#1156)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Addressed PR and review comments
* adding documentation to log_binary* workflows. (#1161)
* Automated binary size log update (#1160)
* Create an allocator to manage persistent arena (#1174)
* Create an allocator to manage persistent arena
This PR corresponds to internal cl/452380512.
Create an allocator to manage persistent arena.
This is another step towards the feature of enabling the client to
have two separate memory arenas.
BUG=https://b/226971240
* Manually patch due to Makefile difference
* Fix virtual environment when building Vela for Arm Ethos-U (#1177)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Removed print statement
Co-authored-by: jwithers <jpwithers@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: TFLM-bot <tflm-github-bot@google.com>
Co-authored-by: deqiangc <86809673+deqiangc@users.noreply.github.com>
Co-authored-by: Måns Nilsson <mans.nilsson@arm.com>
When upgrading the docker container, we see new format
fix/check on existing python file. That is because we did not
fix yapf version. This PR sets the yapf version to the
latest 0.32.0, which is what the latest container is using
BUG=https://issuetracker.google.com/216670232
* adding pandas and matplotlib
* updating container tag to latest
* fixing formatting issues that have been slipping through older ci setup
Co-authored-by: deqiangc <86809673+deqiangc@users.noreply.github.com>
* Update to the latest buildifier release.
For the previous buildifier version, the exit code would always be 0
(regardless of whether files needed formatting or not).
Manually confirmed that with version 4.2.3 the test_code_style.sh script
correctly errors out when a BUILD file has formatting errors (and that
it doesn't with the previous version).
Also,
* Updated the docker container and pointing to the new container.
* Fix the formatting errors in .bzl file that crept in because this check was not working in OSS.
BUG=http://b/203803273
* Exclude the downloads directory.
* Updated docs based on review comments.
* Bump renode and robot versions
Signed-off-by: Jakub Piecuch <jpiecuch@antmicro.com>
* Refactor test_with_renode.sh
Pass the name of the UART device as a parameter, since RISC-V uses
a different name.
Also pass the full list of binaries to test instead of a directory,
so that we can filter some tests out from within make, which is easier
compared to doing it in bash.
Signed-off-by: Jakub Piecuch <jpiecuch@antmicro.com>
* Use TfLiteRound instead of std::round in hard_swish_test
Signed-off-by: Jakub Piecuch <jpiecuch@antmicro.com>
* Don't pass UART name as an argument to test_with_renode.sh
Passing it as an argument works when running tests via `make test`, but
it breaks when we try to run a single test via `make test_foo`, because
the recipe for `make test_foo` doesn't pass the `${UART_NAME}` argument
to the test script.
Signed-off-by: Jakub Piecuch <jpiecuch@antmicro.com>
With this change:
* we are set up to use yapf as part of the docker container
* pigweed patch is updated to use the google Python style (pep8 with indent of 2 spaces)
* some documentation updates.
The code still needs to be formatted, and then the Python formatting check will be turned on as part of ci.
BUG=http://b/194404216
* removed unnecessary bits from tensorflow/BUILD
* updated the sync script accordingly.
* using test_all.sh instead of test_all_new.sh from Dockerfile.micro
(will not update test_all.sh bsed on upstream TF, and will manually
keep that consistent with upstream TF).
