* Switch from the github.com/tensorflow/tensorflow repository to the new github.com/google-ai-edge/LiteRT repository for TFLite sources.
* Fix up include paths.
* More fixes.
* Fix more paths.
* Re-add missing python fixes.
* Re-add missing python fixes.
* Re-run `bazel run //third_party:python_requirements.update -- --upgrade` with Python 3.12.
* Revert third_party/python_requirements.txt.
---------
Co-authored-by: Esun Kim <veblush@google.com>
- Improved ci/Dockerfile.micro further by
- Installing QEMU from packages
- Optimizing the docker definition by combining multiple layers into single layer to reduce the footprint.
BUG=Clean-up
This update improves the `tflm-ci` Docker image and CI stability using it:
- Pinned CI for Deterministic Builds: All GitHub workflows now use the pinned `tflm-ci:0.6.3` image. This replaces the latest tag to ensure all CI runs are deterministic.
- Upgraded Base Image: The Dockerfile now uses Debian bookworm instead of bullseye.
- Updated Dependencies:
- `bazelisk` is updated to version 1.27.0.
- `yapf` is updated to 0.40.2 for consistency with [this](eacd4c404d/third_party/python_requirements.txt (L1170)).
- Improved Documentation: The Dockerfile comment is expanded to include instructions on how to upload new image versions to ghcr.io.
BUG=Clean-up
- Upgraded the Bazel BuildTool to the latest version (8.2.1).
- Updated the `tflm-ci` Docker image and pushed the new tag (0.6.1) to the `ghcr.io/tflm-bot/tflm-ci registry`.
- Updated the WORKSPACE to load `rules_cc` and `rules_shell` explicitly
- Ran `buildifier` to ensure all BUILD files to have all the fixes.
BUG=Clean-up
chore: remove obsolete ci/temp_patches
Remove ci/temp_patches, which was obsoleted in 23f608fd once it
was no longer used by the sync script. It should have been
deleted then.
Remove it not only to clean up dead code, but because it contains
a reference to `micro_copts`, which is about to be refactored
away, and we don't want to leave stray references to it in the
tree.
BUG=#2636
* Sync from upstream TF.
* Fix upstream TF Sync
The CI sync from the upstream TF has been failing for the last month
due to the relocation of certain TFLite files. In particular, the
error_reporter sources were moved to the compiler subdirectory.
This PR also includes a full sync to get us caught up again.
BUG=b/369123029
---------
Co-authored-by: TFLM-bot <tflm-github-bot@google.com>
The CI sync from the upstream TF has been failing for the last month due
to the relocation of certain TFLite files. In particular, some schema
files were relocated to tensorflow/compiler/mlir/lite subfolder. This PR
mirrors the migration and adds a few redirection headers to ensure
source compatibility for now.
This PR also includes a TF sync as well to get us caught up again. While
that could have been done separately, this is done together to ensure
everything continues to build with the relocations.
BUG=b/351824449
The Wave pypi package was recently updated, and the new version caused
failures in our CI for pip installing it. From what I can tell, this
package is no longer being used, so this PR simply removes it as a build
dependency.
BUG=b/329887147
Despite existing in TFLite, the runtime_shape.h has long differed
between TFLite and TFLM. The file is not copied during the sync and the
sync script does a `git checkout` on the file to ensure that the
existing version in the TFLM tree remains.
In PR #2476, we needed to add a runtime_shape.cc file. This PR ensures
that the runtime_shape.cc file will not be removed during the sync by
performing a `git checkout` on the existing file.
BUG=323856831
The Xtensa docker images were using python 3.6, which is incompatible
with some of our python packages. This has not been a problem before
because the Xtensa images only perform make builds with very minimal
python script usage. With the new codegen tools, Bazel is used to invoke
the code generator, and this brings in the incompatible python
dependencies.
This PR changes the base image for the three Xtensa docker containers to
pull to python instead of ubuntu. This is similar to our base TFLM
docker image, which also uses the same python:3.10-bullseye base image.
BUG=b/300655634
This PR adds the existing Xtensa dockerfiles and install scripts to the repo. Notably, this does not include any of the install packages themselves.
An additional change on top of the existing dockerfiles was to have Bazel installed in each image. This is necessary for using Bazel as part of the codegen process. Commit 6d0d214 has the original dockerfiles as they existed prior.
The install_bazelisk.sh script was also slightly tweaked to be compatible with docker as well.
BUG=cleanup
* pointers to why we are installing from source (including a TODO)
* added additional packages to install to the dockerfile (copied from http://doc/1lP_b6NogecZ7HfLKm3bqUE81mTNvNDj8L6OIgbfqNdI and tested locally on my machine).
Note that the dockerfile changes are only tested locally and not as part of the dockerfile.
BUG=documentation
Steps:
* modified ci/tflite_files.txt
* ```./ci/sync_from_upstream_tf.sh```
* additional changes needed to fix the build
There are some assumptions that we had been making that have been
somewhat broken by the changes from http://cl/539690111
More details to follow. This PR is currently to test that all the TFLM
builds are green.
BUG=http://b/287269351
NO_CHECK_TFLITE_FILES=manual sync + fixes
BUG=1891
Update Dockerfile.micro to install QEMU. This allows for CI to run QEMU based tests.
Verification:
1) Build container `docker build -f ci/Dockerfile.micro -t tflm-ci . `
2) Verify `qemu-arm` exists as `/usr/local/qemu/qemu-arm`
3) Run the container with the tflm directory mounted as a volume
`docker run -v $(pwd):/tmp/tflm -it tflm-ci bash`
4. Inside the tflm root in the container, run the qemu ci script.
`tensorflow/lite/micro/tools/ci_build/test_cortex_m_qemu.sh`
Addresses #1891 and adds Cortex M QEMU based unit tests to the CI flow.
Verification:
1. From repo root, run the command `tensorflow/lite/micro/tools/ci_build/test_cortex_m_qemu.sh`
2. Verify that all tests pass
Note this doesn't remove or change anything related to renode, for the sake of brevity, but we can remove renode here if desired.
BUG=1891
Additionally:
1. update to use `bullseye`
2. change how we `install clang` to make it easier to maintain and hopefully avoid occasional errors with the debian package installation
BUG=http://b/275010053
This refactor reduces ci test time for all the tests in ci.yml to 16 to 19 minutes depending on weather.
So our total time on pre-merge ci is going to be ~22 minutes. See bug for more details on runtimes of our current suite. We can get that down to sub 20 minutes if we can move the hifi5 xtensa test to pot-merge which is part of https://github.com/tensorflow/tflite-micro/pull/1772.
This PR will not pass our branch protection automation since it makes substantial changes to our tests. Additionally, the branch protection rules will need to change once this PR is merged.
BUG=https://issuetracker.google.com/issues/266654467
Updated the issue_on_error.py script to generate links to failed runs when creating or updating issues. Added a issue_on_error.yml workflow to be a reusable error handler and refactored the error handling block in generate_intergration_tests.yml to call it. Added error handling blocks to log_binary_size.yml as well as all the run_*.yml scripts.
BUG=https://issuetracker.google.com/issues/264308670
* Explicitly delete shared TFL/TFLM files prior to sync'ing so that stale files are deleted.
* Added a bazel build to the project generation CI. This catches errors like http://b/261106859 that were not caught with the Makefile.
* Manually ran the sync script: `ci/sync_from_upstream_tf.sh` and included the changes as a separate commit (https://github.com/tensorflow/tflite-micro/pull/1606/commits/99c7ff8a5f2907379002603fc13c40f5077f46dc)
Tested:
* Confirmed that the new bazel build in test_project_generation would have caught http://b/261106859 by uploading commits with the stale files and seeing that the presubmit checks fail with errors like:
```
/usr/bin/ld.gold: bazel-out/k8-fastbuild/bin/_objs/libtflm/0/common.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-fastbuild/bin/_objs/libtflm/1/common.pic.o: multiple definition of 'TfLiteOpaqueDelegateDelete'
/usr/bin/ld.gold: bazel-out/k8-fastbuild/bin/_objs/libtflm/0/common.pic.o: previous definition here
collect2: error: ld returned 1 exit status
Target //:libtflm failed to build
```
BUG=http://b/261106859
* added the new files to be sync'd to `ci/tflite_files.txt`
* manually ran the sync script: `ci/sync_from_upstream_tf.sh`
* update BUILD files to match changes from http://cl/488654028
Note that the changes from e6b9bd0190 are also included in this PR due to running sync script.
NO_CHECK_TFLITE_FILES=manual sync from upstream TF
BUG=http://b/259507701
This PR will revert the error_reporter_macro.h. TFLite decided to continue to use the TF_LITE_REPORT_ERROR macro, so hiding this macro is no longer needed.
BUG=b/256646805
Fix problems with the Python extension module build:
- Explicitly build against downloaded NumPy headers instead of relying on the build OS implicitly via the compiler's built-in paths, fixing #1283.
- Use tagged, verified downloads of pybind11_bazel and pybind to silence noise in the bazel output and improve tamper resistance, reproducibility, and caching.
The underlying issue was that the extension module depends on header files from numpy but that dependency was not captured in the BUILD rules.
Upstream Tensorflow's solution is too complicated to import. They manage dependencies on Python runtimes and libraries via much more extensive system of [custom rules](https://github.com/tensorflow/tensorflow/tree/master/third_party/py), designed to meet their many additional requirements.
Background:
- [pybind/pybind11](https://github.com/pybind/pybind11) is a C++ library that helps in creating our extension's interface to CPython.
- [pybind/pybind11_bazel](https://github.com/pybind/pybind11_bazel) provides Bazel rules for building pybind-based extensions, but doesn't cover the unusual case of using C headers provided in another Python module (in our case, NumPy).
- Bazel's own [bazelbuild/rules_python](https://github.com/bazelbuild/rules_python) provides rules for fetching Python dependencies from PyPI for use when *running* Python code via Bazel, but doesn't directly help with the unusual case of *building* code using headers provided by Python packages from PyPI (again, in our case, NumPy).
- The standard way, in Bazel, to build against third-party code is to create an [external repository](https://bazel.build/docs/external) by writing and calling [repository rules](https://bazel.build/extending/repo).
This PR supplies the C headers from NumPy to the build of our extension by adding a repository (via the new repository rule `tflm_py_cc_headers`) that reuses the PyPI package downloaded by rules_python and wraps it in a `cc_library` target on which the build of the extension can depend.
BUG=fixes #1283
