* fix: update comment handling in render-binja-import-script.py
- ret of get_comment_at is str
+ set_comment is unkown in ver5.2
* fix: fix Binary Ninja import script's AppendComment and AppendLvarComment functions
- Remove undefined `pc` variable reference in AppendLvarComment
- Add fallback handling when get_code_refs returns no references in AppendComment
- Add null check for get_functions_containing to prevent crash
- Add bv.set_comment_at fallback when function is not found
- Add int() conversion for address parameters for type safety
- Change `return` to `continue` in AppendComment loop to annotate all references
The max_len fallback branch tested args.libs twice, so --pes without an
explicit --max-len raised ValueError('unknown extraction type'). The
second condition now checks args.pes.
--no only works today via argparse abbreviation of --no-string-type. The
PR's planned --no-section/--no-structure/--no-tag flags will make --no an
ambiguous prefix and break these call sites. Use the canonical spelling.
The default --output-dir hardcoded a path constructed from parents[2],
duplicating the floss/tags/data root that data_root() already provides.
Use data_root() / "oss" so the script follows the same source of truth
as query_string.py.
Implement the first batch of the FLOSS migration development spec.
CLI:
- remove --no-layout, --no-tags, -l/--load, -H, --no, and --only
- add --string-type and --no-string-type to replace --no and --only
- rename --functions to --analyze-functions
- load a saved JSON results document automatically from file content
- print the full option list when run with no arguments (exit 1)
- always show the full option list in --help; --version exits 0
Metrics:
- add layout and tags runtime timing fields to the results document
- record elapsed time for the layout/tag step and tag DB matching
Other:
- point the tag scripts at the current tag database location
Update the CLI tests and the usage documentation for the new flags.
Run build workflow on PRs to quantumstrand (like tests.yml). Put the
sample path before --no stack/tight/decoded so nargs="+" does not
swallow the path in the smoke test and bulk_analyze.
- Smoke standalone floss (not floss quantum) in build.yml
- Fall back to classic statics on any layout/tag pipeline failure
- Read the sample once and reuse the buffer for layout
- Drop deleted floss.quantum/document from PyInstaller hiddenimports
- Restore CLI/product-flag tests; fix stale qs/QuantumStrand docs
- Drop dead utf-8 encoding branch in enrich
Stop re-exporting pipeline/CLI helpers from floss.main. Import
select_functions from floss.pipeline, StringType/set_log_config from
floss.cli, and load stack/tight string modules lazily in conftest to
avoid the identify/features circular import.
Deprecate floss.quantum and document dual schema; floss quantum remains
a thin alias. Update README, bulk_analyze, and QS document tests for the
unified ResultDocument.
Keep build_oss_db under scripts/tags/ with scripts.tags import in tests,
update the OSS DB workflow path, and fix a mypy error where remaining
library names were treated as LibraryDiff objects.
Co-locate the Rust version hash regenerator with rust_version_database.py,
write output via __file__-relative paths, and bootstrap repo root on
sys.path in generate_gp_db.py so it runs when invoked directly.
Group auxiliary scripts by purpose: classic FLOSS tool integrations under
disassemblers/, QuantumStrand tag DB maintenance under tags/ (including
build_oss_db.py), and batch runs under analysis/. Add category READMEs,
update tests and import paths, and point build_oss_db default output at
floss/qs/db/data/oss.
Pre-commit already type-checks scripts/; the exclude skipped tag maintenance
scripts that import floss.tags. Add scripts package markers and mypy_path so
mypy resolves scripts.tags imports, and update the build_oss_db test import.
Also fix query_string.py to load the GP database from floss/qs/db/data/gp/.
Move string extraction out of layout/types into layout/extract, fold
merge_overlapping_ranges into ranges, and consolidate layout-derived tag
checks into tags/checks. Remove the unused enrich shim, add pipeline.py as
the iteration-2 enrichment placeholder, restore script usage docs, and
clarify interim document vs results schemas with docstrings.
msgspec 0.14.2 has no cp312 wheels and fails to build from source on
3.12. Pin 0.21.1 (wheels for 3.10–3.12) in pyproject and requirements,
add layout/tag runtime deps to requirements.txt, and restore full Apache
2.0 license headers on modularized files.
Format the new modules, teach mypy about layout/tag third-party imports,
exclude scripts/tags from mypy dual-module issues, and fix a few type
errors uncovered by the full pre-commit suite.
Remove the floss/qs product package and qs CLI/binary packaging. Move
layout analysis into floss/layout, string tagging into floss/tags (with
DBs still under floss/qs/db/data), and extract FLOSS argparse into
floss/cli. Promote former [qs] dependencies to core install deps.
Truncate entry diffs per library (100 in build_diff.txt, 20 in
build_diff_pr.txt for the PR description) and render the PR body
inside a ```diff fenced code block.
scripts/build_oss_db.py:
- Drop unused Iterable (typing) and field (dataclasses) imports.
- Drop unused env parameter from run(); none of the four call sites
pass it.
- Move 'import time' to the top of the module; no lazy-loading
justification.
- Clean up the awkward config.get('libraries',) call shape.
- Refuse to extract when the vcpkg .list file is missing instead of
silently scanning the entire triplet lib/ directory (which would
attribute other packages' strings/functions to the wrong library).
Log the candidate files we deliberately skipped.
- Fix exit-code logic: with --continue-on-error, return 1 when every
library failed so the CI step doesn't silently go green on a total
pipeline failure (broken vcpkg, wrong jh path, etc.).
tests/test_qs_build_oss_db.py:
- Add unit tests for merge_entries (new wins on collision, dedup
off, both empty, single-side cases).
- Add orchestration tests for main() that stub build_library/Vcpkg/
JHExtractor: per-library output, merge with pre-existing databases,
pre-existing libraries not in the current run are preserved,
--continue-on-error + partial success exits 0,
--continue-on-error + all failures exits 1, no --continue-on-error +
any failure exits 1, build_metrics.json is written with the
expected counts.
floss/qs/db/data/oss/readme.md:
- Add a short 'Why these build parameters' section that justifies
the x64-windows-static / msvc143 / release choice (PE target;
toolchain-version impact expected to be marginal) and documents
the cross-platform string-overlap findings (similar totals, low
per-string overlap due to ISA byte runs, section names, symbol
conventions). Stay Windows-only since FLOSS targets PE.
34 tests passing.
- Remove dead code: Converter.convert and the standalone count_jsonl_rows
helper are unused anywhere in the repo.
- Avoid double-parsing: fold object/function counting into Converter.parse
and return a ParseResult(entries, num_objects, num_functions). The old
code called count_jsonl_rows() and parse() back-to-back, each doing a
full json.loads pass over the same text.
- Consolidate entry schema: introduce make_db_entry() and route the four
call sites (3 in parse, 1 in load_existing_entries) through it so the
six-key dict shape is defined exactly once.
- Add tests/test_build_oss_db.py covering make_db_entry, Converter.parse
(counts, dedup behavior, emit_function_names, explicit function_name
rows, malformed lines, empty lines), and load_existing_entries
(round-trip, missing file, malformed lines, schema tolerance, empty
file). 22 tests, all passing.
Addresses part of #1310.
Addresses part of #1310.
- floss/qs/db/data/oss/readme.md: rewrite the build_oss_db.py paragraph
to reflect the current behavior (no cross-library dedup; non-rebuilt
libraries are not rewritten), and drop trailing whitespace.
- scripts/build_oss_db.py: JHExtractor.extract returns JSONL, not CSV;
update the docstring and the --no-deduplicate help text accordingly.
- floss/qs/db/data/oss/libraries.json: sort the library list A-Z for
stable, predictable ordering.
Strings that appear in 2+ libraries used to be dropped from every
database they showed up in. This silently removed legitimate
indicators when a dependency was vendored (e.g. zlib sources copied
into curl), because the vendored copy's strings would no longer point
back to zlib.
Now the same string can appear in any number of .jsonl.gz files. The
query tagger in floss/qs/main.py already emits one '#<library>' tag
per matching database, so the consumer can see the overlap directly
(a string tagged '#zlib #curl' is observed in both libraries) and
weight the indicator accordingly.
Removes:
- find_shared_strings() and the cross-library filter loops in main()
- the preserved_libraries rewrite block (only existed to handle
entries removed by cross-lib dedup)
- num_shared_removed / num_shared_strings_removed from metrics
Within-library dedup is unchanged: the same string still collapses
to one entry per library (unless --no-deduplicate is passed).
Strings that appeared in 2+ libraries used to be deleted from every
database, which dropped legitimate indicators when a dependency was
vendored (e.g. zlib sources copied into curl).
Instead, every entry is now stamped with the number of distinct
libraries the string appears in (OpenSourceString.count). count=1 means
the string is unique to one library; count>=2 indicates a shared
string that the consumer should weight accordingly. Strings are kept
in every database they appear in, so the vendored-dependency case
preserves the original zlib indicator.
The loader schema gains a defaulted 'count: int = 1' field, so legacy
.jsonl.gz files (no count) still decode and are interpreted as
count=1. Preserved (non-rebuilt) libraries are only rewritten when
their per-string counts actually change, so the workflow does not
produce spurious diffs on runs that don't add or remove libraries.
* Migrate Ghidra import script to PyGhidra with enhanced stack variable annotation
Updated the script for PyGhidra (Ghidra 12.0+) compatibility, migrating from
Jython (Python 2.7) to Python 3. The main improvement is enhanced local variable
commenting that now annotates both the stack variable definition and all
instruction references to it.
Key changes:
- Add PyGhidra compatibility for Ghidra 12.0+
- Implement find_stack_var_references() to track all references to stack variables
- Update append_lvar_comment() to add EOL comments at reference sites
- Refactor using template-based code generation for better maintainability
- Add type hints and comprehensive docstrings
- Support both Pydantic v1 and v2 for result parsing
This makes the generated script more effective at annotating stack strings
by showing decoded strings at the exact instructions where they are referenced,
rather than just at the function entry point.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
* Update scripts/render-ghidra-import-script.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* style: format ghidra import script with black
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Replace the header from source code files using the following script:
```Python
for dir_path, dir_names, file_names in os.walk("flare-floss"):
for file_name in file_names:
try:
file_path = f"{dir_path}/{file_name}"
f = open(file_path, "rb+")
content = f.read()
m = re.search(OLD_HEADER, content)
if not m:
continue
print(f"{file_path}: {m.group('year')}")
content = content.replace(m.group(0), NEW_HEADER % m.group("year"))
f.seek(0)
f.write(content)
except:
continue
```
Some files had the copyright headers inside a `"""` comment and needed
manual changes before applying the script.
The old header had the confusing sentence `All rights reserved`, which
does not make sense for an open source license. Replace the header by
the default Google header that corrects this issue and keep floss
consistent with other Google projects.
