build_fixed_row_by_index evaluated ~2 400 candidate paths per report row
(7 million _pick_value calls per 13 800 rows). A reverse index
"path -> (rule, candidate, probe)" is built once at import; per row we
walk only the record's own fields and keep the best hit per rule, with
the same candidate/probe ordering as _pick_value. Differential test
compares the index with _pick_value on random and parsed payloads.
Also: merged lookup reuses the operation lookup when the participant
payload is empty, and index-free paths skip the regex normalisation.
scripts/bench_report.py reproduces the measurement (0.42 -> 0.26 ms/row).
Refs #1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The uploader took a single .zip, so a day split across several archives, or a
handful of loose XML files, needed one run each.
- main.py and scripts/local_zip_test_app.py now use accept_multiple_files and
take .xml alongside .zip and the tar family. Each attachment is written to
its own temp subdirectory, because two attachments may share a name.
- run_batch takes input_paths instead of zip_path.
- New app/pipeline/attachments.py flattens the attachments into a list of XML
files. Archive format is detected from the content rather than the suffix,
since exports arrive with the wrong extension often enough. .rar and .7z are
recognised only to explain why they are refused.
- A broken attachment no longer aborts the run. It becomes one protocol row
and the remaining attachments are still processed; the reason is also shown
in the UI, not only in the protocol file.
- Files repeated across attachments are processed once and the repeats are
recorded with status "skip". Attaching the same archive twice would
otherwise double every row of the report.
- Archives are unpacked member by member with the target path checked against
the extraction directory. tarfile.data_filter only exists from Python 3.11.4
and extractall would otherwise let a "../" member escape.
- The protocol gains a "Вложение" column, so it is clear which attachment a
file came from when names repeat across archives.
Checked end to end on the 03.02.2023 export: six attachments (two archives,
two loose XML files, one broken archive, one duplicate archive) give 57 files
processed, 9 repeats skipped, one attachment refused with its reason, and the
report is still produced.
- Drop SMB/SAMBA subsystem completely: app/samba_access.py, app/path_settings.py,
app/path_store.py, app/pipeline/file_manager.py and their tests, plus the
smbprotocol dependency and all related UI (login/password fields, path editor,
SMB diagnostics) in main.py.
- run_batch now takes a .zip attached via Streamlit's file_uploader (same as the
local dev tool) instead of reading from SMB or a fictitious OpenCloud "Attach"
folder; renamed from run_local_zip_batch since it's now the single canonical
batch entrypoint used by both main.py and scripts/local_zip_test_app.py.
- Fixed run number resolution: added next_opencloud_run_number() which lists the
Out/year/month/day folder via WebDAV PROPFIND, so repeated runs land in
incrementing subfolders instead of always overwriting "1".
- Added a self-service "Добавить токен" popover in main.py so users can link
their own OpenCloud App Token to their Keycloak account (opencloud_token_store)
without an admin running the CLI script; reworded the missing-token error to
point at it.
Files no longer go into a shared test space with a guessed folder name.
Instead, resolve each user's personal drive id via GET /graph/v1.0/me/drives
(Basic Auth, same App Token as uploads) on every run, and upload into
Out/год/месяц/день/номер_запуска inside that drive, creating the path via
MKCOL since a fresh personal drive won't have it yet.
Removes the now-obsolete OPENCLOUD_SPACE_ID / resolve_opencloud_user_folder
and the shared-space smoke script.
