build_fixed_row_by_index evaluated ~2 400 candidate paths per report row
(7 million _pick_value calls per 13 800 rows). A reverse index
"path -> (rule, candidate, probe)" is built once at import; per row we
walk only the record's own fields and keep the best hit per rule, with
the same candidate/probe ordering as _pick_value. Differential test
compares the index with _pick_value on random and parsed payloads.
Also: merged lookup reuses the operation lookup when the participant
payload is empty, and index-free paths skip the regex normalisation.
scripts/bench_report.py reproduces the measurement (0.42 -> 0.26 ms/row).
Refs #1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Compared our report against the customer's own new-format export for
10.08.2026 (121 columns) over the 124 source files that produced it. Records
and participant rows line up exactly: 11393 records and 22839 rows on both
sides, no record with a different participant count. 90 of the 104 comparable
columns already matched value for value.
Two differences were ours to fix:
- Column 102, the card number in the foreign bank card block, took its value
from СведенияВнесениеПолучениеНалДС/НомерКарта whenever the foreign card
block was missing, because short lookup matches the tag НомерКарта in both
blocks. The reference leaves the column empty there and shows the number in
column 81 only, which is what we now do. 63 records on this export.
- The operation-document aggregate numbered its entries "1. ", the reference
uses "1, ". That was 11377 rows out of 11393.
The remaining differences are the reference's own presentation of aggregate
cells and need the customer's word before touching them: it joins ФИО with
", " and keeps a trailing comma for a missing patronymic, spells the document
that proves identity as labelled lines ("Вид документа: 21\nСерия…") where we
use a comma list, and inserts the name of the region decoded from the ОКАТО
code into address cells, which the schema does not carry at all.
Two columns the reference fills and our form has no place for: the reporting
bank's own registration number, БИК and ОКАТО, and the authorised officer's
name, position, phone and e-mail. Both come from ИнфБанк and СлужЧасть.
Verified our report against the Комита reference export by joining on the
record id. Our report_20260923_5.xlsx covers the 114 files of 30.01.2023 and
all 5266 of its records are inside the reference period, so the two can be
compared row by row.
Fifteen of the twenty-one shared columns already match exactly. The largest
remaining gap was the separator in the code lists: the reference prints
"1101,1106,1499" while the parser joins repeated tags with "; ", which put
336 of 5266 records out of step. Only the two code-list columns are brought
to the reference's separator; "; " stays in the free-text columns, where it
reads better and where the reference has no equivalent.
What the comparison also showed, for the record:
- The empty branch БИК and number on head-office records are correct. Those
files carry no ИнфФилиал block at all and the reference leaves the cells
empty too.
- Of the 105 address differences, 94 are the column 141 leak fixed in
cceee02, and the remaining 11 are values that contain a comma, such as
Пункт = "Южно-Сахалинск г, Хомутово пл.р-н", which the reference cannot
represent in one comma-joined cell.
- Column 121 shows РегнНомер for people in private practice, where the
reference's ОГРН column stays empty. That is the mapping table's fourth
source for the column, not a defect: 3 records out of 13168.
Every rule that emptied a column because the operation code, transfer type or
participant type was not in an allowed list is gone. What is left never loses
data: it assembles a column from several tags, substitutes the zero
placeholder into an empty account column, and clears the ЕИО/ЦП blocks when
the record has no such block at all.
Removed: rules 10, 11, 15 (code validity), 21, 22-24, 26, 28, 31, 36, 39/47,
40, 42-43, 119/122, 124, 125, 141/152/167/178, the ОКАТО-by-country rule, 163,
and the continuation-row validation, which only re-ran the three code checks.
Removing them exposed three defects the rules had been masking, all of them
the other half of the customer's requirement — a value must land in the column
of its own block:
- Columns 40 and 42 pulled the bank name from a foreign block. The tag НаимКО
sits in six blocks, and short lookup matched the first one, so a record with
only СведБанкПолучатель filled the payer's column too. 4180 and 6906 wrong
values on the 03.02.2023 export. Fixed by forbidding short lookup, the same
way columns 41 and 43 already do it.
- Column 141 pulled the participant address from the cash place block for the
same reason. All 380 values it held were wrong; the 311 that remain now come
from the participant's own address block, verified one to one against the
XML.
- The one-line and structured address versions were treated as mutually
exclusive: a filled one-line column wiped the structured column and every
component. That cost 50 structured addresses on this export. Now both are
kept, and only an exact duplicate of the one-line value is suppressed, so a
single tag matching both columns is not echoed twice.
Three path typos in mapping_table.json also came out, checked against the
schema: УчастникФлип, СвебИНБОЮЛ and полноеНаимИНБОЮЛ. The first made the
participant one-line address unreachable by its scoped path, which is why it
depended on the leaking short lookup.
On the 03.02.2023 export only 12 columns change and nothing but column 141
goes down. The 5016 cross-check still reports no losses: every value in the
cash receipt, cash payment, status and account columns reaches the report.
The uploader took a single .zip, so a day split across several archives, or a
handful of loose XML files, needed one run each.
- main.py and scripts/local_zip_test_app.py now use accept_multiple_files and
take .xml alongside .zip and the tar family. Each attachment is written to
its own temp subdirectory, because two attachments may share a name.
- run_batch takes input_paths instead of zip_path.
- New app/pipeline/attachments.py flattens the attachments into a list of XML
files. Archive format is detected from the content rather than the suffix,
since exports arrive with the wrong extension often enough. .rar and .7z are
recognised only to explain why they are refused.
- A broken attachment no longer aborts the run. It becomes one protocol row
and the remaining attachments are still processed; the reason is also shown
in the UI, not only in the protocol file.
- Files repeated across attachments are processed once and the repeats are
recorded with status "skip". Attaching the same archive twice would
otherwise double every row of the report.
- Archives are unpacked member by member with the target path checked against
the extraction directory. tarfile.data_filter only exists from Python 3.11.4
and extractall would otherwise let a "../" member escape.
- The protocol gains a "Вложение" column, so it is clear which attachment a
file came from when names repeat across archives.
Checked end to end on the 03.02.2023 export: six attachments (two archives,
two loose XML files, one broken archive, one duplicate archive) give 57 files
processed, 9 repeats skipped, one attachment refused with its reason, and the
report is still produced.
The customer's ruling of 24.09.2026 is that every value present in the XML
must reach the report, taking into account which block holds it. Two defects
broke that for operation code 5016 with transfer types 4 and 5.
Wrong paths in mapping_table.json. Columns 48-63 spelled the block as
"СведенияПереводыДс" and columns 49-59 spelled the address as
"АдрМестоПриемаВыдача"; the schema uses "СведенияПереводыДС" and
"АдрМестаПриемаВыдача". Those columns carry allow_short_lookup=false, so they
can only match on a fully scoped path and never matched at all: the whole cash
receipt block (AV-BI) and the transfer status (BJ) stayed empty on every file.
The existing test passed only because it fed the same misspellings.
Rules that deleted data instead of filling it:
- Rules 38, 44-45 and 46 wiped the account columns for code 5016. The XML of
those records carries СчетБанкПолучатель = 00000000000000000000, which is
exactly the value the customer asked for. The rules now only substitute the
placeholder into an empty column, and never for 5016.
- Rule 44 also wiped the correspondent accounts for operator types 3 and 5
even when the XML had them.
- The cashless rule wiped columns 50, 61, 65, 76, 84, 95, 99 and 101 for
transfer types 1-4, 7 and 10-14, although type 4 is by definition a transfer
that accepts cash and must carry the receipt block.
- Rule 62 limited the transfer status to types 2, 5 and 8.
- Rule 30 limited the metal name to code C99, while the available exports only
ever carry code A90.
Verified on the sample exports: all 16 records with code 5016 now carry the
zeros in column 45, the real account in column 46, and no invented
placeholders in columns 38 and 44.
- Draw a grid on the two header rows only, data cells stay borderless.
- Align column names to the top so the start of a long name stays visible.
- Narrow the 14 code columns to the length of their values; the rule matches
by column name, so identical indicators in every block get equal widths.
- Halve both payment system name columns (32 -> 16).
- Give "Характеристика операции" the same width as "Назначение платежа".
- Drop the group title above "Статус перевода": it duplicated the column name.
The group cell stays in place so the header grid is not broken.
- Pass the cell format to write_string. Identifier columns (БИК, ИНН, ОГРН,
КПП, СВИФТ, account numbers) were written without one, so text wrapping and
vertical alignment never applied to them - this is why БИК had no line break.
The width padding and the header rotation are single constants: the customer
has not confirmed yet whether "N знаков" means the value length (implemented)
or an amount to subtract, and whether narrow headers should be rotated.
The АдрМестаПриемаВыдача block under СведенияКартаИнБанк (code 1009) has no
single АдресСтрока tag, unlike the same-named block under
СведенияВнесениеПолучениеНалДС — column 99 was mapped directly onto that
nonexistent tag and stayed empty. It now assembles the value from the
individual address components (Индекс, КодОКСМ, КодСубъектаПоОКАТО, Район,
Пункт, Улица, Дом, Корп, Оф), the same way column 153 already does for
registration place. Also disabled allow_short_lookup for this column, since
it was previously able to pick up a same-named tag (e.g. Улица) from an
unrelated address elsewhere in the file.
Added regression tests at both the mapping level and a full parse_xml_content
-> build_fixed_row_by_index pipeline level with a synthetic code-1009 XML.
Each call into opencloud_token_store is wrapped in its own asyncio.run()
(app/app_config.py, main.py's token popover), so every call gets a fresh
event loop. The asyncpg connection pool is bound to the loop it was created
in, so the next asyncio.run() could try to reuse a connection from an
already-closed loop and fail — unrelated to actual DB availability, but the
broad except Exception in resolve_opencloud_access_token() silently
swallowed it and fell through to "token not found", masking the real cause.
- add_token/get_token/remove_token/list_usernames now call engine.dispose()
in a finally block so each asyncio.run() starts with a clean pool.
- resolve_opencloud_access_token() now surfaces the real exception via
st.error() before falling back to the .env token (kept only for local dev
without a DB).
- Added tests/unit/test_opencloud_token_store.py covering that dispose() is
called after every store operation, including when commit() raises.
- Drop SMB/SAMBA subsystem completely: app/samba_access.py, app/path_settings.py,
app/path_store.py, app/pipeline/file_manager.py and their tests, plus the
smbprotocol dependency and all related UI (login/password fields, path editor,
SMB diagnostics) in main.py.
- run_batch now takes a .zip attached via Streamlit's file_uploader (same as the
local dev tool) instead of reading from SMB or a fictitious OpenCloud "Attach"
folder; renamed from run_local_zip_batch since it's now the single canonical
batch entrypoint used by both main.py and scripts/local_zip_test_app.py.
- Fixed run number resolution: added next_opencloud_run_number() which lists the
Out/year/month/day folder via WebDAV PROPFIND, so repeated runs land in
incrementing subfolders instead of always overwriting "1".
- Added a self-service "Добавить токен" popover in main.py so users can link
their own OpenCloud App Token to their Keycloak account (opencloud_token_store)
without an admin running the CLI script; reworded the missing-token error to
point at it.
Files no longer go into a shared test space with a guessed folder name.
Instead, resolve each user's personal drive id via GET /graph/v1.0/me/drives
(Basic Auth, same App Token as uploads) on every run, and upload into
Out/год/месяц/день/номер_запуска inside that drive, creating the path via
MKCOL since a fresh personal drive won't have it yet.
Removes the now-obsolete OPENCLOUD_SPACE_ID / resolve_opencloud_user_folder
and the shared-space smoke script.
