25 Commits

Author SHA1 Message Date
Raykov-MS
d1daecdb31 Kinda fix 2026-10-09 17:35:36 +03:00
Raykov-MS
6229fe7814 Match the reference export on the card number and document prefix
Compared our report against the customer's own new-format export for
10.08.2026 (121 columns) over the 124 source files that produced it. Records
and participant rows line up exactly: 11393 records and 22839 rows on both
sides, no record with a different participant count. 90 of the 104 comparable
columns already matched value for value.

Two differences were ours to fix:

- Column 102, the card number in the foreign bank card block, took its value
  from СведенияВнесениеПолучениеНалДС/НомерКарта whenever the foreign card
  block was missing, because short lookup matches the tag НомерКарта in both
  blocks. The reference leaves the column empty there and shows the number in
  column 81 only, which is what we now do. 63 records on this export.
- The operation-document aggregate numbered its entries "1. ", the reference
  uses "1, ". That was 11377 rows out of 11393.

The remaining differences are the reference's own presentation of aggregate
cells and need the customer's word before touching them: it joins ФИО with
", " and keeps a trailing comma for a missing patronymic, spells the document
that proves identity as labelled lines ("Вид документа: 21\nСерия…") where we
use a comma list, and inserts the name of the region decoded from the ОКАТО
code into address cells, which the schema does not carry at all.

Two columns the reference fills and our form has no place for: the reporting
bank's own registration number, БИК and ОКАТО, and the authorised officer's
name, position, phone and e-mail. Both come from ИнфБанк and СлужЧасть.
2026-09-29 19:41:50 +03:00
Raykov-MS
69fd329f78 Print code lists with the separator the reference report uses
Verified our report against the Комита reference export by joining on the
record id. Our report_20260923_5.xlsx covers the 114 files of 30.01.2023 and
all 5266 of its records are inside the reference period, so the two can be
compared row by row.

Fifteen of the twenty-one shared columns already match exactly. The largest
remaining gap was the separator in the code lists: the reference prints
"1101,1106,1499" while the parser joins repeated tags with "; ", which put
336 of 5266 records out of step. Only the two code-list columns are brought
to the reference's separator; "; " stays in the free-text columns, where it
reads better and where the reference has no equivalent.

What the comparison also showed, for the record:
- The empty branch БИК and number on head-office records are correct. Those
  files carry no ИнфФилиал block at all and the reference leaves the cells
  empty too.
- Of the 105 address differences, 94 are the column 141 leak fixed in
  cceee02, and the remaining 11 are values that contain a comma, such as
  Пункт = "Южно-Сахалинск г, Хомутово пл.р-н", which the reference cannot
  represent in one comma-joined cell.
- Column 121 shows РегнНомер for people in private practice, where the
  reference's ОГРН column stays empty. That is the mapping table's fourth
  source for the column, not a defect: 3 records out of 13168.
2026-09-29 18:07:17 +03:00
Raykov-MS
cceee02243 Fill the report from the XML only, without the filtering rules
Every rule that emptied a column because the operation code, transfer type or
participant type was not in an allowed list is gone. What is left never loses
data: it assembles a column from several tags, substitutes the zero
placeholder into an empty account column, and clears the ЕИО/ЦП blocks when
the record has no such block at all.

Removed: rules 10, 11, 15 (code validity), 21, 22-24, 26, 28, 31, 36, 39/47,
40, 42-43, 119/122, 124, 125, 141/152/167/178, the ОКАТО-by-country rule, 163,
and the continuation-row validation, which only re-ran the three code checks.

Removing them exposed three defects the rules had been masking, all of them
the other half of the customer's requirement — a value must land in the column
of its own block:

- Columns 40 and 42 pulled the bank name from a foreign block. The tag НаимКО
  sits in six blocks, and short lookup matched the first one, so a record with
  only СведБанкПолучатель filled the payer's column too. 4180 and 6906 wrong
  values on the 03.02.2023 export. Fixed by forbidding short lookup, the same
  way columns 41 and 43 already do it.
- Column 141 pulled the participant address from the cash place block for the
  same reason. All 380 values it held were wrong; the 311 that remain now come
  from the participant's own address block, verified one to one against the
  XML.
- The one-line and structured address versions were treated as mutually
  exclusive: a filled one-line column wiped the structured column and every
  component. That cost 50 structured addresses on this export. Now both are
  kept, and only an exact duplicate of the one-line value is suppressed, so a
  single tag matching both columns is not echoed twice.

Three path typos in mapping_table.json also came out, checked against the
schema: УчастникФлип, СвебИНБОЮЛ and полноеНаимИНБОЮЛ. The first made the
participant one-line address unreachable by its scoped path, which is why it
depended on the leaking short lookup.

On the 03.02.2023 export only 12 columns change and nothing but column 141
goes down. The 5016 cross-check still reports no losses: every value in the
cash receipt, cash payment, status and account columns reaches the report.
2026-09-24 18:03:08 +03:00
Raykov-MS
880fa5811d Stop dropping XML values in the transfer and cash blocks
The customer's ruling of 24.09.2026 is that every value present in the XML
must reach the report, taking into account which block holds it. Two defects
broke that for operation code 5016 with transfer types 4 and 5.

Wrong paths in mapping_table.json. Columns 48-63 spelled the block as
"СведенияПереводыДс" and columns 49-59 spelled the address as
"АдрМестоПриемаВыдача"; the schema uses "СведенияПереводыДС" and
"АдрМестаПриемаВыдача". Those columns carry allow_short_lookup=false, so they
can only match on a fully scoped path and never matched at all: the whole cash
receipt block (AV-BI) and the transfer status (BJ) stayed empty on every file.
The existing test passed only because it fed the same misspellings.

Rules that deleted data instead of filling it:
- Rules 38, 44-45 and 46 wiped the account columns for code 5016. The XML of
  those records carries СчетБанкПолучатель = 00000000000000000000, which is
  exactly the value the customer asked for. The rules now only substitute the
  placeholder into an empty column, and never for 5016.
- Rule 44 also wiped the correspondent accounts for operator types 3 and 5
  even when the XML had them.
- The cashless rule wiped columns 50, 61, 65, 76, 84, 95, 99 and 101 for
  transfer types 1-4, 7 and 10-14, although type 4 is by definition a transfer
  that accepts cash and must carry the receipt block.
- Rule 62 limited the transfer status to types 2, 5 and 8.
- Rule 30 limited the metal name to code C99, while the available exports only
  ever carry code A90.

Verified on the sample exports: all 16 records with code 5016 now carry the
zeros in column 45, the real account in column 46, and no invented
placeholders in columns 38 and 44.
2026-09-24 14:35:37 +03:00
Raykov-MS
d07eedfafb Fix column 99: assemble foreign card address from structured components
The АдрМестаПриемаВыдача block under СведенияКартаИнБанк (code 1009) has no
single АдресСтрока tag, unlike the same-named block under
СведенияВнесениеПолучениеНалДС — column 99 was mapped directly onto that
nonexistent tag and stayed empty. It now assembles the value from the
individual address components (Индекс, КодОКСМ, КодСубъектаПоОКАТО, Район,
Пункт, Улица, Дом, Корп, Оф), the same way column 153 already does for
registration place. Also disabled allow_short_lookup for this column, since
it was previously able to pick up a same-named tag (e.g. Улица) from an
unrelated address elsewhere in the file.

Added regression tests at both the mapping level and a full parse_xml_content
-> build_fixed_row_by_index pipeline level with a synthetic code-1009 XML.
2026-09-18 11:09:31 +03:00
Raykov-MS
ccb92fd85f Fix Errors 2026-09-17 13:05:50 +03:00
Raykov-MS
f49a384e69 fix сдвиг columns 2026-08-28 10:30:20 +03:00
Raykov-MS
7beeb710ce fix 2026-08-25 17:17:05 +03:00
Raykov-MS
2e570fb386 fix errors 2026-08-25 14:56:20 +03:00
Raykov-MS
39ab4c4824 . 2026-08-24 11:53:23 +03:00
Raykov-MS
07056e6266 . 2026-08-24 11:44:32 +03:00
Raykov-MS
5935eceadf optimize 11.8x 2026-08-11 15:38:11 +03:00
Raykov-MS
510d668ab8 fix 2026-08-11 13:42:46 +03:00
Raykov-MS
bef38dedd8 fix 2026-08-11 13:08:00 +03:00
Raykov-MS
9968d84e3b fix 2026-08-11 01:09:07 +03:00
Raykov-MS
d1cb2dda71 fix bugs 2026-08-01 08:42:46 +03:00
Raykov-MS
eee68ab4e2 full rule 2026-07-31 11:37:58 +03:00
Raykov-MS
55f0312929 75-123 rule 2026-07-30 13:23:44 +03:00
Raykov-MS
25f96438b1 1-74 rule 2026-07-30 12:37:18 +03:00
Raykov-MS
a4379e5513 move to xlsxwriter 2026-07-29 10:28:17 +03:00
Raykov-MS
32e83c5b21 Правка отчета 2026-07-09 10:16:44 +03:00
Raykov-MS
4482179e82 optimization 2026-07-07 10:04:09 +03:00
Raykov-MS
27a11ae3ee fix 10% write in report to 95% 2026-07-06 13:27:10 +03:00
Raykov-MS
9cbfbd041f parser 2026-06-22 07:25:27 +03:00