cceee02243
Every rule that emptied a column because the operation code, transfer type or participant type was not in an allowed list is gone. What is left never loses data: it assembles a column from several tags, substitutes the zero placeholder into an empty account column, and clears the ЕИО/ЦП blocks when the record has no such block at all. Removed: rules 10, 11, 15 (code validity), 21, 22-24, 26, 28, 31, 36, 39/47, 40, 42-43, 119/122, 124, 125, 141/152/167/178, the ОКАТО-by-country rule, 163, and the continuation-row validation, which only re-ran the three code checks. Removing them exposed three defects the rules had been masking, all of them the other half of the customer's requirement — a value must land in the column of its own block: - Columns 40 and 42 pulled the bank name from a foreign block. The tag НаимКО sits in six blocks, and short lookup matched the first one, so a record with only СведБанкПолучатель filled the payer's column too. 4180 and 6906 wrong values on the 03.02.2023 export. Fixed by forbidding short lookup, the same way columns 41 and 43 already do it. - Column 141 pulled the participant address from the cash place block for the same reason. All 380 values it held were wrong; the 311 that remain now come from the participant's own address block, verified one to one against the XML. - The one-line and structured address versions were treated as mutually exclusive: a filled one-line column wiped the structured column and every component. That cost 50 structured addresses on this export. Now both are kept, and only an exact duplicate of the one-line value is suppressed, so a single tag matching both columns is not echoed twice. Three path typos in mapping_table.json also came out, checked against the schema: УчастникФлип, СвебИНБОЮЛ and полноеНаимИНБОЮЛ. The first made the participant one-line address unreachable by its scoped path, which is why it depended on the leaking short lookup. On the 03.02.2023 export only 12 columns change and nothing but column 141 goes down. The 5016 cross-check still reports no losses: every value in the cash receipt, cash payment, status and account columns reaches the report.
SFM Streamlit
Production-ready Streamlit приложение для обработки ФЭС XML файлов в Excel-отчет и протокол с доступом к файлам через SAMBA (rail_connectors).
Архитектура
main.py
app/
app_config.py # константы приложения и user-id
path_store.py # JSON-хранилище персональных путей
path_settings.py # UI-модалка изменения пути
samba_access.py # проверка доступа к SAMBA папке
auth/
service.py # orchestration auth/session
providers.py
mock_provider.py
pipeline/
config.py # шаблоны имен/путей пайплайна
file_manager.py # scan/move/upload/download в SAMBA
parser.py # namespace-aware парсер ФЭС XML
report.py # генерация report.xlsx
protocol.py # генерация protocol.xlsx
service.py # run_batch orchestration
tests/
unit/
smoke/
Основной поток
- Пользователь вводит SAMBA логин/пароль.
- При необходимости меняет персональный
input_dir(с pre-check доступа). - Нажимает
Запустить обработку xml в excel. - Пайплайн:
- сканирует входную папку;
- невалидные имена переносит в
Err; - валидные XML парсит и формирует строки отчета;
- создает
report.xlsx,protocol.xlsx,process.log; - сохраняет артефакты в
Out/YYYY/MM_YYYY/YYYYMMDD/run_number/; - успешные файлы переносит в
Processed.
- В UI отображаются счетчики и пути к артефактам.
Переменные окружения
Используются переменные авторизации Raisa (OPENBAO__SETTINGS__*).
SMBA source и дефолтный путь входа задаются в коде (app/app_config.py).
Быстрый старт
python -m venv .venv
.\.venv\Scripts\activate
pip install -r requirements.txt
copy .env.example .env
streamlit run main.py
Тесты и качество
pytest
ruff check .
black --check .
isort --check-only .
Pre-commit
pre-commit install
pre-commit run --all-files
Хуки:
blackisortruffruff-format