Единая нормализация ББ/ОФР, ядро — часть А: DBML v5.19, очистка форм, агрегат по коду, правка формы, контракт 2.0 (arm-ka-docs#116) #129
19
README.md
19
README.md
@ -566,12 +566,29 @@ S3 (storage); без S3 — предупреждение в логе, цикл
|
||||
текущим суммам формы и не хранятся; словарь — `api/rules/statement_checks.yaml` (копия словаря tabular-parser — правится
|
||||
в обоих репозиториях).
|
||||
|
||||
- **`GET /documents/{id}/statement`** — `{document, edition, header, columns[], lines[], checks}`: `header` — `inn`,
|
||||
- **`GET /documents/{id}/statement`** — `{document, edition, header, columns[], lines[], checks, issues[], edition_lines[]}`: `header` — `inn`,
|
||||
`report_date`, `units` вида `{value, original_value, confidence, verified, modified}` (`confidence` есть у ИНН, у даты и единиц — `null`) и справочные поля формы
|
||||
(`organization_name`, `okud`, `okpo`, …); `lines[].values[]` — суммы строкой, `is_modified`, `original_amount`; `lines[].parent_code` — у пятизначной подстроки код её четырёхзначной строки-родителя (правило проверки «сумма подстрок = родитель», `domain/statement_checks.subline_parents`: ближайшая выше строка с теми же тремя цифрами, иначе строка с кодом из первых четырёх цифр; родитель считается по строкам, участвующим в проверках, — достроенные из справочника пустые строки родителями не бывают), у остальных строк `null`; поле вычисляется на лету, в БД не хранится; мягкая проверка «сумма подстрок = родитель» по колонке выполняется, только если хотя бы одна подстрока родителя в ней непустая (`amount` не `null`; ноль — значение), иначе не проверяется; `checks` —
|
||||
`{failed, soft_failed, cross_form, accepted}` (нарушение — `{column_index, relation, expected, actual}`, у межформенной
|
||||
проверки `column_index = -1`). Не форма (ОСВ, расшифровка, ББ/ОФР без данных формы — созданные до Среза 3b) — 409 `DOCUMENT_NOT_STATEMENT`.
|
||||
Подтверждённая форма читается. Форма достраивается по справочнику строк редакции (`api/rules/statement_lines.yaml`): недостающие строки (`id = null`, `origin = template`) и ячейки (`id = null`); в БД не пишется.
|
||||
- **Поля v5.19 в ответе `GET …/statement`** (change `unified-statement-normalization`, P3-11, P3-17, P3-20…P3-23):
|
||||
`document.form_variant`; `header.okei`, `header.needs_review_fields[]`, `header.organization_name_modified`,
|
||||
`header.source{поле: {text, confidence}}` (`header_source` или `{}`; `confidence` — строка с двумя знаками, `text` и
|
||||
`confidence` могут быть `null`); `columns[].period_kind`, `label_confidence`; `lines[]` — `row_index`, `source_ref`,
|
||||
`line_status` (`ok` / `detail` / `unreadable` / `missing`), `line_code_text`, `line_code_confidence`,
|
||||
`line_code_modified`, `reference_name` (первое имя кода в словаре редакции ответа, иначе `null`), `parent_row_index`,
|
||||
`needs_review`, `is_confirmed`; `lines[].values[]` — `text` (`source_text`), `needs_review`, `is_confirmed`;
|
||||
`checks.*[]` — `codes` и `line_ids` (строки `ok` кодов правила); `issues[]` — записи `document_issue` по
|
||||
`created_at, id`: `{id, kind, code, severity, message, line_ids, column_id, column_index, field, source, relation,
|
||||
codes, expected, actual}`; `edition_lines[] {code, name}` — справочник кодов для подсказки (состав формы редакции
|
||||
ответа по одному на код; у формы без даты — коды всех редакций). Порядок `lines[]`: строки `ok` (с достроенными
|
||||
шаблонными) по коду, затем `row_index` (`NULL` последними); за строкой `ok` источника — её вложения (`detail` с этим
|
||||
родителем, `unreadable` / `missing` / `detail` без родителя, у которых она — ближайшая предыдущая по `row_index`
|
||||
строка `ok`); вложения без такой строки — в начале (`domain/statement_layout.py`).
|
||||
Карточка документа (`GET /documents/{id}`) несёт `issues[]` (у документа любого типа) и `issue_counts {error, info}`;
|
||||
`documents[]` карточки файла — `issue_counts` (все виды записей по `severity`, в том числе акцептованные; один групповой
|
||||
запрос на список документов).
|
||||
- **`PATCH /documents/{id}/statement`** — `{header?: {inn?, report_date?, units?}, values?: [{line_id | line_code, column_id, amount}]}`
|
||||
(до 5000 ячеек; `amount: null` очищает ячейку; не более двух знаков; `units` — непустая строка после обрезки пробелов до 50 символов,
|
||||
пробельная строка — 422, `null` — «не менять»; `inn: null` — 422 `INVALID_INN`, ИНН не очищается). Правится только неподтверждённая форма
|
||||
|
||||
53
api/src/domain/statement_layout.py
Normal file
53
api/src/domain/statement_layout.py
Normal file
@ -0,0 +1,53 @@
|
||||
"""Порядок строк ответа `GET …/statement` (P3-17; спека `arm-core` «Ответ `GET …/statement`»). Чистая функция.
|
||||
|
||||
Строки `ok` (в том числе достроенные шаблонные и созданные правкой) — по коду, затем `row_index` (`NULL` последними),
|
||||
затем в порядке входа (вызывающий передаёт строки по `id`). За строкой `ok` источника идут её «вложения» по
|
||||
`row_index`: строки `detail` с этим родителем и строки `unreadable` / `missing` / `detail` без (известного) родителя,
|
||||
у которых она — ближайшая предыдущая по `row_index` строка `ok` источника. Вложения без такой строки — в начале.
|
||||
"""
|
||||
|
||||
import bisect
|
||||
from collections.abc import Hashable, Sequence
|
||||
from dataclasses import dataclass
|
||||
|
||||
__all__ = ["LayoutLine", "order_lines"]
|
||||
|
||||
OK = "ok"
|
||||
DETAIL = "detail"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LayoutLine:
|
||||
key: Hashable
|
||||
row_index: int | None
|
||||
line_status: str
|
||||
line_code: str | None
|
||||
parent_row_index: int | None
|
||||
|
||||
|
||||
def order_lines(lines: Sequence[LayoutLine]) -> list[Hashable]:
|
||||
"""Ключи строк в порядке ответа."""
|
||||
ok_lines = [line for line in lines if line.line_status == OK]
|
||||
others = [line for line in lines if line.line_status != OK]
|
||||
ok_ordered = sorted(
|
||||
ok_lines,
|
||||
key=lambda line: (line.line_code is None, line.line_code or "", line.row_index is None, line.row_index or 0),
|
||||
)
|
||||
# строки `ok` источника по `row_index` — адресаты вложений
|
||||
sources = sorted((line for line in ok_lines if line.row_index is not None), key=lambda line: line.row_index or 0)
|
||||
source_rows = [line.row_index or 0 for line in sources]
|
||||
by_row = {line.row_index: line.key for line in sources}
|
||||
attached: dict[Hashable | None, list[LayoutLine]] = {}
|
||||
for line in sorted((line for line in others), key=lambda item: (item.row_index is None, item.row_index or 0)):
|
||||
owner: Hashable | None = None
|
||||
if line.line_status == DETAIL and line.parent_row_index in by_row:
|
||||
owner = by_row[line.parent_row_index]
|
||||
elif line.row_index is not None:
|
||||
position = bisect.bisect_left(source_rows, line.row_index) - 1
|
||||
owner = sources[position].key if position >= 0 else None
|
||||
attached.setdefault(owner, []).append(line)
|
||||
result: list[Hashable] = [line.key for line in attached.get(None, [])]
|
||||
for line in ok_ordered:
|
||||
result.append(line.key)
|
||||
result.extend(item.key for item in attached.get(line.key, []))
|
||||
return result
|
||||
@ -56,6 +56,16 @@ class LineCatalog:
|
||||
return line.name
|
||||
return None
|
||||
|
||||
def edition_lines(self, form: str, edition: str | None) -> list[tuple[str, str]]:
|
||||
"""Справочник кодов для подсказки (P3-22): `(код, имя)` по одной на код, имя — первое. С редакцией — состав её
|
||||
формы в порядке словаря; без редакции — строки всех редакций по порядку (имя — из первой, где код есть)."""
|
||||
sources = [edition] if edition is not None else list(self.editions)
|
||||
result: dict[str, str] = {}
|
||||
for source in sources:
|
||||
for line in self._entries(source, form):
|
||||
result.setdefault(line.code, line.name)
|
||||
return list(result.items())
|
||||
|
||||
def names(self, form: str, code: str) -> tuple[str, ...]:
|
||||
"""Все имена кода во всех редакциях без повторов, в порядке появления."""
|
||||
result: dict[str, None] = {}
|
||||
|
||||
@ -4,6 +4,7 @@
|
||||
import datetime
|
||||
import uuid
|
||||
from collections.abc import Callable, Sequence
|
||||
from dataclasses import dataclass
|
||||
from decimal import Decimal
|
||||
from typing import Any
|
||||
|
||||
@ -12,7 +13,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from src.domain.statement_aggregate import AggLine, CodeAggregate, aggregate_by_code
|
||||
from src.domain.statement_mapping import StatementDraft
|
||||
from src.models.enums import DocumentStatus, DocumentType, FileStatus, StatementLineStatus
|
||||
from src.models.enums import DocumentStatus, DocumentType, FileStatus, IssueSeverity, StatementLineStatus
|
||||
from src.models.files import Document, StoredFile
|
||||
from src.models.statements import (
|
||||
DocumentIssue,
|
||||
@ -30,18 +31,32 @@ __all__ = [
|
||||
"column_ids",
|
||||
"delete_documents_statements",
|
||||
"delete_file_statements",
|
||||
"IssueCounts",
|
||||
"form_aggregate",
|
||||
"form_lines",
|
||||
"get_meta",
|
||||
"has_source_lines",
|
||||
"insert_statement",
|
||||
"issue_counts_by_document",
|
||||
"issues",
|
||||
"load_form",
|
||||
"lock_cells",
|
||||
"pair_amounts",
|
||||
"value_flags",
|
||||
]
|
||||
|
||||
BATCH = 1000 # значений на один INSERT: 8 колонок × 1000 < 32767 параметров asyncpg
|
||||
BATCH = 1000
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class IssueCounts:
|
||||
"""Число записей `document_issue` документа по `severity`."""
|
||||
|
||||
error: int = 0
|
||||
info: int = 0
|
||||
|
||||
|
||||
# значений на один INSERT: 8 колонок × 1000 < 32767 параметров asyncpg
|
||||
|
||||
|
||||
async def insert_statement(
|
||||
@ -69,6 +84,8 @@ async def insert_statement(
|
||||
"label": label,
|
||||
"period_date": period_date,
|
||||
"is_current": is_current,
|
||||
# до маппера A9 (план 3): как миграция v5.19 — `absolute` при дате, иначе `none`
|
||||
"period_kind": "absolute" if period_date is not None else "none",
|
||||
}
|
||||
)
|
||||
if columns:
|
||||
@ -408,3 +425,36 @@ async def pair_amounts(
|
||||
)
|
||||
income = await code_amount(session, income_meta.id, "2400", current=True) if income_meta is not None else None
|
||||
return balance, income
|
||||
|
||||
|
||||
async def issues(session: AsyncSession, document_id: uuid.UUID) -> list[DocumentIssue]:
|
||||
"""Записи `document_issue` документа по `created_at`, `id`."""
|
||||
stmt = (
|
||||
select(DocumentIssue)
|
||||
.where(DocumentIssue.document_id == document_id)
|
||||
.order_by(DocumentIssue.created_at, DocumentIssue.id)
|
||||
)
|
||||
return list((await session.execute(stmt)).scalars())
|
||||
|
||||
|
||||
async def issue_counts_by_document(
|
||||
session: AsyncSession, document_ids: Sequence[uuid.UUID]
|
||||
) -> dict[uuid.UUID, IssueCounts]:
|
||||
"""Число записей `document_issue` по `severity` одним запросом на список документов (P3-23); документа с записями
|
||||
нет в ответе (читать через `.get(id, IssueCounts())`)."""
|
||||
if not document_ids:
|
||||
return {}
|
||||
stmt = (
|
||||
select(DocumentIssue.document_id, DocumentIssue.severity, func.count(DocumentIssue.id))
|
||||
.where(DocumentIssue.document_id.in_(document_ids))
|
||||
.group_by(DocumentIssue.document_id, DocumentIssue.severity)
|
||||
)
|
||||
raw: dict[uuid.UUID, dict[IssueSeverity, int]] = {}
|
||||
for document_id, severity, count in (await session.execute(stmt)).all():
|
||||
raw.setdefault(document_id, {})[severity] = count
|
||||
return {
|
||||
document_id: IssueCounts(
|
||||
error=by_severity.get(IssueSeverity.error, 0), info=by_severity.get(IssueSeverity.info, 0)
|
||||
)
|
||||
for document_id, by_severity in raw.items()
|
||||
}
|
||||
|
||||
@ -21,6 +21,7 @@ from src.models.enums import (
|
||||
OsvGroupType,
|
||||
PeriodQuarter,
|
||||
)
|
||||
from src.schemas.issues import IssueCountsOut, IssueOut
|
||||
|
||||
DEFAULT_ENTRIES_LIMIT = 500
|
||||
MAX_ENTRIES_LIMIT = 2000
|
||||
@ -54,6 +55,8 @@ class DocumentOut(BaseModel):
|
||||
group_schema: list[GroupOut] | None
|
||||
warnings: list[str]
|
||||
comments_count: int = 0 # число комментариев документа (D-11)
|
||||
issues: list[IssueOut] = Field(default_factory=list) # v5.19: записи `document_issue` документа (любого типа)
|
||||
issue_counts: IssueCountsOut = Field(default_factory=IssueCountsOut) # P3-23
|
||||
|
||||
|
||||
class EntryOut(BaseModel):
|
||||
|
||||
@ -19,6 +19,7 @@ from src.models.enums import (
|
||||
PeriodQuarter,
|
||||
RecognitionStatus,
|
||||
)
|
||||
from src.schemas.issues import IssueCountsOut
|
||||
|
||||
|
||||
class AcceptedOut(BaseModel):
|
||||
@ -77,6 +78,7 @@ class DocumentBrief(BaseModel):
|
||||
counterparty_check: DocumentCheckResult | None
|
||||
locator: str | None
|
||||
comments_count: int = 0 # число комментариев документа (D-11); в ORM-модели поля нет — задаётся при сборке
|
||||
issue_counts: IssueCountsOut = IssueCountsOut() # P3-23; так же задаётся при сборке
|
||||
|
||||
|
||||
class FileOut(BaseModel):
|
||||
|
||||
56
api/src/schemas/issues.py
Normal file
56
api/src/schemas/issues.py
Normal file
@ -0,0 +1,56 @@
|
||||
"""Записи `document_issue` в API: `IssueOut` — общая для `StatementOut.issues` и `DocumentOut.issues`;
|
||||
`IssueCountsOut` — счётчики по `severity` для карточки файла (P3-23)."""
|
||||
|
||||
import uuid
|
||||
from decimal import Decimal
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
from src.models.enums import DocumentIssueKind, IssueSeverity, IssueSource
|
||||
from src.models.statements import DocumentIssue
|
||||
|
||||
|
||||
class IssueOut(BaseModel):
|
||||
id: uuid.UUID
|
||||
kind: DocumentIssueKind
|
||||
code: str
|
||||
severity: IssueSeverity
|
||||
message: str
|
||||
line_ids: list[uuid.UUID]
|
||||
column_id: uuid.UUID | None
|
||||
column_index: int | None
|
||||
field: str | None
|
||||
source: IssueSource
|
||||
relation: str | None
|
||||
codes: list[str]
|
||||
expected: Decimal | None
|
||||
actual: Decimal | None
|
||||
|
||||
|
||||
class IssueCountsOut(BaseModel):
|
||||
"""Число записей `document_issue` документа по `severity`, всех видов."""
|
||||
|
||||
error: int = 0
|
||||
info: int = 0
|
||||
|
||||
|
||||
def issue_out(issue: DocumentIssue) -> IssueOut:
|
||||
"""JSONB `line_ids` / `codes` (`NULL` — пусто) приводятся к спискам uuid и строк."""
|
||||
line_ids = issue.line_ids if isinstance(issue.line_ids, list) else []
|
||||
codes = issue.codes if isinstance(issue.codes, list) else []
|
||||
return IssueOut(
|
||||
id=issue.id,
|
||||
kind=issue.kind,
|
||||
code=issue.code,
|
||||
severity=issue.severity,
|
||||
message=issue.message,
|
||||
line_ids=[uuid.UUID(str(item)) for item in line_ids],
|
||||
column_id=issue.column_id,
|
||||
column_index=issue.column_index,
|
||||
field=issue.field,
|
||||
source=issue.source,
|
||||
relation=issue.relation,
|
||||
codes=[str(item) for item in codes],
|
||||
expected=issue.expected,
|
||||
actual=issue.actual,
|
||||
)
|
||||
@ -7,22 +7,35 @@ design D-5, D-14).
|
||||
|
||||
import datetime
|
||||
import uuid
|
||||
from collections.abc import Mapping, Sequence
|
||||
from decimal import Decimal
|
||||
from collections.abc import Callable, Mapping, Sequence
|
||||
from decimal import Decimal, InvalidOperation
|
||||
from typing import Annotated, Literal
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field, StringConstraints, model_validator
|
||||
|
||||
from src.domain.statement_aggregate import CodeAggregate
|
||||
from src.domain.statement_checks import ChecksReport, FailedCheck
|
||||
from src.domain.statement_lines import TemplateLine
|
||||
from src.models.enums import DocumentCheckResult, DocumentStatus, DocumentType, PeriodQuarter, StatementLineStatus
|
||||
from src.domain.statement_layout import LayoutLine, order_lines
|
||||
from src.domain.statement_lines import LineCatalog, TemplateLine
|
||||
from src.models.enums import (
|
||||
DocumentCheckResult,
|
||||
DocumentStatus,
|
||||
DocumentType,
|
||||
PeriodQuarter,
|
||||
StatementFormVariant,
|
||||
StatementLineStatus,
|
||||
)
|
||||
from src.models.files import Document
|
||||
from src.models.statements import (
|
||||
DocumentIssue,
|
||||
FinancialStatementColumn,
|
||||
FinancialStatementLine,
|
||||
FinancialStatementMeta,
|
||||
FinancialStatementValue,
|
||||
)
|
||||
from src.schemas.issues import IssueOut, issue_out
|
||||
|
||||
HUNDREDTHS = Decimal("0.01")
|
||||
|
||||
|
||||
class HeaderFieldOut(BaseModel):
|
||||
@ -35,6 +48,14 @@ class HeaderFieldOut(BaseModel):
|
||||
modified: bool | None
|
||||
|
||||
|
||||
class HeaderSourceOut(BaseModel):
|
||||
"""Подсказка источника для поля шапки (`header_source`): текст как в источнике и уверенность; `text: null` — нет
|
||||
подсказки."""
|
||||
|
||||
text: str | None = None
|
||||
confidence: Decimal | None = None
|
||||
|
||||
|
||||
class StatementHeaderOut(BaseModel):
|
||||
inn: HeaderFieldOut
|
||||
report_date: HeaderFieldOut
|
||||
@ -45,6 +66,10 @@ class StatementHeaderOut(BaseModel):
|
||||
okopf: str | None
|
||||
okfs: str | None
|
||||
address: str | None
|
||||
okei: str | None
|
||||
needs_review_fields: list[str]
|
||||
source: dict[str, HeaderSourceOut]
|
||||
organization_name_modified: bool
|
||||
|
||||
|
||||
class ColumnOut(BaseModel):
|
||||
@ -55,6 +80,8 @@ class ColumnOut(BaseModel):
|
||||
label: str | None
|
||||
period_date: datetime.date | None
|
||||
is_current: bool | None
|
||||
period_kind: str | None
|
||||
label_confidence: Decimal | None
|
||||
|
||||
|
||||
LineOrigin = Literal["source", "template"]
|
||||
@ -72,13 +99,25 @@ class ValueOut(BaseModel):
|
||||
original_amount: Decimal | None
|
||||
modified_by: uuid.UUID | None
|
||||
modified_at: datetime.datetime | None
|
||||
text: str | None # `source_text`: текст ячейки как в источнике
|
||||
needs_review: bool
|
||||
is_confirmed: bool
|
||||
|
||||
|
||||
class LineOut(BaseModel):
|
||||
id: uuid.UUID | None # None — строка достроена из справочника редакции, в БД её нет
|
||||
origin: LineOrigin
|
||||
row_index: int | None
|
||||
source_ref: str | None
|
||||
line_status: StatementLineStatus
|
||||
line_code: str | None
|
||||
line_code_text: str | None
|
||||
line_code_confidence: Decimal | None
|
||||
line_code_modified: bool
|
||||
reference_name: str | None # первое имя кода в словаре редакции ответа (P3-11)
|
||||
parent_row_index: int | None
|
||||
needs_review: bool
|
||||
is_confirmed: bool
|
||||
line_name: str | None
|
||||
line_name_confidence: Decimal | None
|
||||
# код четырёхзначной строки-родителя у пятизначной подстроки (вычисляется по `subline_parents`, в БД не хранится);
|
||||
@ -96,6 +135,8 @@ class FailedCheckOut(BaseModel):
|
||||
relation: str
|
||||
expected: Decimal
|
||||
actual: Decimal
|
||||
codes: list[str] = []
|
||||
line_ids: list[uuid.UUID] = []
|
||||
|
||||
|
||||
class ChecksAcceptedOut(BaseModel):
|
||||
@ -121,6 +162,14 @@ class StatementDocumentOut(BaseModel):
|
||||
counterparty_check: DocumentCheckResult | None
|
||||
locator: str | None
|
||||
file_id: uuid.UUID
|
||||
form_variant: StatementFormVariant | None
|
||||
|
||||
|
||||
class EditionLineOut(BaseModel):
|
||||
"""Строка справочника для подсказки кода (P3-22)."""
|
||||
|
||||
code: str
|
||||
name: str
|
||||
|
||||
|
||||
class StatementOut(BaseModel):
|
||||
@ -130,11 +179,20 @@ class StatementOut(BaseModel):
|
||||
columns: list[ColumnOut]
|
||||
lines: list[LineOut]
|
||||
checks: ChecksOut
|
||||
issues: list[IssueOut]
|
||||
edition_lines: list[EditionLineOut]
|
||||
|
||||
|
||||
def _failed(check: FailedCheck) -> FailedCheckOut:
|
||||
def _failed(check: FailedCheck, aggregates: Mapping[str, CodeAggregate]) -> FailedCheckOut:
|
||||
"""`line_ids` — строки `ok` кодов правила по агрегату формы (кода нет в форме — вклада нет)."""
|
||||
line_ids = [line_id for code in check.codes if code in aggregates for line_id in aggregates[code].line_ids]
|
||||
return FailedCheckOut(
|
||||
column_index=check.column_index, relation=check.relation, expected=check.expected, actual=check.actual
|
||||
column_index=check.column_index,
|
||||
relation=check.relation,
|
||||
expected=check.expected,
|
||||
actual=check.actual,
|
||||
codes=list(check.codes),
|
||||
line_ids=list(dict.fromkeys(line_ids)),
|
||||
)
|
||||
|
||||
|
||||
@ -151,6 +209,9 @@ def _value_out(value: FinancialStatementValue, column_index: dict[uuid.UUID, int
|
||||
original_amount=value.original_amount,
|
||||
modified_by=value.modified_by,
|
||||
modified_at=value.modified_at,
|
||||
text=value.source_text,
|
||||
needs_review=value.needs_review,
|
||||
is_confirmed=value.is_confirmed,
|
||||
)
|
||||
|
||||
|
||||
@ -173,29 +234,47 @@ def _empty_value(column: FinancialStatementColumn) -> ValueOut:
|
||||
original_amount=None,
|
||||
modified_by=None,
|
||||
modified_at=None,
|
||||
text=None,
|
||||
needs_review=False,
|
||||
is_confirmed=False,
|
||||
)
|
||||
|
||||
|
||||
def _lines_out(
|
||||
def _lines_out( # noqa: PLR0913, PLR0917 - сборка строк из частей формы
|
||||
columns: Sequence[FinancialStatementColumn],
|
||||
lines: Sequence[tuple[FinancialStatementLine, Sequence[FinancialStatementValue]]],
|
||||
template: Sequence[TemplateLine],
|
||||
parents: Mapping[str, str],
|
||||
reference_name: Callable[[str], str | None],
|
||||
) -> list[LineOut]:
|
||||
"""Строки БД с ячейками по всем колонкам + строки справочника, которых нет в форме (design D-3), по коду."""
|
||||
"""Строки БД с ячейками по всем колонкам + строки справочника, которых нет в форме (design D-3); порядок —
|
||||
`order_lines` (P3-17)."""
|
||||
column_index = {column.id: column.column_index for column in columns}
|
||||
out: list[LineOut] = []
|
||||
for line, values in lines:
|
||||
layout: list[LayoutLine] = []
|
||||
for line, values in sorted(lines, key=lambda item: str(item[0].id)):
|
||||
present = {value.column_id for value in values}
|
||||
cells = [_value_out(value, column_index) for value in values]
|
||||
cells += [_empty_value(column) for column in columns if column.id not in present]
|
||||
cells.sort(key=lambda cell: cell.column_index)
|
||||
layout.append(
|
||||
LayoutLine(len(out), line.row_index, str(line.line_status), line.line_code, line.parent_row_index)
|
||||
)
|
||||
out.append(
|
||||
LineOut(
|
||||
id=line.id,
|
||||
origin=line_origin(line),
|
||||
row_index=line.row_index,
|
||||
source_ref=line.source_ref,
|
||||
line_status=line.line_status,
|
||||
line_code=line.line_code,
|
||||
line_code_text=line.line_code_text,
|
||||
line_code_confidence=line.line_code_confidence,
|
||||
line_code_modified=bool(line.line_code_modified),
|
||||
reference_name=reference_name(line.line_code) if line.line_code is not None else None,
|
||||
parent_row_index=line.parent_row_index,
|
||||
needs_review=line.needs_review,
|
||||
is_confirmed=line.is_confirmed,
|
||||
line_name=line.line_name,
|
||||
line_name_confidence=line.line_name_confidence,
|
||||
parent_code=parents.get(line.line_code) if line.line_code is not None else None,
|
||||
@ -207,20 +286,50 @@ def _lines_out(
|
||||
for item in template:
|
||||
if item.code in codes:
|
||||
continue
|
||||
layout.append(LayoutLine(len(out), None, StatementLineStatus.ok.value, item.code, None))
|
||||
out.append(
|
||||
LineOut(
|
||||
id=None,
|
||||
origin="template",
|
||||
row_index=None,
|
||||
source_ref=None,
|
||||
line_status=StatementLineStatus.ok,
|
||||
line_code=item.code,
|
||||
line_code_text=None,
|
||||
line_code_confidence=None,
|
||||
line_code_modified=False,
|
||||
reference_name=item.name,
|
||||
parent_row_index=None,
|
||||
needs_review=False,
|
||||
is_confirmed=False,
|
||||
line_name=item.name,
|
||||
line_name_confidence=None,
|
||||
parent_code=None,
|
||||
values=[_empty_value(column) for column in columns],
|
||||
)
|
||||
)
|
||||
out.sort(key=lambda line: (line.line_code is None, line.line_code or ""))
|
||||
return out
|
||||
return [out[position] for position in order_lines(layout) if isinstance(position, int)]
|
||||
|
||||
|
||||
def _source_out(raw: object) -> dict[str, HeaderSourceOut]:
|
||||
"""`header_source` JSONB → подсказки по полям; число уверенности — строка с двумя знаками (P3-20), `text: null` и
|
||||
нечисловая уверенность — «нет подсказки» по своему полю."""
|
||||
if not isinstance(raw, dict):
|
||||
return {}
|
||||
result: dict[str, HeaderSourceOut] = {}
|
||||
for name, entry in raw.items():
|
||||
if not isinstance(entry, dict):
|
||||
continue
|
||||
text = entry.get("text")
|
||||
confidence: Decimal | None = None
|
||||
value = entry.get("confidence")
|
||||
if isinstance(value, int | float | str) and not isinstance(value, bool):
|
||||
try:
|
||||
confidence = Decimal(str(value)).quantize(HUNDREDTHS)
|
||||
except InvalidOperation:
|
||||
confidence = None
|
||||
result[str(name)] = HeaderSourceOut(text=text if isinstance(text, str) else None, confidence=confidence)
|
||||
return result
|
||||
|
||||
|
||||
def _accepted(meta: FinancialStatementMeta) -> ChecksAcceptedOut | None:
|
||||
@ -242,9 +351,13 @@ def statement_out( # noqa: PLR0913, PLR0917 - сборка ответа из ч
|
||||
edition: str | None,
|
||||
template: Sequence[TemplateLine],
|
||||
parents: Mapping[str, str],
|
||||
aggregates: Mapping[str, CodeAggregate],
|
||||
issues: Sequence[DocumentIssue],
|
||||
catalog: LineCatalog,
|
||||
) -> StatementOut:
|
||||
"""Ответ экрана проверки: документ, шапка, колонки, строки со значениями и проверки; строки достроены по
|
||||
справочнику редакции (D-3)."""
|
||||
form = document.document_type.value
|
||||
return StatementOut(
|
||||
document=StatementDocumentOut(
|
||||
id=document.id,
|
||||
@ -256,6 +369,7 @@ def statement_out( # noqa: PLR0913, PLR0917 - сборка ответа из ч
|
||||
counterparty_check=document.counterparty_check,
|
||||
locator=document.locator,
|
||||
file_id=document.stored_file_id,
|
||||
form_variant=meta.form_variant,
|
||||
),
|
||||
edition=edition,
|
||||
header=StatementHeaderOut(
|
||||
@ -284,15 +398,23 @@ def statement_out( # noqa: PLR0913, PLR0917 - сборка ответа из ч
|
||||
okopf=meta.okopf,
|
||||
okfs=meta.okfs,
|
||||
address=meta.address,
|
||||
okei=meta.okei,
|
||||
needs_review_fields=[str(item) for item in meta.needs_review_fields]
|
||||
if isinstance(meta.needs_review_fields, list)
|
||||
else [],
|
||||
source=_source_out(meta.header_source),
|
||||
organization_name_modified=bool(meta.organization_name_modified),
|
||||
),
|
||||
columns=[ColumnOut.model_validate(column) for column in columns],
|
||||
lines=_lines_out(columns, lines, template, parents),
|
||||
lines=_lines_out(columns, lines, template, parents, lambda code: catalog.reference_name(edition, form, code)),
|
||||
checks=ChecksOut(
|
||||
failed=[_failed(check) for check in report.failed],
|
||||
soft_failed=[_failed(check) for check in report.soft_failed],
|
||||
cross_form=_failed(cross) if cross is not None else None,
|
||||
failed=[_failed(check, aggregates) for check in report.failed],
|
||||
soft_failed=[_failed(check, aggregates) for check in report.soft_failed],
|
||||
cross_form=_failed(cross, aggregates) if cross is not None else None,
|
||||
accepted=_accepted(meta),
|
||||
),
|
||||
issues=[issue_out(issue) for issue in issues],
|
||||
edition_lines=[EditionLineOut(code=code, name=name) for code, name in catalog.edition_lines(form, edition)],
|
||||
)
|
||||
|
||||
|
||||
|
||||
@ -22,7 +22,7 @@ from src.common.exceptions import ConflictError, NotFoundError
|
||||
from src.domain.policy import VisibilityScope
|
||||
from src.models.access import AppUser
|
||||
from src.models.audit_ex import Comment
|
||||
from src.models.enums import AuditAction, AuditSection, DocumentStatus, FileStatus
|
||||
from src.models.enums import AuditAction, AuditSection, DocumentStatus, FileStatus, IssueSeverity
|
||||
from src.models.files import Document, RecognitionJob, StoredFile
|
||||
from src.models.osv import OsvEntry
|
||||
from src.models.statements import FinancialStatementMeta
|
||||
@ -31,6 +31,7 @@ from src.repositories import statements as statements_repo
|
||||
from src.schemas.comments import CommentAuthorOut, CommentIn, CommentOut
|
||||
from src.schemas.documents import DocumentOut, EntriesPageParams, GroupOut
|
||||
from src.schemas.files import DocumentBrief, FileOut, JobBrief
|
||||
from src.schemas.issues import IssueCountsOut, IssueOut, issue_out
|
||||
from src.services import consolidation_documents
|
||||
from src.services.result_mapping import MAPPING_ERROR_KEY, MAPPING_WARNINGS_KEY
|
||||
|
||||
@ -75,12 +76,16 @@ def file_out(
|
||||
job: RecognitionJob | None,
|
||||
documents: Sequence[Document],
|
||||
counts: Mapping[uuid.UUID, int] | None = None,
|
||||
issue_counts: Mapping[uuid.UUID, statements_repo.IssueCounts] | None = None,
|
||||
) -> FileOut:
|
||||
"""Карточка файла (spec documents-api «Файлы организации и карточка файла»).
|
||||
|
||||
`counts` — `document_id → число комментариев` (`files_repo.comments_count_by_document`); нет — нули.
|
||||
`issue_counts` — `document_id → число записей document_issue по severity`
|
||||
(`statements_repo.issue_counts_by_document`); нет — нули.
|
||||
"""
|
||||
counts = counts or {}
|
||||
issue_counts = issue_counts or {}
|
||||
return FileOut(
|
||||
id=file.id,
|
||||
legal_entity_id=file.legal_entity_id,
|
||||
@ -96,7 +101,12 @@ def file_out(
|
||||
consolidation_id=file.consolidation_id,
|
||||
job=job_brief(job) if job is not None else None,
|
||||
documents=[
|
||||
DocumentBrief.model_validate(document).model_copy(update={"comments_count": counts.get(document.id, 0)})
|
||||
DocumentBrief.model_validate(document).model_copy(
|
||||
update={
|
||||
"comments_count": counts.get(document.id, 0),
|
||||
"issue_counts": _counts_out(issue_counts.get(document.id)),
|
||||
}
|
||||
)
|
||||
for document in documents
|
||||
],
|
||||
)
|
||||
@ -113,9 +123,19 @@ def _group_schema(job: RecognitionJob | None, locator: str | None) -> list[Group
|
||||
return None
|
||||
|
||||
|
||||
def document_out(document: Document, job: RecognitionJob | None, comments_count: int = 0) -> DocumentOut:
|
||||
def _counts_out(counts: statements_repo.IssueCounts | None) -> IssueCountsOut:
|
||||
return IssueCountsOut(error=counts.error, info=counts.info) if counts is not None else IssueCountsOut()
|
||||
|
||||
|
||||
def document_out( # noqa: PLR0913, PLR0917 - карточка собирается из частей разных запросов
|
||||
document: Document,
|
||||
job: RecognitionJob | None,
|
||||
comments_count: int = 0,
|
||||
issues: Sequence[IssueOut] = (),
|
||||
issue_counts: statements_repo.IssueCounts | None = None,
|
||||
) -> DocumentOut:
|
||||
"""Карточка документа; `job` — текущее задание его файла (схема группировок и предупреждения маппинга),
|
||||
`comments_count` — число комментариев документа."""
|
||||
`comments_count` — число комментариев документа, `issues` — его записи `document_issue`."""
|
||||
return DocumentOut(
|
||||
id=document.id,
|
||||
file_id=document.stored_file_id,
|
||||
@ -135,6 +155,15 @@ def document_out(document: Document, job: RecognitionJob | None, comments_count:
|
||||
# ASSUMPTION(R-6): в DBML нет привязки предупреждения к документу — отдаются предупреждения маппинга файла
|
||||
warnings=_strings(_metrics(job).get(MAPPING_WARNINGS_KEY)),
|
||||
comments_count=comments_count,
|
||||
issues=list(issues),
|
||||
issue_counts=_counts_out(issue_counts),
|
||||
)
|
||||
|
||||
|
||||
def _issue_counts(issues: Sequence[Any]) -> statements_repo.IssueCounts:
|
||||
return statements_repo.IssueCounts(
|
||||
error=sum(1 for issue in issues if issue.severity is IssueSeverity.error),
|
||||
info=sum(1 for issue in issues if issue.severity is IssueSeverity.info),
|
||||
)
|
||||
|
||||
|
||||
@ -170,6 +199,16 @@ class DocumentService:
|
||||
counts = await files_repo.comments_count_by_document(self._session, [document_id])
|
||||
return counts.get(document_id, 0)
|
||||
|
||||
async def _card(self, document: Document, file: StoredFile | None) -> DocumentOut:
|
||||
issues = await statements_repo.issues(self._session, document.id)
|
||||
return document_out(
|
||||
document,
|
||||
await self._current_job(file),
|
||||
await self._comments_count(document.id),
|
||||
[issue_out(issue) for issue in issues],
|
||||
_issue_counts(issues),
|
||||
)
|
||||
|
||||
async def comments(self, document_id: uuid.UUID) -> list[CommentOut]:
|
||||
"""Комментарии видимого документа по `created_at` с ФИО автора; иначе 404."""
|
||||
document = await self._visible(document_id)
|
||||
@ -197,7 +236,7 @@ class DocumentService:
|
||||
"""Карточка видимого документа; иначе 404."""
|
||||
document = await self._visible(document_id)
|
||||
file = await self._session.get(StoredFile, document.stored_file_id)
|
||||
return document_out(document, await self._current_job(file), await self._comments_count(document.id))
|
||||
return await self._card(document, file)
|
||||
|
||||
async def entries(self, document_id: uuid.UUID, page: EntriesPageParams) -> tuple[list[OsvEntry], int]:
|
||||
"""Страница строк видимого документа по `row_number`; иначе 404."""
|
||||
@ -284,4 +323,4 @@ class DocumentService:
|
||||
before_value={"status": before.value},
|
||||
after_value={"status": DocumentStatus.CONFIRMED.value},
|
||||
)
|
||||
return document_out(document, await self._current_job(file), await self._comments_count(document.id))
|
||||
return await self._card(document, file)
|
||||
|
||||
@ -52,6 +52,7 @@ from src.models.files import RecognitionJob, S3Object, StoredFile
|
||||
from src.repositories import consolidations as consolidations_repo
|
||||
from src.repositories import directories as directories_repo
|
||||
from src.repositories import files as files_repo
|
||||
from src.repositories import statements as statements_repo
|
||||
from src.schemas.common import PageParams
|
||||
from src.schemas.files import FileOut
|
||||
from src.services import consolidation_documents
|
||||
@ -350,7 +351,10 @@ class PrimaryFilesService:
|
||||
)
|
||||
current = next((job for job in jobs if job.id == file.current_recognition_job_id), None)
|
||||
counts = await files_repo.comments_count_by_document(self._session, [document.id for document in documents])
|
||||
before = file_out(file, current, documents, counts).model_dump(mode="json")
|
||||
issue_counts = await statements_repo.issue_counts_by_document(
|
||||
self._session, [document.id for document in documents]
|
||||
)
|
||||
before = file_out(file, current, documents, counts, issue_counts).model_dump(mode="json")
|
||||
object_ids = [file.s3_object_id, *(job.result_s3_object_id for job in jobs if job.result_s3_object_id)]
|
||||
await files_repo.delete_file_rows(self._session, file)
|
||||
# M-3: только объекты папок этого файла — ссылка задания может указывать на объект другого файла
|
||||
@ -371,10 +375,10 @@ class PrimaryFilesService:
|
||||
async def _cards(self, files: list[StoredFile]) -> list[FileOut]:
|
||||
jobs = await files_repo.current_jobs(self._session, files)
|
||||
documents = await files_repo.documents_by_file(self._session, [file.id for file in files])
|
||||
counts = await files_repo.comments_count_by_document(
|
||||
self._session, [document.id for group in documents.values() for document in group]
|
||||
)
|
||||
return [file_out(file, jobs.get(file.id), documents.get(file.id, []), counts) for file in files]
|
||||
document_ids = [document.id for group in documents.values() for document in group]
|
||||
counts = await files_repo.comments_count_by_document(self._session, document_ids)
|
||||
issue_counts = await statements_repo.issue_counts_by_document(self._session, document_ids)
|
||||
return [file_out(file, jobs.get(file.id), documents.get(file.id, []), counts, issue_counts) for file in files]
|
||||
|
||||
async def find_duplicate(self, entity_id: uuid.UUID, sha256: str) -> StoredFile | None:
|
||||
"""Файл ЮЛ с тем же содержимым, кроме заменённых (`SUPERSEDED`): дубль по spec «Дедуп по содержимому»."""
|
||||
|
||||
@ -157,11 +157,23 @@ class StatementService:
|
||||
"""Ответ формы по текущему состоянию БД, достроенный по справочнику редакции (design D-3; используется и
|
||||
ответами правки/акцепта). В БД ничего не пишет."""
|
||||
columns, lines = await repo.load_form(self._session, meta.id)
|
||||
report, cross, parents = await self.checks_report(document, meta)
|
||||
_, aggregates = await form_aggregate(self._session, document, meta)
|
||||
report, cross, parents = await self._report(document, columns, aggregates)
|
||||
edition = edition_for(document.reporting_date)
|
||||
template = load_line_catalog().template_lines(edition, document.document_type.value)
|
||||
catalog = load_line_catalog()
|
||||
return schemas.statement_out(
|
||||
document, meta, columns, lines, report, cross, edition=edition, template=template, parents=parents
|
||||
document,
|
||||
meta,
|
||||
columns,
|
||||
lines,
|
||||
report,
|
||||
cross,
|
||||
edition=edition,
|
||||
template=catalog.template_lines(edition, document.document_type.value),
|
||||
parents=parents,
|
||||
aggregates=aggregates,
|
||||
issues=await repo.issues(self._session, document.id),
|
||||
catalog=catalog,
|
||||
)
|
||||
|
||||
async def hard_failures(self, document: Document, meta: FinancialStatementMeta) -> tuple[FailedCheck, ...]:
|
||||
|
||||
@ -140,6 +140,8 @@ async def test_document_card(seeded: AsyncSession, auth_client: AuthClientFactor
|
||||
# R-6: предупреждения маппинга файла целиком, без фильтра по листу
|
||||
"warnings": ["sheet=Лист_1: повтор", "sheet=Лист_2: сироты"],
|
||||
"comments_count": 0,
|
||||
"issues": [],
|
||||
"issue_counts": {"error": 0, "info": 0},
|
||||
}
|
||||
|
||||
|
||||
|
||||
@ -16,8 +16,18 @@ from sqlalchemy import event, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from src.models.directories import LegalEntity, RegionalOffice
|
||||
from src.models.enums import DocumentStatus, FileStatus, PeriodQuarter, RecognitionStatus
|
||||
from src.models.enums import (
|
||||
DocumentIssueKind,
|
||||
DocumentStatus,
|
||||
DocumentType,
|
||||
FileStatus,
|
||||
IssueSeverity,
|
||||
IssueSource,
|
||||
PeriodQuarter,
|
||||
RecognitionStatus,
|
||||
)
|
||||
from src.models.files import RecognitionJob, StoredFile
|
||||
from src.models.statements import DocumentIssue
|
||||
from tests.conftest import AuthClientFactory
|
||||
from tests.factories import make_company_group, make_document, make_legal_entity, make_stored_file
|
||||
|
||||
@ -143,6 +153,7 @@ async def test_list_processed_with_document_and_error(seeded: AsyncSession, auth
|
||||
"counterparty_check": None,
|
||||
"locator": "sheet:Лист_1",
|
||||
"comments_count": 0,
|
||||
"issue_counts": {"error": 0, "info": 0},
|
||||
}
|
||||
]
|
||||
|
||||
@ -280,3 +291,46 @@ async def test_unconfirmed_document_status_is_listed(seeded: AsyncSession, auth_
|
||||
body = (await client.get(f"/files/{file.id}")).json()
|
||||
|
||||
assert [document["status"] for document in body["documents"]] == ["NEEDS_CHECK"]
|
||||
|
||||
|
||||
# --- issue_counts (P3-23) --------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _issue(document_id: uuid.UUID, kind: DocumentIssueKind, severity: IssueSeverity) -> DocumentIssue:
|
||||
return DocumentIssue(
|
||||
document_id=document_id,
|
||||
kind=kind,
|
||||
code="X",
|
||||
severity=severity,
|
||||
message="тест",
|
||||
source=IssueSource.core,
|
||||
created_at=T0,
|
||||
)
|
||||
|
||||
|
||||
async def test_issue_counts_in_file_and_document(seeded: AsyncSession, auth_client: AuthClientFactory) -> None:
|
||||
"""Все виды записей считаются по `severity`; документ без записей — нули; запросов не больше, чем без документов."""
|
||||
entity = await _rostov_entity(seeded)
|
||||
file, job = await _file(seeded, entity, minutes=1)
|
||||
balance = await make_document(seeded, file=file, job=job, account="60")
|
||||
balance.document_type = DocumentType.BALANCE
|
||||
income = await make_document(seeded, file=file, job=job, account="62")
|
||||
income.document_type = DocumentType.INCOME
|
||||
seeded.add_all(
|
||||
[
|
||||
_issue(balance.id, DocumentIssueKind.warning, IssueSeverity.error),
|
||||
_issue(balance.id, DocumentIssueKind.relation, IssueSeverity.error),
|
||||
_issue(balance.id, DocumentIssueKind.soft, IssueSeverity.info),
|
||||
]
|
||||
)
|
||||
await seeded.flush()
|
||||
client = await auth_client(KA_RF)
|
||||
|
||||
card = (await client.get(f"/files/{file.id}")).json()
|
||||
counts = {item["id"]: item["issue_counts"] for item in card["documents"]}
|
||||
assert counts == {str(balance.id): {"error": 2, "info": 1}, str(income.id): {"error": 0, "info": 0}}
|
||||
listed = (await client.get(_files_url(entity.id))).json()["items"][0]
|
||||
assert {item["id"]: item["issue_counts"] for item in listed["documents"]} == counts
|
||||
document = (await client.get(f"/documents/{balance.id}")).json()
|
||||
assert document["issue_counts"] == {"error": 2, "info": 1}
|
||||
assert len(document["issues"]) == 3 # noqa: PLR2004
|
||||
|
||||
@ -7,6 +7,7 @@ import uuid
|
||||
from decimal import Decimal
|
||||
from http import HTTPStatus
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import delete, select, update
|
||||
@ -16,9 +17,21 @@ from src.domain.statement_checks import CROSS_FORM_RELATION
|
||||
from src.models.access import AppUser
|
||||
from src.models.audit_ex import AuditEvent
|
||||
from src.models.directories import CompanyGroup, LegalEntity, RegionalOffice
|
||||
from src.models.enums import AuditAction, DocumentStatus, DocumentType, FileStatus, PeriodQuarter, RecognitionStatus
|
||||
from src.models.enums import (
|
||||
AuditAction,
|
||||
DocumentIssueKind,
|
||||
DocumentStatus,
|
||||
DocumentType,
|
||||
FileStatus,
|
||||
IssueSeverity,
|
||||
IssueSource,
|
||||
PeriodQuarter,
|
||||
RecognitionStatus,
|
||||
StatementLineStatus,
|
||||
)
|
||||
from src.models.files import Document, StoredFile
|
||||
from src.models.statements import (
|
||||
DocumentIssue,
|
||||
FinancialStatementColumn,
|
||||
FinancialStatementLine,
|
||||
FinancialStatementMeta,
|
||||
@ -158,15 +171,44 @@ async def test_get_statement_shape_and_decimal_strings(
|
||||
"counterparty_check",
|
||||
"locator",
|
||||
"file_id",
|
||||
"form_variant",
|
||||
}
|
||||
assert {
|
||||
"organization_name",
|
||||
"okud",
|
||||
"okpo",
|
||||
"okopf",
|
||||
"okfs",
|
||||
"address",
|
||||
"okei",
|
||||
"needs_review_fields",
|
||||
"source",
|
||||
"organization_name_modified",
|
||||
} <= set(body["header"])
|
||||
assert set(body["columns"][0]) == {
|
||||
"id",
|
||||
"column_index",
|
||||
"label",
|
||||
"period_date",
|
||||
"is_current",
|
||||
"period_kind",
|
||||
"label_confidence",
|
||||
}
|
||||
assert {"organization_name", "okud", "okpo", "okopf", "okfs", "address"} <= set(body["header"])
|
||||
assert set(body["columns"][0]) == {"id", "column_index", "label", "period_date", "is_current"}
|
||||
line = next(line for line in body["lines"] if any(v["amount"] is not None for v in line["values"]))
|
||||
assert set(line) == {
|
||||
"id",
|
||||
"origin",
|
||||
"row_index",
|
||||
"source_ref",
|
||||
"line_status",
|
||||
"line_code",
|
||||
"line_code_text",
|
||||
"line_code_confidence",
|
||||
"line_code_modified",
|
||||
"reference_name",
|
||||
"parent_row_index",
|
||||
"needs_review",
|
||||
"is_confirmed",
|
||||
"line_name",
|
||||
"line_name_confidence",
|
||||
"parent_code",
|
||||
@ -186,6 +228,9 @@ async def test_get_statement_shape_and_decimal_strings(
|
||||
"original_amount",
|
||||
"modified_by",
|
||||
"modified_at",
|
||||
"text",
|
||||
"needs_review",
|
||||
"is_confirmed",
|
||||
}
|
||||
index_by_id = {column["id"]: column["column_index"] for column in body["columns"]}
|
||||
assert all(index_by_id[v["column_id"]] == v["column_index"] for line in body["lines"] for v in line["values"])
|
||||
@ -414,11 +459,15 @@ async def test_cross_form_violation_is_reported_on_both_forms(
|
||||
client = await auth_client(KA_RF)
|
||||
for document in (balance, income):
|
||||
body = (await client.get(f"/documents/{document.id}/statement")).json()
|
||||
own = "1370" if document is balance else "2400" # line_ids — строки той формы, что читается
|
||||
own_ids = [line["id"] for line in body["lines"] if line["line_code"] == own and line["id"] is not None]
|
||||
assert body["checks"]["cross_form"] == {
|
||||
"column_index": -1,
|
||||
"relation": CROSS_FORM_RELATION,
|
||||
"expected": "200.00",
|
||||
"actual": "150.00",
|
||||
"codes": ["1370", "2400"],
|
||||
"line_ids": own_ids,
|
||||
}, document.document_type
|
||||
# мягкая межформенная проверка не блокирует подтверждение: подмена 1370 ломает и жёсткие проверки самого ББ —
|
||||
# после их акцепта форма с нарушением 1370↔2400 подтверждается
|
||||
@ -1506,3 +1555,196 @@ async def test_origin_by_row_index(seeded: AsyncSession, fixtures_dir: Path, aut
|
||||
client = await auth_client(KA_RF)
|
||||
body = (await client.get(f"/documents/{document.id}/statement")).json()
|
||||
assert next(item for item in body["lines"] if item["id"] == str(line.id))["origin"] == "source"
|
||||
|
||||
|
||||
# --- v5.19: поля ответа, порядок строк, issues, справочник строк -----------------------------------------------------
|
||||
|
||||
|
||||
async def _meta(session: AsyncSession, document: Document) -> FinancialStatementMeta:
|
||||
return (
|
||||
await session.execute(select(FinancialStatementMeta).where(FinancialStatementMeta.document_id == document.id))
|
||||
).scalar_one()
|
||||
|
||||
|
||||
async def _statement(client: Any, document: Document) -> dict[str, Any]:
|
||||
response = await client.get(f"/documents/{document.id}/statement")
|
||||
assert response.status_code == HTTPStatus.OK, response.text
|
||||
return response.json()
|
||||
|
||||
|
||||
async def test_get_statement_v5_19_fields_on_1x_form(
|
||||
seeded: AsyncSession, fixtures_dir: Path, auth_client: AuthClientFactory
|
||||
) -> None:
|
||||
document = await form(seeded, fixtures_dir)
|
||||
body = await _statement(await auth_client(KA_RF), document)
|
||||
source = [line for line in body["lines"] if line["origin"] == "source"]
|
||||
assert all(line["line_status"] == "ok" and line["source_ref"] is None for line in source)
|
||||
assert sorted(line["row_index"] for line in source) == list(range(len(source)))
|
||||
assert all(not line["needs_review"] and not line["is_confirmed"] for line in body["lines"])
|
||||
assert all(
|
||||
line["row_index"] is None and line["line_status"] == "ok" for line in body["lines"] if line["id"] is None
|
||||
)
|
||||
cells = [value for line in source for value in line["values"] if value["id"] is not None]
|
||||
assert all(value["text"] == value["string_value"] for value in cells)
|
||||
assert body["document"]["form_variant"] is None
|
||||
assert body["header"]["needs_review_fields"] == [] and body["header"]["organization_name_modified"] is False
|
||||
assert {column["period_kind"] for column in body["columns"]} <= {"absolute", "none"}
|
||||
assert body["issues"] == []
|
||||
|
||||
|
||||
async def test_reference_name_first_name(
|
||||
seeded: AsyncSession, fixtures_dir: Path, auth_client: AuthClientFactory
|
||||
) -> None:
|
||||
document = await form(seeded, fixtures_dir)
|
||||
body = await _statement(await auth_client(KA_RF), document)
|
||||
assert body["edition"] == "fsbu4"
|
||||
line = next(line for line in body["lines"] if line["line_code"] == "1520")
|
||||
assert line["reference_name"] == "Кредиторская задолженность"
|
||||
|
||||
|
||||
async def test_issues_in_statement_and_document(
|
||||
seeded: AsyncSession, fixtures_dir: Path, auth_client: AuthClientFactory
|
||||
) -> None:
|
||||
document = await form(seeded, fixtures_dir)
|
||||
seeded.add(
|
||||
DocumentIssue(
|
||||
document_id=document.id,
|
||||
kind=DocumentIssueKind.warning,
|
||||
code="COLUMN_SKIPPED",
|
||||
severity=IssueSeverity.info,
|
||||
message="колонка пропущена",
|
||||
column_id=None,
|
||||
column_index=5,
|
||||
source=IssueSource.service,
|
||||
created_at=datetime.datetime(2026, 10, 18, tzinfo=datetime.UTC),
|
||||
)
|
||||
)
|
||||
await seeded.flush()
|
||||
client = await auth_client(KA_RF)
|
||||
expected = {
|
||||
"kind": "warning",
|
||||
"code": "COLUMN_SKIPPED",
|
||||
"severity": "info",
|
||||
"message": "колонка пропущена",
|
||||
"line_ids": [],
|
||||
"column_id": None,
|
||||
"column_index": 5,
|
||||
"field": None,
|
||||
"source": "service",
|
||||
"relation": None,
|
||||
"codes": [],
|
||||
"expected": None,
|
||||
"actual": None,
|
||||
}
|
||||
[issue] = (await _statement(client, document))["issues"]
|
||||
assert {key: value for key, value in issue.items() if key != "id"} == expected
|
||||
card = (await client.get(f"/documents/{document.id}")).json()
|
||||
assert [{key: value for key, value in item.items() if key != "id"} for item in card["issues"]] == [expected]
|
||||
assert card["issue_counts"] == {"error": 0, "info": 1}
|
||||
|
||||
|
||||
async def test_line_code_confidence_and_okei(
|
||||
seeded: AsyncSession, fixtures_dir: Path, auth_client: AuthClientFactory
|
||||
) -> None:
|
||||
document = await form(seeded, fixtures_dir)
|
||||
meta = await _meta(seeded, document)
|
||||
meta.okei = "384"
|
||||
line = (
|
||||
await seeded.execute(
|
||||
select(FinancialStatementLine).where(
|
||||
FinancialStatementLine.meta_id == meta.id, FinancialStatementLine.line_code == "1210"
|
||||
)
|
||||
)
|
||||
).scalar_one()
|
||||
line.line_code_confidence = Decimal("67.50")
|
||||
await seeded.flush()
|
||||
body = await _statement(await auth_client(KA_RF), document)
|
||||
assert body["header"]["okei"] == "384"
|
||||
assert next(item for item in body["lines"] if item["line_code"] == "1210")["line_code_confidence"] == "67.50"
|
||||
|
||||
|
||||
async def test_header_source_confidence_is_decimal_string(
|
||||
seeded: AsyncSession, fixtures_dir: Path, auth_client: AuthClientFactory
|
||||
) -> None:
|
||||
document = await form(seeded, fixtures_dir)
|
||||
meta = await _meta(seeded, document)
|
||||
meta.header_source = {
|
||||
"inn": {"text": "x", "confidence": 61.538464},
|
||||
"units": {"text": "тыс. руб.", "confidence": 100},
|
||||
"okpo": {"text": None, "confidence": None},
|
||||
}
|
||||
meta.needs_review_fields = ["inn"]
|
||||
await seeded.flush()
|
||||
header = (await _statement(await auth_client(KA_RF), document))["header"]
|
||||
assert header["source"]["inn"] == {"text": "x", "confidence": "61.54"}
|
||||
assert header["source"]["units"]["confidence"] == "100.00"
|
||||
assert header["source"]["okpo"] == {"text": None, "confidence": None}
|
||||
assert header["needs_review_fields"] == ["inn"]
|
||||
|
||||
|
||||
async def test_edition_lines(seeded: AsyncSession, fixtures_dir: Path, auth_client: AuthClientFactory) -> None:
|
||||
document = await form(seeded, fixtures_dir)
|
||||
client = await auth_client(KA_RF)
|
||||
body = await _statement(client, document)
|
||||
codes = [item["code"] for item in body["edition_lines"]]
|
||||
assert len(codes) == len(set(codes)) and "1120" not in codes and "1215" in codes
|
||||
assert {item["code"]: item["name"] for item in body["edition_lines"]}["1520"] == "Кредиторская задолженность"
|
||||
document.reporting_date = None # форма без даты — редакции нет
|
||||
await seeded.flush()
|
||||
codes = [item["code"] for item in (await _statement(client, document))["edition_lines"]]
|
||||
assert len(codes) == len(set(codes)) and "1120" in codes and "1215" in codes
|
||||
|
||||
|
||||
async def test_failed_check_has_codes_and_line_ids(
|
||||
seeded: AsyncSession, fixtures_dir: Path, auth_client: AuthClientFactory
|
||||
) -> None:
|
||||
document = await form(seeded, fixtures_dir)
|
||||
await set_amount(seeded, document, "1600", Decimal("1"), current=True)
|
||||
body = await _statement(await auth_client(KA_RF), document)
|
||||
failed = body["checks"]["failed"]
|
||||
assert failed, body["checks"]
|
||||
by_code = {line["line_code"]: line["id"] for line in body["lines"] if line["line_status"] == "ok"}
|
||||
for item in failed:
|
||||
assert "1600" in item["codes"] or item["codes"]
|
||||
assert item["line_ids"] and {by_code[code] for code in item["codes"] if code in by_code} >= set(
|
||||
item["line_ids"]
|
||||
)
|
||||
|
||||
|
||||
async def test_lines_order_follows_layout(
|
||||
seeded: AsyncSession, fixtures_dir: Path, auth_client: AuthClientFactory
|
||||
) -> None:
|
||||
"""Строки `ok` — по коду; нечитаемая строка источника идёт сразу за предшествующей строкой `ok`."""
|
||||
document = await form(seeded, fixtures_dir)
|
||||
meta = await _meta(seeded, document)
|
||||
anchor = (
|
||||
await seeded.execute(
|
||||
select(FinancialStatementLine).where(
|
||||
FinancialStatementLine.meta_id == meta.id, FinancialStatementLine.line_code == "1210"
|
||||
)
|
||||
)
|
||||
).scalar_one()
|
||||
seeded.add(
|
||||
FinancialStatementLine(
|
||||
meta_id=meta.id,
|
||||
line_code=None,
|
||||
row_index=(anchor.row_index or 0) * 1000 + 1,
|
||||
line_status=StatementLineStatus.unreadable,
|
||||
line_code_text="12?0",
|
||||
)
|
||||
)
|
||||
await seeded.flush()
|
||||
await seeded.execute(
|
||||
update(FinancialStatementLine)
|
||||
.where(
|
||||
FinancialStatementLine.meta_id == meta.id,
|
||||
FinancialStatementLine.line_status == StatementLineStatus.unreadable,
|
||||
)
|
||||
.values(row_index=(anchor.row_index or 0) * 1000 + 1)
|
||||
)
|
||||
lines = (await _statement(await auth_client(KA_RF), document))["lines"]
|
||||
ok_codes = [line["line_code"] for line in lines if line["line_status"] == "ok"]
|
||||
assert ok_codes == sorted(ok_codes)
|
||||
unreadable = next(i for i, line in enumerate(lines) if line["line_status"] == "unreadable")
|
||||
assert unreadable > 0
|
||||
assert lines[unreadable]["line_code_text"] == "12?0"
|
||||
|
||||
46
api/tests/test_domain_statement_layout.py
Normal file
46
api/tests/test_domain_statement_layout.py
Normal file
@ -0,0 +1,46 @@
|
||||
"""Порядок строк ответа `GET …/statement` (P3-17): строки `ok` по коду, вложения источника — за своей строкой."""
|
||||
|
||||
from src.domain.statement_layout import LayoutLine, order_lines
|
||||
|
||||
|
||||
def L( # noqa: N802 - короткая фабрика строки для таблиц тестов
|
||||
key: str, row_index: int | None, status: str, code: str | None, parent: int | None
|
||||
) -> LayoutLine:
|
||||
return LayoutLine(key=key, row_index=row_index, line_status=status, line_code=code, parent_row_index=parent)
|
||||
|
||||
|
||||
def test_order_lines_attachments() -> None:
|
||||
lines = [
|
||||
L("m0", 0, "missing", None, None),
|
||||
L("1230", 1, "ok", "1230", None),
|
||||
L("u2", 2, "unreadable", None, None),
|
||||
L("1210", 3, "ok", "1210", None),
|
||||
L("d4", 4, "detail", None, 3),
|
||||
L("t1250", None, "ok", "1250", None),
|
||||
L("2410b", 9, "ok", "2410", None),
|
||||
L("2410a", 6, "ok", "2410", None),
|
||||
]
|
||||
assert order_lines(lines) == ["m0", "1210", "d4", "1230", "u2", "t1250", "2410a", "2410b"]
|
||||
|
||||
|
||||
def test_detail_without_parent_follows_previous_ok() -> None:
|
||||
assert order_lines([L("1110", 0, "ok", "1110", None), L("d1", 1, "detail", None, None)]) == ["1110", "d1"]
|
||||
|
||||
|
||||
def test_detail_with_unknown_parent_falls_back_to_previous_ok() -> None:
|
||||
lines = [L("1110", 0, "ok", "1110", None), L("d5", 5, "detail", None, 4)]
|
||||
assert order_lines(lines) == ["1110", "d5"]
|
||||
|
||||
|
||||
def test_attachments_keep_row_order() -> None:
|
||||
lines = [
|
||||
L("u3", 3, "unreadable", None, None),
|
||||
L("1110", 0, "ok", "1110", None),
|
||||
L("d2", 2, "detail", None, 0),
|
||||
L("m1", 1, "missing", None, None),
|
||||
]
|
||||
assert order_lines(lines) == ["1110", "m1", "d2", "u3"]
|
||||
|
||||
|
||||
def test_empty() -> None:
|
||||
assert order_lines([]) == []
|
||||
@ -101,3 +101,22 @@ def test_invalid_catalog_is_value_error(tmp_path: Path, body: str, message: str)
|
||||
path.write_text(body, encoding="utf-8")
|
||||
with pytest.raises(ValueError, match=message):
|
||||
load_line_catalog(path)
|
||||
|
||||
|
||||
def test_edition_lines_one_per_code_first_name() -> None:
|
||||
lines = CATALOG.edition_lines("BALANCE", "fsbu4")
|
||||
codes = [code for code, _ in lines]
|
||||
assert len(codes) == len(set(codes)) and "1120" not in codes
|
||||
assert dict(lines)["1520"] == "Кредиторская задолженность"
|
||||
assert codes == [line.code for line in CATALOG.template_lines("fsbu4", "BALANCE")]
|
||||
|
||||
|
||||
def test_edition_lines_without_edition_merges_editions() -> None:
|
||||
lines = CATALOG.edition_lines("BALANCE", None)
|
||||
codes = [code for code, _ in lines]
|
||||
assert len(codes) == len(set(codes))
|
||||
assert "1120" in codes and "1215" in codes # 1120 — редакция 66n, 1215 — fsbu4
|
||||
assert (
|
||||
dict(lines)["1120"] == dict((line.code, line.name) for line in CATALOG.template_lines("66n", "BALANCE"))["1120"]
|
||||
)
|
||||
assert CATALOG.edition_lines("OSV", None) == []
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user