diff --git a/README.md b/README.md index 4af0eda..e6ba629 100644 --- a/README.md +++ b/README.md @@ -61,22 +61,6 @@ Requires FCCS to be open with export destination already configured. Automates t - **Defensive** -- one bad drawer won't crash the entire run - **Crash recovery** -- some documents (e.g. UltraTax "Diagnostics" files) crash FCCS's converter (`FileConversionEngine::convert() failed`), which aborts that drawer's export. The script detects the error dialog, screenshots and logs the crashing document, dismisses it, and records the drawer in `crashed.txt` so it's skipped on future runs instead of stalling. Handle crashed drawers manually (export them excluding the poison document); delete a line from `crashed.txt` to retry after fixing. -### Step 2b (optional): Verify Export Completeness - -``` -python fccs_verify.py -``` - -During export, each drawer's document list is captured from the FCCS dialog and saved as a manifest. This script compares those manifests against the actual exported files to flag any drawers with missing or extra files. - -To spot-check specific drawers (e.g. ones the log marked failed, to see whether they actually finished exporting in the background), run: - -``` -python fccs_check.py -``` - -It prompts for one or more drawer IDs and reports, per drawer, which manifest documents are present vs missing. It accounts for FCCS page-splitting (a document exported as `Name Page 1`, `Name Page 2`, … counts as present) and for filename sanitization (titles containing characters illegal in filenames, like `:`, still match). - ### Step 3: Reorganize Files ``` @@ -105,6 +89,32 @@ output/ Exported filenames follow the format `{drawer_id}_{client_name}_{folder_name}_{creation_date}_{document_name}.ext`. The parser uses folder templates from `fccs_folders.txt` (with `YYYY` expanded via regex) and the creation date (`MM-DD-YYYY`) as anchors to reliably split the underscore-delimited fields. Folder names are decomposed into nested paths that match the FCCS UI structure (e.g. `2025 Tax Documents` becomes `Tax Documents/2025/`). UltraTax CS folders are matched by a built-in pattern. +### Step 4 (optional): Verify Export Completeness + +During export, each drawer's document list is captured from the FCCS dialog and saved as a manifest (in `manifest_dir`). These tools compare the manifests against the files actually in the export folder to confirm nothing was missed. + +Both compare at the **document level** and share identical matching logic. They account for: + +- **Page-splitting** -- a multi-page document exported as `Name Page 1`, `Name Page 2`, … counts as that one document being present. +- **Filename sanitization** -- document titles containing characters illegal in Windows filenames (e.g. `:` `/` `?`) still match the exported files. + +Batch-check every drawer that has a manifest: + +``` +python fccs_verify.py +``` + +Reports each drawer as `OK` or `INCOMPLETE` (listing the missing documents), plus a summary and any exported drawers that have no manifest. + +Spot-check specific drawers interactively (e.g. ones the log marked failed, to see whether they actually finished exporting in the background): + +``` +python fccs_check.py +Drawer ID(s): 08097 18430 +``` + +> Note: because page-splitting means the number of files can't be mapped one-to-one to documents, completeness is judged by document *presence* (is each manifest document represented by at least one exported file), not by exact file counts. + ## Config Reference All scripts read from `config.ini` (or specify `--config path\to\config.ini`). diff --git a/__pycache__/fccs_check.cpython-313.pyc b/__pycache__/fccs_check.cpython-313.pyc index 2fe6db5..0799fbc 100644 Binary files a/__pycache__/fccs_check.cpython-313.pyc and b/__pycache__/fccs_check.cpython-313.pyc differ diff --git a/__pycache__/fccs_config.cpython-313.pyc b/__pycache__/fccs_config.cpython-313.pyc index 413bac7..c2e53a7 100644 Binary files a/__pycache__/fccs_config.cpython-313.pyc and b/__pycache__/fccs_config.cpython-313.pyc differ diff --git a/__pycache__/fccs_verify.cpython-313.pyc b/__pycache__/fccs_verify.cpython-313.pyc index 684a9ef..a41e3e0 100644 Binary files a/__pycache__/fccs_verify.cpython-313.pyc and b/__pycache__/fccs_verify.cpython-313.pyc differ diff --git a/fccs_check.py b/fccs_check.py index b5bb608..60e6f02 100644 --- a/fccs_check.py +++ b/fccs_check.py @@ -6,7 +6,7 @@ its manifest (the documents FCCS said it would export, captured during Step 2) against the files actually sitting in the export folder, and reports any missing documents. -Handles two quirks of the FCCS export: +Handles two quirks of the FCCS export (see fccs_config.evaluate_drawer): 1. Page-splitting: a single manifest document (e.g. "Donations") is exported as one file if it's a single page, or as "Donations Page 1", @@ -14,10 +14,11 @@ Handles two quirks of the FCCS export: that one document being present. 2. Filename sanitization: FCCS strips characters that are illegal in Windows - filenames (: / \\ ? * " < > |) from document titles, so a manifest title - like 'US Tax Return (... 01:35PM)' won't match the exported file - character-for-character. Comparison is done on a normalized key - (lowercase, alphanumerics only) so these still match. + filenames (: / \\ ? * " < > |) from document titles, so comparison is done + on a normalized key so these still match. + +fccs_verify.py applies this same logic in batch across every drawer; this tool +is for spot-checking specific drawers (e.g. ones the log marked failed). USAGE ----- @@ -29,94 +30,37 @@ import os import re import sys -from fccs_config import parse_args, load_config -from fccs_verify import load_manifest # reuse the both-format manifest loader +from fccs_config import ( + parse_args, load_config, evaluate_drawer, index_files_by_drawer, +) -# Trailing " Page N" (optionally "Page N of M") appended to multi-page exports. -_PAGE_RE = re.compile(r"\s*Page\s+\d+(?:\s+of\s+\d+)?\s*$", re.IGNORECASE) -# Creation-date field ("_MM-DD-YYYY_") that precedes the document name. -_DATE_ANCHOR = re.compile(r"_\d{2}-\d{2}-\d{4}_") +def report_drawer(drawer_id, files, manifest_dir, out): + """Evaluate one drawer and print a human-readable completeness report.""" + r = evaluate_drawer(drawer_id, files, manifest_dir) - -def match_key(name): - """Normalize a document name for tolerant comparison. - - Strips a trailing 'Page N' page-split suffix, then reduces to lowercase - alphanumerics so punctuation and filename-sanitization differences don't - cause false mismatches. - """ - base = _PAGE_RE.sub("", name) - return re.sub(r"[^a-z0-9]+", "", base.lower()) - - -def manifest_doc_names(path, drawer_id): - """Return the expected document (Page Title) names from a manifest file.""" - rows = load_manifest(path) - names = [] - for row in rows: - if len(row) >= 3 and row[0].strip() == drawer_id: - names.append(row[1]) # old format: DrawerID, PageTitle, Application - elif row: - names.append(row[0]) # new format: PageTitle, Application - return names - - -def exported_doc_name(filename): - """Extract the document-name portion from an exported filename, or None. - - Format: {drawer}_{client}_{folder}_{MM-DD-YYYY}_{docname}.ext - The creation-date field is a reliable anchor; the doc name follows the last - one (the client/folder fields don't carry an "_MM-DD-YYYY_" pattern). - """ - stem = os.path.splitext(filename)[0] - anchors = list(_DATE_ANCHOR.finditer(stem)) - if not anchors: - return None - return stem[anchors[-1].end():] - - -def check_drawer(drawer_id, files, manifest_dir, out): - """Report completeness of one drawer's export.""" - manifest_path = os.path.join(manifest_dir, drawer_id + ".txt") - if not os.path.exists(manifest_path): - out("") - out(f"[{drawer_id}] NO MANIFEST at {manifest_path} — cannot verify " + out("") + if not r["has_manifest"]: + out(f"[{drawer_id}] NO MANIFEST at {r['manifest_path']} — cannot verify " "(was this drawer exported by the tool?)") return - expected = manifest_doc_names(manifest_path, drawer_id) - - # Group exported files by normalized doc name; page-splits collapse together. - exported = {} # key -> list of full doc names (one entry per file/page) - unparsed = [] - for f in files: - doc = exported_doc_name(f) - if doc is None: - unparsed.append(f) - continue - exported.setdefault(match_key(doc), []).append(doc) - - missing = [name for name in expected if match_key(name) not in exported] - - expected_keys = {match_key(n) for n in expected} - extras = [names[0] for k, names in exported.items() if k not in expected_keys] - - out("") + expected = r["expected"] + missing = r["missing"] out(f"[{drawer_id}] manifest lists {len(expected)} document(s); " - f"{len(files)} file(s) in export folder.") + f"{r['file_count']} file(s) in export folder.") if missing: out(f" INCOMPLETE — {len(missing)} document(s) missing from export:") for m in missing: out(f" - {m}") else: out(f" COMPLETE — all {len(expected)} manifest document(s) present.") - if extras: - out(f" Note: {len(extras)} exported document(s) not in the manifest:") - for e in extras: + if r["extras"]: + out(f" Note: {len(r['extras'])} exported document(s) not in the manifest:") + for e in r["extras"]: out(f" - {e}") - if unparsed: - out(f" Note: {len(unparsed)} file(s) had no recognizable date anchor " + if r["unparsed"]: + out(f" Note: {r['unparsed']} file(s) had no recognizable date anchor " "and were skipped.") @@ -130,14 +74,7 @@ def main(): print(f"ERROR: export directory not found: {export_dir}") sys.exit(1) - # Index every export file by its leading drawer-ID token (before first "_"). - # The underscore boundary keeps clashing IDs separate (04289 vs 04289TS). - files_by_drawer = {} - for f in os.listdir(export_dir): - if not os.path.isfile(os.path.join(export_dir, f)): - continue - token = f.split("_", 1)[0] - files_by_drawer.setdefault(token, []).append(f) + files_by_drawer = index_files_by_drawer(export_dir) print("FCCS export completeness check") print(f" export folder : {export_dir}") @@ -153,8 +90,8 @@ def main(): break ids = [i for i in re.split(r"[\s,]+", raw) if i] for drawer_id in ids: - check_drawer(drawer_id, files_by_drawer.get(drawer_id, []), - manifest_dir, print) + report_drawer(drawer_id, files_by_drawer.get(drawer_id, []), + manifest_dir, print) if __name__ == "__main__": diff --git a/fccs_config.py b/fccs_config.py index f26d3ca..1a29108 100644 --- a/fccs_config.py +++ b/fccs_config.py @@ -157,3 +157,135 @@ def check_for_clashes(drawer_ids): if matches: clashes.append((short, matches)) return clashes + + +# --------------------------------------------------------------------------- +# MANIFEST / EXPORT-COMPLETENESS HELPERS +# --------------------------------------------------------------------------- +# Shared by fccs_verify.py (batch) and fccs_check.py (interactive) so both +# judge completeness identically. + +# Trailing " Page N" (optionally "Page N of M") appended to multi-page exports. +_PAGE_RE = re.compile(r"\s*Page\s+\d+(?:\s+of\s+\d+)?\s*$", re.IGNORECASE) +# Creation-date field ("_MM-DD-YYYY_") that precedes the document name. +_DATE_ANCHOR = re.compile(r"_\d{2}-\d{2}-\d{4}_") + + +def load_manifest(path): + """Load a manifest file and return one row (list of cells) per document. + + Handles both formats: + - New: one document per line (tab-separated columns). + - Old: a raw ListView dump led by 'List1', then row-major cells with + 3 columns per document (Drawer ID, Page Title, Application), reshaped + so the document count is correct. + """ + with open(path, "r", encoding="utf-8") as f: + lines = [line.rstrip("\n") for line in f if line.strip()] + + if lines and lines[0].strip() == "List1": + cells = lines[1:] + rows = [cells[i:i + 3] for i in range(0, len(cells), 3)] + return [r for r in rows if len(r) == 3] + + return [line.split("\t") for line in lines] + + +def manifest_doc_names(path, drawer_id): + """Return the expected document (Page Title) names from a manifest file.""" + rows = load_manifest(path) + names = [] + for row in rows: + if len(row) >= 3 and row[0].strip() == drawer_id: + names.append(row[1]) # old format: DrawerID, PageTitle, Application + elif row: + names.append(row[0]) # new format: PageTitle, Application + return names + + +def match_key(name): + """Normalize a document name for tolerant comparison. + + Strips a trailing 'Page N' page-split suffix, then reduces to lowercase + alphanumerics so punctuation and filename-sanitization differences (FCCS + strips characters illegal in Windows filenames) don't cause false + mismatches. + """ + base = _PAGE_RE.sub("", name) + return re.sub(r"[^a-z0-9]+", "", base.lower()) + + +def exported_doc_name(filename): + """Extract the document-name portion from an exported filename, or None. + + Format: {drawer}_{client}_{folder}_{MM-DD-YYYY}_{docname}.ext + The creation-date field is a reliable anchor; the doc name follows the last + one (client/folder fields don't carry an "_MM-DD-YYYY_" pattern). + """ + stem = os.path.splitext(filename)[0] + anchors = list(_DATE_ANCHOR.finditer(stem)) + if not anchors: + return None + return stem[anchors[-1].end():] + + +def index_files_by_drawer(export_dir): + """Map each export file to its leading drawer-ID token (before first '_'). + + The underscore boundary keeps clashing IDs separate (04289 vs 04289TS). + Returns {drawer_id: [filenames]}. + """ + index = {} + for f in os.listdir(export_dir): + if not os.path.isfile(os.path.join(export_dir, f)): + continue + token = f.split("_", 1)[0] + index.setdefault(token, []).append(f) + return index + + +def evaluate_drawer(drawer_id, files, manifest_dir): + """Compare a drawer's manifest against its exported files. + + `files` is the list of export filenames belonging to this drawer. Accounts + for page-splitting (a document exported as 'Name Page 1/2/...' counts as + present) and filename sanitization (via match_key). + + Returns a dict: + has_manifest, manifest_path, expected (list), missing (list), + extras (list), unparsed (int), file_count (int) + """ + manifest_path = os.path.join(manifest_dir, drawer_id + ".txt") + result = { + "has_manifest": os.path.exists(manifest_path), + "manifest_path": manifest_path, + "expected": [], + "missing": [], + "extras": [], + "unparsed": 0, + "file_count": len(files), + } + if not result["has_manifest"]: + return result + + expected = manifest_doc_names(manifest_path, drawer_id) + result["expected"] = expected + + # Group exported files by normalized doc name; page-splits collapse together. + exported = {} # key -> list of full doc names (one entry per file/page) + unparsed = 0 + for f in files: + doc = exported_doc_name(f) + if doc is None: + unparsed += 1 + continue + exported.setdefault(match_key(doc), []).append(doc) + + result["missing"] = [name for name in expected + if match_key(name) not in exported] + + expected_keys = {match_key(n) for n in expected} + result["extras"] = [names[0] for k, names in exported.items() + if k not in expected_keys] + result["unparsed"] = unparsed + return result diff --git a/fccs_verify.py b/fccs_verify.py index aa633df..5e28f9b 100644 --- a/fccs_verify.py +++ b/fccs_verify.py @@ -1,35 +1,24 @@ """ Step 4 (optional): Verify exported files against manifests. -Compares the per-drawer manifests captured during export (Step 2) -against the actual files in the export directory to identify -missing or extra files. +Batch check across EVERY drawer that has a manifest: compares each drawer's +manifest (captured during Step 2) against the files in the export directory and +reports which drawers are complete vs missing documents. + +Matching is document-level and identical to fccs_check.py (via +fccs_config.evaluate_drawer): it accounts for page-splitting (a document +exported as 'Name Page 1', 'Name Page 2', ... counts as present) and for +filename sanitization (titles with characters illegal in filenames still +match). Use fccs_check.py to spot-check individual drawers interactively. """ import os import sys -from fccs_config import parse_args, load_config, make_logger - - -def load_manifest(path): - """Load a manifest file and return one row per document. - - Handles both formats: - - New: one document per line (tab-separated columns). - - Old: a raw ListView dump led by 'List1', then row-major cells with - 3 columns per document (Drawer ID, Page Title, Application). We - reshape it so the document count is correct. - """ - with open(path, "r", encoding="utf-8") as f: - lines = [line.rstrip("\n") for line in f if line.strip()] - - if lines and lines[0].strip() == "List1": - cells = lines[1:] - rows = [cells[i:i + 3] for i in range(0, len(cells), 3)] - return [r for r in rows if len(r) == 3] - - return [line.split("\t") for line in lines] +from fccs_config import ( + parse_args, load_config, make_logger, + evaluate_drawer, index_files_by_drawer, +) def main(): @@ -49,77 +38,56 @@ def main(): log(f"ERROR: export directory not found: {export_dir}") sys.exit(1) - # Load all manifests - manifests = {} - for fname in sorted(os.listdir(manifest_dir)): - if fname.endswith(".txt"): - drawer_id = os.path.splitext(fname)[0] - items = load_manifest(os.path.join(manifest_dir, fname)) - manifests[drawer_id] = items - - if not manifests: + drawer_ids = sorted( + os.path.splitext(f)[0] + for f in os.listdir(manifest_dir) + if f.endswith(".txt") + ) + if not drawer_ids: log("No manifest files found.") sys.exit(1) - # Index exported files by drawer ID prefix - exported_by_drawer = {} - all_exported = [ - f for f in os.listdir(export_dir) - if os.path.isfile(os.path.join(export_dir, f)) - ] - - for fname in all_exported: - sep = fname.find("_") - if sep != -1: - did = fname[:sep] - exported_by_drawer.setdefault(did, []).append(fname) + files_by_drawer = index_files_by_drawer(export_dir) log("=" * 60) log("Export Verification Report") log("=" * 60) - log(f"Manifests loaded : {len(manifests)} drawers") - log(f"Exported files : {len(all_exported)} total") + log(f"Manifests loaded : {len(drawer_ids)} drawers") log("") - total_expected = 0 - total_actual = 0 - mismatched = [] + complete_ids = [] + incomplete = [] # (drawer_id, missing_list) - for drawer_id in sorted(manifests): - expected = manifests[drawer_id] - actual = exported_by_drawer.get(drawer_id, []) - n_expected = len(expected) - n_actual = len(actual) - total_expected += n_expected - total_actual += n_actual - - if n_expected == n_actual: - log(f" {drawer_id}: OK ({n_actual} files)") + for drawer_id in drawer_ids: + r = evaluate_drawer(drawer_id, files_by_drawer.get(drawer_id, []), + manifest_dir) + n_expected = len(r["expected"]) + missing = r["missing"] + if not missing: + complete_ids.append(drawer_id) + log(f" {drawer_id}: OK ({n_expected} docs, {r['file_count']} files)") else: - diff = n_actual - n_expected - sign = "+" if diff > 0 else "" - log(f" {drawer_id}: MISMATCH — expected {n_expected}, " - f"got {n_actual} ({sign}{diff})") - mismatched.append(drawer_id) + incomplete.append((drawer_id, missing)) + log(f" {drawer_id}: INCOMPLETE — {len(missing)}/{n_expected} " + f"document(s) missing:") + for m in missing: + log(f" - {m}") - # Check for exported files with no manifest - orphan_drawers = set(exported_by_drawer.keys()) - set(manifests.keys()) - orphan_count = sum(len(exported_by_drawer[d]) for d in orphan_drawers) + # Exported files whose drawer has no manifest at all. + orphan_drawers = sorted(set(files_by_drawer) - set(drawer_ids)) + orphan_count = sum(len(files_by_drawer[d]) for d in orphan_drawers) log("") log("-" * 60) - log(f"Expected total : {total_expected}") - log(f"Actual total : {total_actual}") - + log(f"Complete drawers : {len(complete_ids)}") + log(f"Incomplete drawers : {len(incomplete)}") + if incomplete: + log(f" Incomplete IDs: {', '.join(d for d, _ in incomplete)}") if orphan_drawers: - log(f"No manifest for : {', '.join(sorted(orphan_drawers))} " + log(f"No manifest for : {', '.join(orphan_drawers)} " f"({orphan_count} files)") - - if mismatched: - log(f"Mismatched drawers: {', '.join(mismatched)}") - elif not orphan_drawers: - log("All drawers match.") - + if not incomplete and not orphan_drawers: + log("All drawers complete.") log("=" * 60)