125 lines
6.6 KiB
Markdown
125 lines
6.6 KiB
Markdown
# FCCS Data Extraction Tool
|
|
|
|
Extracts and organizes documents from FileCabinet CS (Thomson Reuters) using GUI automation. Designed to work across multiple client engagements without code changes.
|
|
|
|
## Prerequisites
|
|
|
|
- 32-bit Python (must match FCCS architecture)
|
|
- `pywinauto` (`pip install pywinauto`)
|
|
- `Pillow` (`pip install Pillow`) — required for failure screenshots
|
|
- FileCabinet CS installed and restored with client backup data
|
|
|
|
## Project Layout
|
|
|
|
| File | Purpose |
|
|
|------|---------|
|
|
| `config.ini` | All paths, timeouts, and FCCS control identifiers |
|
|
| `fccs_folders.txt` | FCCS folder templates (manually populated per engagement) |
|
|
| `fccs_config.py` | Shared config loading, logging, folder pattern building |
|
|
| `fccs_scan.py` | Step 1: Scan backup directory for drawer IDs |
|
|
| `fccs_export.py` | Step 2: Automate FCCS GUI to export all drawers |
|
|
| `fccs_reorganize.py` | Step 3: Parse filenames and rebuild folder structure |
|
|
| `fccs_verify.py` | Step 4 (optional): Compare manifests against exported files |
|
|
| `fccs_check.py` | Utility: Interactively check if specific drawers exported completely |
|
|
| `fccs_dump_controls.py` | Utility: Dump control identifiers of an on-screen dialog |
|
|
|
|
## Setup Per Engagement
|
|
|
|
1. Edit `config.ini` -- point `backup_dir`, `export_dir`, and `output_dir` at the client's directories.
|
|
2. Populate `fccs_folders.txt` -- open FCCS > System Configuration > Document Folders and list each folder template exactly as shown, one per line. Keep the `YYYY` prefix on recurring folders. UltraTax CS folders are detected automatically and do not need to be listed.
|
|
|
|
## Workflow
|
|
|
|
### Step 1: Scan Drawers
|
|
|
|
```
|
|
python fccs_scan.py
|
|
```
|
|
|
|
Reads the restored FCCS backup directory and writes all drawer IDs (subfolder names) to `drawer_ids.txt`. It also:
|
|
|
|
- **Flags prefix clashes** — if one drawer ID is a prefix of another (e.g. `02218` and `02218A`), searching the **base** ID in FCCS pops up a selection box that breaks plain automated navigation. These clashes are reported, and the base (shorter) IDs are auto-seeded into `ignore.txt`. The longer, more-specific IDs (`02218A`) search fine and export normally; the base IDs are handled by the separate clash-export script.
|
|
- **Reports ignored drawers** — any IDs listed in `ignore.txt` that exist in this backup are shown as ones the export will skip.
|
|
|
|
**Ignoring drawers:** The scan creates `ignore.txt` (at `ignore_file`, default `C:\Migration\ignore.txt`) if it doesn't exist and pre-fills it with the clash base IDs — searching those in FCCS shows a selection box that stalls the plain export, so they're skipped by the main export. Open the file and:
|
|
|
|
- **Delete or comment out** any clash base you'd rather handle fully by hand.
|
|
- **Add** any other drawers to skip (e.g. password-protected folders), one ID per line.
|
|
|
|
Re-running the scan never overwrites your edits — it only appends newly-discovered clashes. `drawer_ids.txt` stays a full inventory; the export skips anything active in the ignore list. Lines starting with `#` are comments.
|
|
|
|
### Step 2: Export Documents
|
|
|
|
```
|
|
python fccs_export.py
|
|
```
|
|
|
|
Requires FCCS to be open with export destination already configured. Automates the GUI to export every drawer via File > Send To > File. Features:
|
|
|
|
- **Resumable** -- tracks completed drawers in `completed.txt`; safe to restart
|
|
- **Screenshots** -- captures failure states for diagnosis
|
|
- **Defensive** -- one bad drawer won't crash the entire run
|
|
- **Crash recovery** -- some documents (e.g. UltraTax "Diagnostics" files) crash FCCS's converter (`FileConversionEngine::convert() failed`), which aborts that drawer's export. The script detects the error dialog, screenshots and logs the crashing document, dismisses it, and records the drawer in `crashed.txt` so it's skipped on future runs instead of stalling. Handle crashed drawers manually (export them excluding the poison document); delete a line from `crashed.txt` to retry after fixing.
|
|
|
|
### Step 3: Reorganize Files
|
|
|
|
```
|
|
python fccs_reorganize.py
|
|
```
|
|
|
|
Parses the flat exported filenames and copies them into an organized structure:
|
|
|
|
```
|
|
output/
|
|
ABRAHAM, REBEKAH L./
|
|
Tax Documents/
|
|
2025/
|
|
030126 E-mail re Tax Info.pdf
|
|
Billing & Invoices/
|
|
2026/
|
|
030826 Invoice for 2025 Forms 1040 & IL-1040.pdf
|
|
UltraTax CS/
|
|
12-31-2008/
|
|
2008 Form 1040 Filing Instructions.doc
|
|
Permanent File/
|
|
Driver's License.pdf
|
|
_unparsed/
|
|
(files that couldn't be parsed go here for manual review)
|
|
```
|
|
|
|
Exported filenames follow the format `{drawer_id}_{client_name}_{folder_name}_{creation_date}_{document_name}.ext`. The parser uses folder templates from `fccs_folders.txt` (with `YYYY` expanded via regex) and the creation date (`MM-DD-YYYY`) as anchors to reliably split the underscore-delimited fields. Folder names are decomposed into nested paths that match the FCCS UI structure (e.g. `2025 Tax Documents` becomes `Tax Documents/2025/`). UltraTax CS folders are matched by a built-in pattern.
|
|
|
|
### Step 4 (optional): Verify Export Completeness
|
|
|
|
During export, each drawer's document list is captured from the FCCS dialog and saved as a manifest (in `manifest_dir`). These tools compare the manifests against the files actually in the export folder to confirm nothing was missed.
|
|
|
|
Both compare at the **document level** and share identical matching logic. They account for:
|
|
|
|
- **Page-splitting** -- a multi-page document exported as `Name Page 1`, `Name Page 2`, … counts as that one document being present.
|
|
- **Filename sanitization** -- document titles containing characters illegal in Windows filenames (e.g. `:` `/` `?`) still match the exported files.
|
|
|
|
Batch-check every drawer that has a manifest:
|
|
|
|
```
|
|
python fccs_verify.py
|
|
```
|
|
|
|
Reports each drawer as `OK` or `INCOMPLETE` (listing the missing documents), plus a summary and any exported drawers that have no manifest.
|
|
|
|
Spot-check specific drawers interactively (e.g. ones the log marked failed, to see whether they actually finished exporting in the background):
|
|
|
|
```
|
|
python fccs_check.py
|
|
Drawer ID(s): 08097 18430
|
|
```
|
|
|
|
> Note: because page-splitting means the number of files can't be mapped one-to-one to documents, completeness is judged by document *presence* (is each manifest document represented by at least one exported file), not by exact file counts.
|
|
|
|
## Config Reference
|
|
|
|
All scripts read from `config.ini` (or specify `--config path\to\config.ini`).
|
|
|
|
- **`[paths]`** -- backup_dir, export_dir, output_dir, drawer_id_file, completed_file, ignore_file, crashed_file, log_file, screenshot_dir, manifest_dir, folder_list
|
|
- **`[timeouts]`** -- nav_timeout, dialog_timeout, progress_appear, progress_finish, settle, confirm_timeout
|
|
- **`[controls]`** -- FCCS window class names and button titles (rarely need changing)
|