# FCCS Data Extraction Tool

Extracts and organizes documents from FileCabinet CS (Thomson Reuters) using GUI automation. Designed to work across multiple client engagements without code changes.

## Prerequisites

- 32-bit Python (must match FCCS architecture)
- `pywinauto` (`pip install pywinauto`)
- `Pillow` (`pip install Pillow`) — required for failure screenshots
- FileCabinet CS installed and restored with client backup data

## Project Layout

| File | Purpose |
|------|---------|
| `config.ini` | All paths, timeouts, and FCCS control identifiers |
| `fccs_folders.txt` | FCCS folder templates (manually populated per engagement) |
| `fccs_config.py` | Shared config loading, logging, folder pattern building |
| `fccs_scan.py` | Step 1: Scan backup directory for drawer IDs |
| `fccs_export.py` | Step 2: Automate FCCS GUI to export all drawers |
| `fccs_reorganize.py` | Step 3: Parse filenames and rebuild folder structure |
| `fccs_verify.py` | Step 4 (optional): Compare manifests against exported files |
| `fccs_check.py` | Utility: Interactively check if specific drawers exported completely |
| `fccs_dump_controls.py` | Utility: Dump control identifiers of an on-screen dialog |

## Setup Per Engagement

1. Edit `config.ini` -- point `backup_dir`, `export_dir`, and `output_dir` at the client's directories.
2. Populate `fccs_folders.txt` -- open FCCS > System Configuration > Document Folders and list each folder template exactly as shown, one per line. Keep the `YYYY` prefix on recurring folders. UltraTax CS folders are detected automatically and do not need to be listed.

## Workflow

### Step 1: Scan Drawers

```
python fccs_scan.py
```

Reads the restored FCCS backup directory and writes all drawer IDs (subfolder names) to `drawer_ids.txt`. It also:

- **Flags prefix clashes** — if one drawer ID is a prefix of another (e.g. `02218` and `02218A`), searching the **base** ID in FCCS pops up a selection box that breaks plain automated navigation. These clashes are reported, and the base (shorter) IDs are auto-seeded into `ignore.txt`. The longer, more-specific IDs (`02218A`) search fine and export normally; the base IDs are handled by the separate clash-export script.
- **Reports ignored drawers** — any IDs listed in `ignore.txt` that exist in this backup are shown as ones the export will skip.

**Ignoring drawers:** The scan creates `ignore.txt` (at `ignore_file`, default `C:\Migration\ignore.txt`) if it doesn't exist and pre-fills it with the clash base IDs — searching those in FCCS shows a selection box that stalls the plain export, so they're skipped by the main export. Open the file and:

- **Delete or comment out** any clash base you'd rather handle fully by hand.
- **Add** any other drawers to skip (e.g. password-protected folders), one ID per line.

Re-running the scan never overwrites your edits — it only appends newly-discovered clashes. `drawer_ids.txt` stays a full inventory; the export skips anything active in the ignore list. Lines starting with `#` are comments.

### Step 2: Export Documents

```
python fccs_export.py
```

Requires FCCS to be open with export destination already configured. Automates the GUI to export every drawer via File > Send To > File. Features:

- **Resumable** -- tracks completed drawers in `completed.txt`; safe to restart
- **Screenshots** -- captures failure states for diagnosis
- **Defensive** -- one bad drawer won't crash the entire run
- **Crash recovery** -- some documents (e.g. UltraTax "Diagnostics" files) crash FCCS's converter (`FileConversionEngine::convert() failed`), which aborts that drawer's export. The script detects the error dialog, screenshots and logs the crashing document, dismisses it, and records the drawer in `crashed.txt` so it's skipped on future runs instead of stalling. Handle crashed drawers manually (export them excluding the poison document); delete a line from `crashed.txt` to retry after fixing.

### Step 3: Reorganize Files

```
python fccs_reorganize.py
```

Parses the flat exported filenames and copies them into an organized structure:

```
output/
  ABRAHAM, REBEKAH L./
    Tax Documents/
      2025/
        030126 E-mail re Tax Info.pdf
    Billing & Invoices/
      2026/
        030826 Invoice for 2025 Forms 1040 & IL-1040.pdf
    UltraTax CS/
      12-31-2008/
        2008 Form 1040 Filing Instructions.doc
    Permanent File/
      Driver's License.pdf
  _unparsed/
    (files that couldn't be parsed go here for manual review)
```

Exported filenames follow the format `{drawer_id}_{client_name}_{folder_name}_{creation_date}_{document_name}.ext`. The parser uses folder templates from `fccs_folders.txt` (with `YYYY` expanded via regex) and the creation date (`MM-DD-YYYY`) as anchors to reliably split the underscore-delimited fields. Folder names are decomposed into nested paths that match the FCCS UI structure (e.g. `2025 Tax Documents` becomes `Tax Documents/2025/`). UltraTax CS folders are matched by a built-in pattern.

### Step 4 (optional): Verify Export Completeness

During export, each drawer's document list is captured from the FCCS dialog and saved as a manifest (in `manifest_dir`). These tools compare the manifests against the files actually in the export folder to confirm nothing was missed.

Both compare at the **document level** and share identical matching logic. They account for:

- **Page-splitting** -- a multi-page document exported as `Name Page 1`, `Name Page 2`, … counts as that one document being present.
- **Filename sanitization** -- document titles containing characters illegal in Windows filenames (e.g. `:` `/` `?`) still match the exported files.

Batch-check every drawer that has a manifest:

```
python fccs_verify.py
```

Reports each drawer as `OK` or `INCOMPLETE` (listing the missing documents), plus a summary and any exported drawers that have no manifest.

Spot-check specific drawers interactively (e.g. ones the log marked failed, to see whether they actually finished exporting in the background):

```
python fccs_check.py
Drawer ID(s): 08097 18430
```

> Note: because page-splitting means the number of files can't be mapped one-to-one to documents, completeness is judged by document *presence* (is each manifest document represented by at least one exported file), not by exact file counts.

## Config Reference

All scripts read from `config.ini` (or specify `--config path\to\config.ini`).

- **`[paths]`** -- backup_dir, export_dir, output_dir, drawer_id_file, completed_file, ignore_file, crashed_file, log_file, screenshot_dir, manifest_dir, folder_list
- **`[timeouts]`** -- nav_timeout, dialog_timeout, progress_appear, progress_finish, settle, confirm_timeout
- **`[controls]`** -- FCCS window class names and button titles (rarely need changing)