Terminal Engineering Data for Agent Training and Evaluation

01Your task is to analyze the directory tree at `/app/legacy_infra/` — a simulated legacy data infrastructure generated by running `/app/setup_infra.sh` — and produce a comprehensive forensic report at `/app/output/forensic_report.json` with dependency mapping, health classification, and a prioritized modernization plan.02 03## Context04 05The input is the directory tree under `/app/legacy_infra/`, which simulates a poorly maintained data infrastructure left behind by a departed engineer. The directory contains cron job definitions, ETL scripts (Python 2, Python 3, Bash, SQL), configuration fragments, and table schema dumps — all with inconsistent naming, no documentation, and tangled dependencies.06 07Run the setup script first to generate `/app/legacy_infra/`:08 09```bash10bash /app/setup_infra.sh11```12 13## Objective14 15Write a Python program at `/app/analyze_infra.py` that:16 171. Scans the entire `/app/legacy_infra/` directory tree.182. Parses all discovered artifacts (scripts, cron definitions, SQL files, config files).193. Produces a single JSON report at `/app/output/forensic_report.json` describing the system's topology, health, and a modernization roadmap.20 21Run it via:22 23```bash24python3 /app/analyze_infra.py25```26 27The program must exit `0` on success and non-zero on any error.28 29## Infrastructure Layout30 31After running the setup script, `/app/legacy_infra/` will contain:32 33| Path pattern | Contents |34|---|---|35| `servers/{alpha,beta,gamma}/crontab` | Crontab files with scheduled jobs |36| `scripts/*.py` | ETL scripts (mix of Python 2 and Python 3 syntax) |37| `scripts/*.sh` | Bash pipeline scripts |38| `sql/*.sql` | Stored procedures and raw queries |39| `configs/*.conf` | Flat key=value config files referencing DB hosts, API endpoints, credentials placeholders |40| `schemas/tables.json` | JSON array of table definitions with names and column lists |41 42Each crontab entry references a script by relative or absolute path. Scripts may reference tables (via string literals matching table names), other scripts (via `subprocess` calls or `source` commands), API endpoints (via URLs), and config files.43 44## Output Schema45 46`/app/output/forensic_report.json` must be a JSON object with exactly these top-level keys:47 48```json49{50  "artifacts": [ ... ],51  "dependency_graph": { ... },52  "table_lineage": { ... },53  "health_classification": { ... },54  "anomalies": [ ... ],55  "modernization_plan": [ ... ]56}57```58 59### `artifacts` (array of objects)60 61Every discovered file. Each object:62 63| Field | Type | Description |64|---|---|---|65| `path` | string | Absolute path to the file |66| `type` | string | One of: `"python2"`, `"python3"`, `"bash"`, `"sql"`, `"config"`, `"crontab"`, `"schema"`, `"unknown"` |67| `size_bytes` | integer | File size in bytes |68| `line_count` | integer | Number of lines |69| `server` | string or null | Server name (`"alpha"`, `"beta"`, `"gamma"`) if under a server directory, else `null` |70 71Python version detection rules:72- Classify as `"python2"` if the file contains `print ` statements without parentheses (e.g., `print "hello"`) OR contains `#!/usr/bin/env python2` or `#!/usr/bin/python2` in the shebang.73- Classify as `"python3"` otherwise for `.py` files.74 75### `dependency_graph` (object)76 77Keys are absolute file paths. Values are arrays of absolute file paths that the key file depends on (references, calls, or sources). If a crontab entry references `scripts/etl_load.py`, the crontab file depends on that script. If a script references a config file or another script, add that edge. Only include files that actually exist in the tree.78 79### `table_lineage` (object)80 81Keys are table names (from `schemas/tables.json`). Values are objects:82 83| Field | Type | Description |84|---|---|---|85| `referenced_by` | array of strings | Absolute paths of scripts/SQL files that mention this table name |86| `operation_types` | array of strings | Subset of `["SELECT", "INSERT", "UPDATE", "DELETE", "CREATE", "DROP"]` detected in referencing files (case-insensitive keyword search preceding or near the table name) |87 88### `health_classification` (object)89 90Keys are absolute file paths of scripts and SQL files only. Values are one of:91 92| Classification | Criteria |93|---|---|94| `"critical"` | Referenced by at least one active cron job AND references at least one table |95| `"orphaned"` | Not referenced by any cron job |96| `"deprecated"` | References an API endpoint URL that contains `deprecated`, `v0`, `v1` in the path, OR is a Python 2 script |97| `"unknown"` | Does not match any above category |98 99Apply classifications in priority order: `deprecated` > `critical` > `orphaned` > `unknown`. A script matching multiple categories gets the highest-priority one.100 101### `anomalies` (array of objects)102 103Detected issues. Each object:104 105| Field | Type | Description |106|---|---|---|107| `type` | string | One of: `"dead_reference"`, `"deprecated_api"`, `"python2_script"`, `"long_query"`, `"no_error_handling"`, `"naming_inconsistency"` |108| `file` | string | Absolute path of the file with the issue |109| `detail` | string | Human-readable description (English, 1-3 sentences) |110 111Detection rules:112- `"dead_reference"`: A crontab or script references a file path that does not exist on disk.113- `"deprecated_api"`: A script contains a URL with `deprecated`, `v0`, or `v1` in the path.114- `"python2_script"`: File classified as `python2`.115- `"long_query"`: A SQL file or inline SQL string exceeds 150 lines or 5000 characters.116- `"no_error_handling"`: A Python script contains no `try`/`except` blocks and no `if __name__` guard.117- `"naming_inconsistency"`: A table name matches patterns like `temp_*`, `*_v[0-9]*`, `*_final_*`, `new_*`, `*_latest*`, `*_backup*`, `*_old*`.118 119### `modernization_plan` (array of objects)120 121Ordered by `priority` ascending (1 = highest). Each object:122 123| Field | Type | Description |124|---|---|---|125| `priority` | integer | 1-based rank |126| `action` | string | One of: `"migrate_python2_to_3"`, `"remove_dead_jobs"`, `"add_error_handling"`, `"consolidate_cron_to_orchestrator"`, `"rename_tables"`, `"replace_deprecated_apis"`, `"add_data_quality_checks"`, `"document_pipeline"` |127| `affected_files` | array of strings | Absolute paths of files this action applies to |128| `effort_estimate` | string | One of: `"low"`, `"medium"`, `"high"` |129| `rationale` | string | 1-2 sentence justification |130 131Priority ordering rules:1321. `"remove_dead_jobs"` — effort: `"low"`1332. `"replace_deprecated_apis"` — effort: `"medium"`1343. `"migrate_python2_to_3"` — effort: `"medium"`1354. `"add_error_handling"` — effort: `"low"`1365. `"consolidate_cron_to_orchestrator"` — effort: `"high"`1376. `"rename_tables"` — effort: `"medium"`1387. `"add_data_quality_checks"` — effort: `"medium"`1398. `"document_pipeline"` — effort: `"low"`140 141Only include an action if at least one file qualifies for it. `affected_files` must not be empty.142 143## Technical Constraints144 145- Use Python 3.11 standard library only. No third-party packages.146- The output directory `/app/output/` must be created by your program if it does not exist.147- JSON output must be UTF-8 encoded, pretty-printed with 2-space indentation, and have a trailing newline.148- All file paths in the output must be absolute paths.149- The `dependency_graph` must not contain self-references.150- The `anomalies` array must contain no duplicate entries (same `type` + `file` combination).151- The `modernization_plan` must be sorted by `priority` ascending with no gaps in numbering.