Drawing Viewer — Memory & OOM Resilience Roadmap

Status: Track A + Track B + Track C all shipped in fix/parser-oom-resilience. 5 of 5 customer drawings render. Drawing 3 ALD verified — peak 1.5 GB, 47 s build (was SIGKILL at 13 GB cgroup limit pre-Track-C). Created: 2026-05-04 (PIN-230 testing with Conor's files surfaced the issue) Reviewed: 2026-05-04 + 2026-05-05 by Codex (gpt-5.5 xhigh) Last updated: 2026-05-05 (Track C shipped) Linear parent: PIN-146

Problem

The DXF parser pipeline (apps/api/parser/) holds the entire ezdxf entity tree in memory plus per-layout scene-builder transients. For dense modelspace or paperspace projections the working set exceeds the container memory budget, the parse worker is SIGKILL'd by the kernel, and the drawing fails to render.

Observed during PIN-230 testing on Conor's files:

Drawing	Source	DXF size	Failure point	Container at OOM
3	OpenAI 25.04.14 SM 2nd Floor	267 MB DXF	Layout 4 of 11 (ALD, paperspace)	13 GiB
4	390 Park Avenue 5th Floor SM	97 MB DXF	Layout 1 (Model, modelspace)	13 GiB

DXF size is not a reliable predictor of memory peak — Drawing 4 is 2.7× smaller than Drawing 3 yet OOMed earlier because its modelspace is more entity-dense.

Root cause

ezdxf.readfile() loads the full document into a Python entity tree (≈3-5× the file size in resident memory due to Python object overhead). On top of that, scene_builder.build_scene accumulates emitted entities in a protobuf scene.entities list before serializing. For paperspace layouts an additional pre-emission stage (expand_paperspace_to_entities → _flatten_instances) materializes a flat instances = list(...) of every projected entity — which on dense viewports peaks at several million references.

Peak memory per layout = ezdxf_doc + paperspace_instances_list + scene_proto_transient

Each term has a separate fix:

• ezdxf_doc — addressed by byte-level sharding (Track A) so each subprocess parses ⅓-¼ of the modelspace
• scene_proto_transient — addressed by streaming proto write (Track B): write entities to gzip stream as they emit, never accumulate the full proto
• paperspace_instances_list — addressed by Track C (lazy-generator 2-pass) so the chain is never fully materialised; bounds computed via running min/max on a first generator pass, emit via a second generator pass.

What shipped in fix/parser-oom-resilience

Track A — Sharded Model fallback ✅ (closes Drawing 4)

Pre-process the DXF into N byte-level shards using iterdxf.structure.index byte offsets (NOT iterdxf.export() which silently drops ACAD_PROXY_ENTITY and 3DSOLID). Each shard contains a full HEADER/CLASSES/TABLES/BLOCKS/OBJECTS plus its slice of ENTITIES. Parse each shard in its own subprocess, merge per-shard scenes via merge_scenes.merge_layout_scenes (layer dedup, layer_idx remap, instance_id renumber).

Triggered automatically when the orchestrator detects a Model-layout OOM (subprocess exit code -9 / SIGKILL).

Files: apps/api/parser/shard_dxf.py (new, 175 lines), apps/api/parser/merge_scenes.py (new, ~190 lines).

Drawing 4 result: 4 shards × 66-80 MB, exact entity count match (7751 = original modelspace), 93k-entity merged scene, full visual fidelity preserved.

Track B — Streaming proto write ✅ (infrastructure, partial coverage)

scene_builder.build_scene_streaming() writes entities directly to gzip output as one-entity DrawingScene deltas instead of accumulating scene.entities in memory. Reader merges deltas + metadata footer back into one scene on parse. Three emit call-sites (_emit_polyface_faces, _emit_hatch_loops, main loop) parameterized via entity_consumer callback.

Regression: byte-identical bounds vs in-memory baseline on Battery Park Model (21072 entities).

build_layout.py accepts streaming: bool flag; in streaming mode skips compute_layout_metrics and scale_detect (both need post-build scene.entities scan).

Pre-emptive rather than reactive: orchestrator detects large input files (shard_dxf.decide_shard_size().enabled, > 80 MB threshold) and runs all paperspace layouts in streaming mode up-front. Reactive retry-on-OOM does not work because the Linux cgroup OOM-killer kills the entire process tree (orchestrator + subprocess) at once — Python try/except on the orchestrator side never gets to fire.

Drawing 3 (post-Track-C): all 11 layouts complete via streaming. ALD specifically: 1.5 GB peak / 47 s. Pre-Track-C the same layout SIGKILL'd at the cgroup ceiling (~13 GB).

Subprocess-per-layout

Each layout parses in an isolated Python subprocess. After exit, the OS reclaims ezdxf state. Bounds the failure mode: one bad layout doesn't take down its neighbours.

Dead-signal classification

signal: killed and MemoryError added to dead-signals list in drawing_parse_worker.go. OOMs no longer trigger the 5-retry exponential-backoff loop — go straight to FAILED.

XREF "spider-web" fix (side quest)

Architectural background XREFs flooded canvas with crosshatch noise. AutoCAD's plotter screens these via plot style table (.ctb) at ~10% opacity; we don't have CTBs. Fix:

1. Parser (scene_builder.py): skip pattern_segments tessellation for HATCH on XREF-dependent layers (regex \$\d+\$ against layer name).
2. Renderer (sceneRenderer.js): set transparent: true, opacity: 0.2 on materials for XREF layers.

Drawing 5 verification: XREF hatch pattern_segments 1,113,892 floats → 0. Non-XREF hatches preserved (insulation/section-cut). Scene blob -10%.

Track C — Lazy-generator refactor ✅ (closes Drawing 3 ALD)

scene_builder.build_scene now branches on streaming_mode = entity_consumer is not None and layout_name != "Model":

• Streaming mode (paperspace + custom consumer, used by build_scene_streaming for big input files): a single _make_instance_chain() factory rebuilds a fresh expand_paperspace_to_entities → _flatten_instances → page-clip generator chain. Pass 1 computes bounds via running min/max over _entity_output_coords — no list, no _trim_bounds sorted arrays. Pass 2 walks the same factory and emits via entity_consumer. Skips _trim_bounds entirely (full bounds is the documented trade-off; see "Streaming-mode bounds caveat" below).
• Legacy mode (Model layout, in-memory build_scene, small drawings): unchanged. instances = list(...) materialised, _trim_bounds adaptive trim preserved. No regression.

Plus a defensive cap on per-parent virtual_entities():

viewport_expansion.virtual_children_capped(ent, cap=20_000) (env override PARSER_VIRTUAL_CHILDREN_CAP) iterates ent.virtual_entities() with an early break and exception swallow. Replaces the unbounded list(ent.virtual_entities()) calls in two places (viewport_expansion.py viewport loop + scene_builder._flatten_instances). One runaway ACAD_PROXY_ENTITY can no longer mid-iteration OOM the layout — children get truncated and a single warning lands in the operator log.

Drawing 3 verification (2026-05-05):

Metric	Pre-Track-C	Post-Track-C
ALD layout result	SIGKILL at ~13 GB cgroup ceiling	313 entities, 11 KB blob
Peak resident memory	(kill before measurable)	1.5 GB
ALD build time	(never completes)	47 s
Drawing 3 layout coverage	3 of 11 (8.5x11 + 8.5x11(contractor) + Model in 5-retry-burnt-attempt)	11 of 11

Functional regression check (Drawings 1 + 4): in-memory and streaming entity counts byte-identical across all 12 layouts (Model + 5 paperspace each). _trim_bounds preserved for legacy paths.

Streaming-mode bounds caveat: legacy _trim_bounds removes outlier entity-bbox centers (≤10% percentile cuts) so a few stray glyphs ~1000× away from main content don't widen the camera fit-to-extents window. Streaming mode skips this — full bounds includes outliers. This is acceptable because streaming mode is only triggered on input files > 80 MB (per shard_dxf.decide_shard_size), where outlier risk is much smaller and an OOM-killed parse is the much bigger concern. If a future big drawing demonstrates pathological outliers, switch streaming-mode bounds to a sampling-based trim approach (track 1% of leaf bboxes in a reservoir).

Files touched: scene_builder.py (build_scene refactor + import), viewport_expansion.py (virtual_children_capped helper).

Coverage after fix/parser-oom-resilience

Drawing	Status	Mechanism
1 — Battery Park	✅ READY	Single-doc, regression preserved
2 — Sheetmetal Shop Rev 9	✅ READY	Existing pre-branch
3 — OpenAI 267 MB	✅ READY	Track A (Model) + Track C (paperspace 2-pass)
4 — 390 Park 97 MB	✅ READY	Track A (sharded Model fallback)
5 — Netflix	✅ READY	Existing + XREF noise fix

Other solutions (status update)

Solution 5 — Physical DXF sharding ✅ implemented as Track A

Originally believed blocked because iterdxf.SUPPORTED_TYPES excludes ACAD_PROXY_ENTITY (Pinley content has 70%+ of HVAC geometry wrapped as proxies). Workaround found: iterdxf.structure.index exposes byte offsets for every entity type including proxies. Skip IterDXFWriter.write(), use raw byte copy instead. Confirmed correct on Drawing 4: 7751 modelspace entities exact-match across 4 shards.

Limitation kept from original analysis: only Model layout is sharded. Paperspace layouts have block-record-resident entities (title blocks, viewport borders) that get duplicated across shards — merging would triple-count them. Their entities live in BLOCKS section which is byte-copied into every shard.

Solution 1 — Streaming DXF parser

Re-evaluated: not needed wholesale. The narrow case (Drawing 3 ALD) is closed by Track C above. Full streaming parser remains a 4-6 week project that touches every entity type and would re-do the proxy fidelity work. Defer indefinitely.

Solution 2 — Replace ezdxf with native parser (last resort)

Unchanged. ODA SDK is the only realistic native option (LibreDWG is GPLv3+, libdxfrw / custom Rust both rebuild proxy decoding from scratch). 4-8 weeks plus licence cost. Only revisit if Track C fails to land or new content categories surface.

Solution 3 — Hybrid with Autodesk Platform Services

Unchanged. Cost re-estimate via Codex: $15-60/month at our 50-200 file/month volume (~$0.30/file), not the original $0.50-2/file. Rejected per product preference (avoid recurring third-party costs and customer-data-residency concerns).

Comparison

Solution / Track	Effort	Drawing 3 ALD	Drawing 4	Future large files	Cost	Status
Subprocess-per-layout	shipped	partial (3/11 layouts)	no	depends on file	$0	✅ shipped
Dead-signal OOM classifier	shipped	n/a	n/a	bounds retry loop	$0	✅ shipped
Track A — Sharded Model	shipped	n/a (paperspace)	✅ full	✅ Model OOMs	$0	✅ shipped
Track B — Streaming proto	shipped	partial (3/11 layouts)	n/a	✅ paperspace output	$0	✅ shipped
Track C — Lazy-generator	3-5 days	✅ full	n/a	✅ paperspace input	$0	deferred
~~Solution 5 — DXF sharding~~	n/a	n/a	n/a	n/a	n/a	superseded by Track A
Solution 1 — Streaming parser	4-6 weeks	✅ full	✅ full	✅ full	$0	superseded by Track C
Solution 2 — ODA SDK	4-8 weeks + $$	✅ full	✅ full	✅ full	licence $	last resort
Solution 3 — APS hybrid	1-2 weeks	✅ full	✅ full	✅ full	$15-60/mo	rejected

Recommended next steps

Immediate (this PR)

1. Ship fix/parser-oom-resilience as-is — covers 4/5 customer drawings, robust failure mode for the 5th. Track A + Track B + side improvements are all production-ready.
2. Open Linear follow-up for Track C (lazy-generator refactor). Target: next sprint after this branch merges.

Production hardening (independent of parser fix)

1. Pod resource limits in apps/api/k8/base/deployment.yaml. Currently no resources.limits block — production pods can consume the entire node's memory before kernel intervenes, taking neighbouring pods down. Set memory limit to 16 GiB until Track C lands, then revisit.
2. OOM telemetry: surface parser.oom_killed_total counter so we see failure rate in metrics, not user reports.
3. READY_PARTIAL state (proposed): drawings with ≥1 successful layout become READY-with-warnings instead of FAILED. Critical edge: do not mark Model-OOM drawings as plain READY — that lies to the user. Drawing 3 currently has 0 layouts persisted to disk (orchestrator dies mid-flight on ALD before manifest write), so READY_PARTIAL would also need either (a) Track C to land first, or (b) orchestrator changes to checkpoint successful layouts before the next subprocess starts.

Track C (~3-5 days)

1. Refactor scene_builder.build_scene to drop both instances = list(...) materializations.
2. Redesign compute_layout_metrics / scale_detect / equipment-extractor proximity matching for streaming inputs.
3. Verify Drawing 3 ALD parses; verify Battery Park regression preserved.

After Q3 launch — measure first

1. Track parser.oom_killed_total and parser.shard_required_total for a month.
2. If failure rate > 5%: investigate Solution 1 (full streaming parser) or Solution 3 (APS).
3. If failure rate < 1%: stop. Don't over-engineer.

Production-side concerns

apps/api/k8/base/deployment.yaml has no resources block:

• Pod can consume entire node memory before kernel intervenes — likely takes neighbouring pods down
• No "pod hit memory limit" observability — failures look like silent restart loops
• Behavior depends on k8s node size and free RAM, opaque to application engineers

DevOps action needed independently of this roadmap:

resources:
  requests:
    memory: "4Gi"
    cpu: "500m"
  limits:
    memory: "16Gi"   # bounds the worst case at the cgroup level
    cpu: "4"

Once Track C lands the worst case drops further; until then the limit should be at least 16 GiB.

References

• apps/api/parser/drawing_worker.py — orchestrator (subprocess-per-layout, sharded fallback, pre-emptive streaming)
• apps/api/parser/build_layout.py — per-layout subprocess (in-memory + streaming modes)
• apps/api/parser/scene_builder.py — main scene-emission loop (Track C target)
• apps/api/parser/viewport_expansion.py — paperspace expansion (Track C target)
• apps/api/parser/shard_dxf.py — Track A byte-level sharding via iterdxf.structure.index
• apps/api/parser/merge_scenes.py — Track A scene merger
• apps/api/service/drawing_parse_worker.go — Go orchestrator, retry classifier with dead-signal list
• docs/superpowers/specs/2026-05-04-parser-oom-resilience-spike.md — implementation journal (chronological)
• docs/superpowers/research/3DSOLID-IMPLEMENTATION-GUIDE.md — original Forge / Layer 3 design
• PIN-230 — DWG viewer iPad optimization (where this issue surfaced)