View-Dependent 2D Sprite Projection for 3D Articulated Motion

A living technical study of how mathematically three-dimensional orientation can be represented at runtime using stored two-dimensional sprite views, geometric transformation, interpolation, and later depth-aware articulated composition.

Current Status

The basic research pipeline is working end to end:

Phase 5 through Phase 8 remain preserved as rigid, articulation, and first-body checkpoints. Phase 9 freezes the final reference-candidate architecture: a sixteen-part body reconstructed from ten reusable bank types, a richer FBX retarget, per-bank yaw density, E9 evaluation, and final art-workload export. The neck remains a logical joint but is visually merged into the head bank.

Research Problem

Given a known 3D object pose and camera pose, determine how accurately, smoothly, and efficiently the projected appearance of a rigid or articulated object can be approximated using a finite set of 2D sprite views plus screen-space transformations and depth-aware composition.

This is not a computer-vision problem. The pose is already known mathematically. The uncertainty lies in how much visual information must be stored and how well intermediate orientations can be reconstructed.

Coordinate Model

• +X points right.
• +Y points up.
• +Z points toward the default camera.
• The camera looks along -Z.
• The system is right-handed.
• Human-facing rotation order is yaw, then pitch, then roll.

Orthographic projection intentionally removes Z from screen X and Y. Depth is still retained internally for ordering and later articulation work.

Subjects

C0

C0 is the permanent unit-cube calibration subject. It is used to expose sign errors, anchor wrap problems, and projection mistakes before more ambiguous geometry is introduced.

A1-r1

A1 is deliberately asymmetric so directional errors cannot hide behind cube symmetry.

CORE center=( 0.00, 0.00, 0.00) half=(0.82,0.54,0.34)
TOP  center=( 0.36, 0.73, 0.08) half=(0.32,0.24,0.18)
ARM  center=(-0.93,-0.18,-0.06) half=(0.27,0.21,0.20)

Phase 5 stress set

Phase 6 two-link rig

The first articulated subject is intentionally non-anatomical. An upper rectangular link connects to a lower rectangular link at one hinge. The hinge axis is diagonal in local XZ, so bending changes both apparent 2D direction and depth. This makes it useful for testing multi-sprite orientation, pivot registration, and front/back ordering without anatomy introducing additional variables.

Phase 8 simplified body

The first full-body subject contains ten rendered parts: torso, head, paired upper arms and forearms, and paired thighs and shins. Left and right symmetric limbs share the same directional bank, so ten visible parts are reconstructed from six reusable bank types.

Phase 9 final reference rig

The final body contains sixteen rendered parts: head with neck merged visually, upper body, lower body, hips, paired upper and lower arms, paired hands, paired upper and lower legs, and paired feet. Symmetric left/right parts share assets, producing ten unique bank types.

Animation Input

Phase 8 separates motion input from sprite reconstruction. FBX is treated as an authoring and validation source rather than as part of the renderer.

FBX animation
    ↓
Assimp 6.0.5 import
    ↓
humanoid joint mapping
    ↓
30 FPS joint sampling
    ↓
source-basis normalization
    ↓
fixed-proportion NM-PM body retarget
    ↓
per-part directional sprites

The importer reads the first animation clip and requires recognizable equivalents of pelvis, chest, head, upper arms, forearms, hands, thighs, lower legs, and feet. Common Mixamo-style names and several generic left/right aliases are supported. Missing required joints cause an explicit import failure rather than silent substitution.

The executable can auto-load assets/body_rig/input.fbx, accepts drag-and-drop FBX files, and always includes a deterministic built-in fallback clip so the research harness remains runnable without external assets.

Orientation Bank

The current bank contains eight yaw columns and five pitch rows:

Yaw:   0, 45, 90, 135, 180, 225, 270, 315 degrees
Pitch: -60, -30, 0, +30, +60 degrees
Roll:  not captured

This produces 40 stored 128 x 128 captures for each rigid or unique part-bank subject. Captures are exported as PNG files, then loaded back from disk before being used by the approximation view. In Phase 8, six unique body-part banks produce 240 stored captures while symmetric left/right limbs reuse the same bank.

Phase 8 sheet export

The Phase 8 body lab preserves its six-bank 240-sprite export.

Phase 9 mixed-density bank policy

The final rig uses ten unique bank types. Eight large-part banks retain eight yaw columns at 45-degree spacing and five pitch rows, producing 40 sprites each. Hand and foot banks use only four yaw columns at 90-degree spacing with the same five pitch rows, producing 20 sprites each. The shared bank layer stores the yaw count per bank, so selection, blending, residual handling, capture generation, loading, and export all use the correct spacing.

8 full banks × 40 sprites = 320
2 sparse banks × 20 sprites =  40
Final maximum              = 360 unique sprites

The final exporter writes eight 1024 x 640 full sheets, two 512 x 640 sparse sheets, a labeled contact sheet, and TXT/JSON manifests under exports/final_rig_sheets/.

Reconstruction Methods

Roll Policy

Roll is currently applied as ordinary 2D screen rotation after yaw and pitch reconstruction. It does not have its own sprite-bank dimension. Phase 4 includes a dedicated roll check so this assumption can be rejected later if the evidence does not support it.

Measurements

The quantitative evaluator currently records:

• silhouette intersection-over-union;
• symmetric boundary error in pixels;
• centroid displacement in pixels;
• requested orientation, selected views, and residual angles.

E5 extends the same measurements across A1-A5. E6 evaluates the two-link rig, E7 the three-link chain, and E8 the first simplified body. E9 evaluates the final sixteen-part rig at 18 temporal samples across four yaw views and three pitch views, producing 216 final-body poses before reconstruction-method multiplication. E8 and E9 can operate on either their deterministic fallback clip or an imported FBX clip. No universal pass/fail threshold is imposed before measurement. Numerical findings are not written into this notebook until the corresponding native export has actually been executed and reviewed.

Experiment Registry

Durable experiment identifiers remain stable even if implementation phases are later renamed or merged. Full protocols are preserved under experiments/.

Visual Evidence

SP010-FIG-001 - Final Phase 9 visual evidence

User-provided native evidence from the final Phase 9 rig. The montage contains frontal, oblique, and profile comparisons between the retargeted 3D reference and the directional-sprite reconstruction, a representative head-and-neck directional bank, and the final application interface. The captured interface reports 41/41 tests passing, 16 rendered parts, 10 bank types, and a 360-sprite maximum. E9 is shown as pending, so these images are qualitative evidence rather than quantitative E9 results.

When visual behavior cannot be understood reliably from text, screenshots or recordings should be collected instead of guessed from descriptions.

Failed Approaches

Custom large-view textured quad

The anchor thumbnails rendered correctly while the main approximation viewport stayed blank. The custom low-level path was removed and replaced with ordinary raylib texture drawing plus strip-based deformation.

Always-visible reference ghost

A smoothly rotating diagnostic reference underneath the sprite made discrete view switching look artificially smooth. It is now optional and disabled by default.

Oversized readability pass

Increasing the whole window size improved apparent readability but hid normal window controls on the target desktop. Quality improvements now come from source resolution, high-DPI rendering, and font rasterization rather than a larger default window.

Direct mathematical roll passed to raylib

Phase 6 originally passed mathematical roll directly to raylib texture rotation. Because the projection flips world Y into screen Y, the visible rotation sign is reversed. The true geometry and sprite parts therefore rotated in opposite screen directions under roll and the hinge visibly separated. The shared reconstruction layer now converts mathematical roll to the opposite-sign raylib rotation, and the convention has a regression test.

Promotion Rule

A technique cannot enter the eventual reusable reference implementation in the same experiment that invents it. Promotion requires:

1. a procedural or mathematical definition;
2. evidence that it beats or usefully complements a frozen baseline;
3. testing on a second object not used to design it;
4. characterized failure conditions;
5. reproducible parameters;
6. no frame-specific manual tuning;
7. a documented fallback.

Current Implementation

1. Read camera-relative yaw, pitch, and roll.
2. Locate surrounding yaw columns.
3. Locate nearest or surrounding pitch rows.
4. Select a reconstruction method.
5. Read only frozen disk-backed sprite captures.
6. Reconstruct yaw and pitch appearance.
7. Convert mathematical roll to the opposite-sign raylib screen rotation, then apply it as 2D texture rotation.
8. Draw at the projected screen position and depth.
9. When requested, compare against the reference projection.
10. For E5, repeat the frozen pipeline across A1-A5.
11. For SP-007, compute a two-link 3D pose with matrix-composed hinge motion.
12. Reconstruct each link independently from its own pivot-registered sprite bank.
13. Sort complete part sprites by 3D center depth.
14. For E6, compare the composed two-link silhouette against true geometry.
15. For SP-008, compute a three-link chain with two different diagonal hinge axes.
16. Reconstruct upper, middle, and lower links from three independent pivot-registered banks.
17. Sort all three sprites by 3D center depth and compose them.
18. For E7, evaluate 216 three-link poses across all six reconstruction methods.
19. For SP-009, sample either the built-in clip or the first FBX animation into a common body-pose representation.
20. Normalize the source coordinate basis and retarget joint directions onto fixed NM-PM body proportions.
21. Build ten segment poses and reconstruct them from six reusable directional banks.
22. Sort complete body-part sprites by center depth and compose the full body.
23. For E8, evaluate 216 temporal/view body poses across all six reconstruction methods.
24. When requested, export the six Phase 8 body banks.
25. For SP-010, build a sixteen-part final rig from ten symmetric reusable bank types.
26. Keep head and neck visually merged while retaining neck as a logical retarget joint.
27. Use eight yaw anchors for major parts and four yaw anchors for hands and feet.
28. Retarget FBX spine, neck, hand-tip, and toe information when present and synthesize optional helpers when absent.
29. For E9, evaluate 216 temporal/view poses across all six reconstruction methods.
30. Export the final 360-sprite workload, mixed-width sheets, contact sheet, and manifests.

The native project now builds five experiment harnesses. vdspr_phase5 through vdspr_phase8 preserve earlier checkpoints, while vdspr_phase9 runs the final reference-rig study. External FBX scene objects still terminate at the retarget boundary; the reconstruction layer consumes only NM-PM pose data.

Architecture

An architecture review was performed before articulation. The Phase 5 harness had grown to roughly 77 KB in one source file, with subject data, sprite reconstruction, capture I/O, experiments, and UI mixed together. A limited refactor extracted the parts that must remain identical across experiments instead of rewriting the whole application.

The detailed review is preserved in ARCHITECTURE.md. UI helper duplication is currently tolerated, but reconstruction and pose mathematics must not be duplicated between experiments.

Repository

The canonical source is maintained directly inside the NM-PM monorepo:

nm-pm/
└── projects/
    └── sprite-projection/
        ├── index.html
        ├── DOCUMENTATION.html
        ├── README.md
        ├── RELEASES.md
        ├── DEPENDENCIES.md
        ├── CMakeLists.txt
        ├── src/
        ├── tests/
        ├── experiments/
        ├── figures/
        └── assets/

index.html is a stable entry point that redirects here. The documentation itself remains standalone so it can still be copied, archived, or opened locally without relying on shared CSS or JavaScript files.

AI Transparency

Generative AI was used extensively as a development tool for implementation, debugging, mathematical exploration, and documentation support. The study direction, constraints, testing, visual evidence, and acceptance or rejection of approaches were decided by the human researcher.

Limitations

• The pitch bank currently clamps outside +/-60 degrees.
• GRID4 can ghost when visibility changes strongly between stored views.
• Perspective projection is outside the current experimental baseline.
• SP-007 and SP-008 use complete-part center Z for ordering. More complicated overlap may require finer depth handling.
• Articulated sprites still use a fixed proximal pivot at the capture center rather than arbitrary per-view pivot metadata.
• The final rig still omits fingers as separate articulated sprites, facial parts, clothing, and per-bone twist preservation.
• Hands and feet deliberately use only four yaw anchors; E9 may show conditions where that density is insufficient.
• Whole-part center-Z sorting can still fail for interpenetrating or internally depth-spanning sprites.
• FBX bone-name mapping is heuristic and requires a recognizable humanoid hierarchy. Missing required major joints reject the import.
• Imported root translation and arbitrary source proportions are not preserved in the retargeted research body.
• Phase 3 through Phase 9 numerical findings still require native execution and review.
• The final harness is a reference candidate; reconstruction-method promotion still depends on reviewed evidence.

Research Log

Version History

Public milestone releases use separate tags such as sprite-projection-v0.1.0.