Inference, synchronization, and AnimGraph architecture

Processing flow

SoundWave → Decode → 16 kHz PCM → Windowed inference → Post-process → Clip
                                                                    ↓
Audio playback clock → Current expression snapshot → AnimGraph → ARKit curves

LAMAudio2Expression is the Runtime module. LAMAudio2ExpressionEditor is UncookedOnly and provides AnimGraph editor nodes, model configuration, example generation, and PIE tests. No engine modification is required.

Implementation	Responsibility
LAMAnalyzeAsync	Async loading, analysis jobs, BP notifications, PCM hashes, LRU cache
LAMDecoder	Cooked SoundWave decoding and streaming-chunk retention
LAMCore	Resampling, window extraction, NNE inference, CPU fallback, post-processing
LAMAudio2ExpressionComponent	Playback control, synchronization clock, expression snapshots
AnimNode_LAMARKit	Snapshot acquisition and curve blending
LAMLive	Microphone / PCM, continuous resampling, live inference, presentation delay
LAMTypes	52 curve names, settings, Clip, Profile, interpolation

Threads and lifetime

Model UObject loading and NNE model creation run on the game thread. Decoding, resampling, NNE model-instance creation, inference, and post-processing use one dedicated worker. Blueprint progress and completion notifications return to the game thread.

An in-flight NNE call is not interrupted. Cancellation flags suppress subsequent work and result notifications. EndPlay and async-action destruction also invalidate jobs.

The AnimNode copies curves and the Profile in PreUpdate. Evaluate_AnyThread uses only that snapshot and SourcePose; it performs no inference or UObject lookup.

Analysis windows and time

Offline analysis advances by 16,000 samples. Each 34,133-sample window produces 64 frames; frames 34–63 are retained. Samples before the start and after the end are zero-padded. The final frame count uses integer arithmetic: (samples * 30 + 15999) / 16000.

Playback uses audio time, allowing at most 1/30 second of interpolation prediction until the next callback. When seeking recreates internal audio playback, notifications from the old playback are ignored. Physical output-device latency is not compensated.

Cache and ownership

The analysis cache defaults to 64 MiB. Its key includes the 16 kHz PCM SHA-1, model GUID, style, post-processing settings, and processing version. A playback Clip owns its result array independently, so cache eviction does not invalidate it. Model memory and Clips retained by callers are outside the cache budget.

Live processing

The pending input queue is capped at two seconds, with separate historical context for inference. Old-generation results are discarded, and delayed processing catches up to the latest position. Version 0.2 supports adjustable live intervals; see live input. Audio routing, concurrency, and event behavior are documented in playback controls.

Saved-clip path (v0.3.0)

Plugin 1f06ac8 adds ULAMBakedExpressionClip, an asset containing analyzed ARKit 52 curves. Playback shares its array and interpolates 52 values, skipping model loading, analysis decoding, inference, and a full-array copy per play. Viseme conversion remains a runtime operation. This v0.3.0 path is separate from the dynamic analysis cache above. See generation and load validation.

Sequencer integration (public source)

Public-source Plugin 1642ffa adds the Runtime LAMAudio2ExpressionSequencer module and Editor LAMAudio2ExpressionSequencerEditor module. Track / Section inherit UE’s standard Audio Track / Section and delegate audio evaluation to UE. A MovieScene Track Instance samples expressions at the evaluation time on the game thread, supplies a snapshot, then updates the mesh. This is separate from ordinary playback’s audio clock.

Sections on the same Actor are combined by weight and pass ARKit 52 values to the existing AnimBP. Viseme conversion and Profile correction remain in the AnimBP. C++ AcquireSequencerControl / SetSequencerFrame / ReleaseSequencerControl use owner tokens; Blueprint exposes read-only Is Controlled By Sequencer. Existing Apply LAM pins and the Clip format are unchanged.

See timing, ownership and release conditions. This path is not included in v0.3.0 ZIPs.