Halon Entertainment helped preserve the world’s largest animal using LiDAR, photogrammetry, and a healthy respect for marine anatomy. We sat down with Halon’s Andrew Ritter and Jess Marley to unpack the technical choices, on-site surprises, and what it takes to turn a blue whale into a rigged digital asset ready for AR.
Jess Marley is a creative supervisor at Halon Entertainment with a background in virtual production, realtime workflows, and Unreal Engine toolsets. For the Ms. Blue project, he initiated the digital preservation effort, led the scanning strategy, and oversaw asset creation from phone scan to AR-ready whale.
Andrew Ritter serves as producer at Halon Entertainment, with experience managing CG and technical teams across previs, virtual production, and VFX. For Ms. Blue, he coordinated field capture, data processing, and technical integration—ensuring the whale didn’t swim off mid-pipeline.
DP: How did Halon get involved in the Ms. Blue project?
Andrew Ritter: Jess went for a run! Right place at the right time. Jess has to tell this story himself.
Jess Marley: While visiting some of my family up in Santa Cruz, I went for an evening run and happened to sort of stumble upon Ms.Blue and the Seymour Center. I stopped in awe, took a few photos, and marvelled at her before continuing on.
I returned back to talk with my aunt Martha (who also volunteers at the center); she mentioned that some of the bones needed to come down due to safety concerns, and that Ms.Blue was pretty important to the community. Light bulb. The following morning I went back to the Seymour Center early, and with a nice fog settling in around 6:30a, perfect for scanning. Using my phone and reality scan, I spent the next 30 minutes dragging around a rickety bench, climbing on railings, and trying to cover as much of her 87ft skeleton as I could in 300 photos.
After processing that scan, exporting it to sketchfab; I then had a prototype AR (augmented reality) model which I could walk around and share. A digital double, which could likely replace her in a new digital medium. We contacted the Seymour Center, had a meeting with Jonathan and his team. The project took off. To start, we determined we would scan her, create a high resolution digital double; and then design and create two AR apps for the center.
DP: Why did you choose LiDAR scanning over traditional modeling or photogrammetry?
Jess Marley: We wanted to be as accurate as possible, and do so without damaging or removing the bones from their current position. The LiDAR is used to capture highly accurate down to the millimeter geometric information. That is paired with traditional photogrammetry photography which is then used to capture color data and reproject the whale’s texture information back onto that geometry. When captured and processed correctly, it provides stunning details which can be baked down onto lower resolution meshes used for real-time playback.
Andrew Ritter: Our goal was to end up with a multi-purpose model of the skeleton for 3D printing, AR, and digital preservation. This was LiDAR combined with photogrammetry to make sure we’d have millimeter accuracy and high resolution textures for each and every step.
DP: What was the situation “on the ground” when you arrived at the Seymour Center?
Andrew Ritter: Ms.Blue’s skeleton is fragile and mounted in place near the main museum building. It’s also outside, on the windy coast, and surrounded by landscaping. Oh and she’s the world’s largest animal at 87 feet in length…
Our capture team, Pat Surace, Tripp Topping, and Meredith Brace Sloss spent a full day scanning her from multiple locations–including from up on the museum’s roof–to ensure that we’d have what we needed once we retreated back inside.
DP: How did you prepare for the scanning pass?
Jess Marley: Due to my informal scout, we had a low resolution scan which also allowed the team to examine some of the structure prior to getting started. This gave them additional insight as to areas that might be hard to reach. So they could wisely plan their route and use their time efficently throughout the day.
Andrew Ritter: Pat Surace and Tripp Topping picked our locations and time of day based on google maps data and weather reports. By the time we show up on site, the plan’s already set, we just have to carry it out.
DP: Can you walk us through your use of the Leica RTC360 scanner?
Andrew Ritter: It’s hard to go bigger and better than the RTC360 as it’ll pull all the data accurately without having to get up close and personal. We absolutely did NOT do any crawling through Ms. Blue! And drones would require strict permits (and the cooperation of mother nature! The wind means business out there!) We briefly considered other options, but pretty quickly came to the conclusion that this was the right approach.
DP: What did your data wrangling process look like right after scanning?
Andrew Ritter: We weren’t on a strict production deadline or anything like that, so we simply downloaded the scans to an external drive and brought it back to the office. Pure capture and retreat, no on set DIT.
DP: Once back in the studio, what did your pipeline look like from raw point cloud to usable 3D asset?
Jess Marley: Processing of the LiDAR and photogrammetry went through Leica Cyclone REGISTER 360 paired with Reality Capture. A high resolution mesh with color information was then imported into Zbrush where we could sculpt and clean up any anomalies, fix any bone orientations, and add additional details lost to weathering and time. Then the process begins to fork into two paths; high resolution water tight mesh for printing, and the creation of a low resolution optimized mesh for real-time playback.
In order to print the bones, in sections we brought the high resolution mesh into CAD Fusion. This process involved a final quality check and cleanup to ensure the bones were water tight, clean, and ready for print. Those exports from Fusion were packaged up and delivered to Swellcycle.
The lower resolution mesh was decimated with some quick UVs generated, then exported from Zbrush and imported into Maya. There we refined any smaller and more intricate areas of the mesh, cleaned up UV seams and packed them in preparation for surfacing. The previous high resolution mesh and textures were then projected in Substance Painter onto the lower res asset. Additional work was then done to clean up normal artifacts and finesse the color data (while applying additional textures like roughness, metallic, gloss) to make Ms.Blue PBR ready for unity.
The last phase was to test the app and make changes based on experience. This involved adjusting and embellishing the roughness or adding some weathering in spots to generate some additional areas of interest. Once the Seymour team was happy, we delivered our final to the development team.
DP: You chose ZBrush and Substance Painter for “rebuilding” Ms. Blue. Why those tools?
Jess Marley: There’s more than one way to skin a whale. We chose to use Zbrush and Substance for a few reasons. Our pipeline supports it, and we know both of them pretty well given our experience creating and preparing content for Unreal Engine. Efficiency and control; we had a limited budget and timeline and lots to do; plus other projects on our plate at that time. A few different artists would be shuffling on and off and sharing tasks. So the familiarity with the programs made handoff fast and easy.
DP: How did you approach digitally “repairing” the damaged bones?
Jess Marley: A combination of research, reference, and the Seymour team of scientists and their wealth of knowledge.
Andrew Ritter: Primarily research. We went back and forth with experts at Seymour and their colleagues around the world to give notes on the bones and the skinning. Sometimes it was as simple as “yeah, some teens sat on her fin in the 90s and that’s why the bones are warped. Go ahead and mirror her right fin.”
DP: How much of the animation process was grounded in science, and how much was creative liberty?
Jess Marley: We animated in traditional pose-to-pose fashion. This allowed us to animate, get feedback and refine until the team felt we had the most realistic result possible.
Andrew Ritter: The animation was very grounded in science. We were provided amazing reference footage of blue whales swimming and feeding. Stuff that has only been captured on camera in the last decade. After our passes, they’d be assessed by experts and we’d apply their notes until it was deemed as accurate as possible.
DP: Once the digital skeleton was done, what was your rigging and animation process like?
Jess Marley: Rigging was relatively simple and straightforward. An FK spine and fin system paired with blendshapes to create more complicated animations. This included a tongue, baleen, and her mouth pocket (which expands and moves a great deal while feeding).
DP: For the educational experience: Did you deliver a simple USD scene or a rigged interactive character?
Jess Marley: We packaged up a variety of rigs, props, characters, effects – all in fbx form. Once in Unity, Cavy Labs, the project’s developers were able to implement these animations into the timeline and begin on the spatial and interactive elements. The result was a series of apps for both android and iOS, accessible at the Seymour center for education and entertainment.
DP: What kind of scientific research informed those restorations?
Andrew Ritter: I have to say, we had several asset artists take passes on Ms. Blue over several months and I think they all really enjoyed it BECAUSE the notes we were getting were objective fact. The anatomical accuracy made it easy to go back and forth and nerd out between artists and scientists.
DP: Looking back, what was the most surprising challenge in the whole Ms. Blue pipeline?
Andrew Ritter: One of the most challenging aspects was actually her size. AR models are not easy to keep locked into location, and Ms. Blue has to sit atop her real-life location. We went back and forth with our developer at Cavy Labs and tried all kinds of different techniques before landing on using custom markers situated around Ms. Blue. All to keep her from swimming away…
We also set out to use this model for digital preservation and she’s already making her mark. The same assets that were used for the AR app are used for repairing the physical Ms. Blue with 3D prints and used in virtual production for news broadcasts on CBS and even shared with animators from ETC USC who needed an accurate whale skeleton for an experimental short film. I’m pretty proud that not only is this scan preserving Ms. Blue, but it’s letting her travel far and wide. Can’t imagine a better outcome than that.
DP: Are there any resources or case studies that inspired your approach or helped along the way?
Jess Marley: The team at Seymour had a plethora of research and information for us to use. We were inspired by a piece called “AR dinosaur exhibit” created by Handbuilt Creative. We had seen it while researching possibilities early on in the process (youtube.com). Tiny dinosaurs in a container environment running around in AR, going from bones to skinned. Seemed like a really good concept in relation to blending education with entertainment from our perspective. Conceptually we wanted to hit that balance.
DP: Can you share what Halon is working on next (insofar as it’s not under NDA)?
Jess Marley: As the industry seems to be shifting, we need to find new ways and avenues for work. The technology is now at everyone’s finger tips. I hope that this blend between science and entertainment becomes more regular. A lot of what we do is fun and engaging; I would love to do it on an educational level and do more teaching.
Andrew Ritter: We always jump at the chance to do projects like this with a cultural or educational impact. We certainly use these techniques in our entertainment projects, but we’re hoping that Ms. Blue’s story opens up opportunities to do more of this kind of work.
