Urban planning visualization and content production using AI tools and workflows (Stable Diffusion / Nerfstudio / Blender)
Still image rendered out from a Neural Radiance Field of the brownfield site in the Malung-Sälen municipality
This project formed the final part of my education at Futuregames, through an internship at the Swedish community development consultancy Tyréns. The task was to make informational material directed towards a non-specialist audience for a sanitation project in the municipality of Malung-Sälen in Dalarna.
The project concerned the sanitation and restoration of a brown-field site where an industrial tannery had been located. After roughly hundred years of activity, it there is a significant ecological debt to be paid, with the Swedish state picking up the tab.
The project is quite large scale, especially relative the local community located in rural Sweden. It has already spanned over a decade of tests, planning, and judicial review, and still has many years of process and execution until its completion sometime later in this decade.
Scoping out the project
Photo of the Malung tannery in its prime, colorized with Palette_FM and upscaled with Stable Diffusion.
The main need of the municipality was to communicate the very significant amount of disturbance that would be the result of the project, and that it would be going on for quite a few years. It also included non-trivial risks, such as transport of low-grade toxic landfill and dredging the river for smelly industrial waste.
It quickly became apparent that the best way to approach this project was through storytelling, putting the sanitation project in a historical context, and putting as much emphasis on the park that will come after the project is completed as the years of digging, dredging, and heavy traffic that will precede it.
Taking the future into account opened up a lot of possibilities for presentation, but also posed a challenge. In the future reconstruction phase it is intended for a park to replace the brown-field site, but at the time of my participation the exact look and functionality of the future park was still being defined. This meant being able to show and talk about what was to come in only the most general terms, with any specific feature being at best a suggestion.
In the end, time and necessity dictated that my personal contribution was to be a movie for the homepage and social media, as well coming up with design and content for a trio of signs that would be put on the bridge overlooking the project site.
Making the movie
Finished video done for the Malung-Sälen municipality. Texting in Swedish. Can also be seen at: https://malung-salen.se/bygga-bo-och-miljo/malungs-garveri
Neural Radiance Fields (NeRFs for short) represent completely new way of representing 3D space when compared polygons (used in pretty much all computer generated graphics and movies), voxels, or point clouds. It is related to methods such as photogrammetry and lidar, but has a very different set of strengths and weaknesses.
Above all, it has much greater potential for storytelling since it is capable of capturing context and shifts in light, and not only isolated objects or features. As chance would have it, NeRFs had just evolved far enough, and allowed enough customization, that it was conceivable to try doing a project such as this with it.
Point cloud extracted from the same NeRF that was used to render out the movie. The point cloud can then be used for a number of purposes, the most important being the alignment of the virtual camera in Nerfstudio and the Blender camera. It is then also used to align 3d-objects and shadows with surfaces in the movie. While the point cloud is made invisible for the finished product, parts of it can be retained to create occulsion effects (such as some of the trees in this case).
The workflow starts with using a Mini 3 Pro drone with a wide angle lens to capture the environment around the project site, being careful to get as even coverage in a rough 360 degree circular movement, and to make sure to capture a slice of the horizon all the way round.
The next step involed running the collected movies through the program Nerfstudio on a powerful computer equipped with a RTX3090 card that has enough VRAM to fit the entire AI model at the resolution required. While the methodology of NeRFs has advanced since this project was made, capturing large open world scenes is very demanding on processing hardware.
Layout of composition in Blender between the underlying movie, and the transparent layer with 3d-objects.
The last step involved composing the resulting video with polygon objects and point clouds imported from the NeRF in Blender. NeRFs can export both movies and point clouds, and these can be added together with VFX techniques that create the illusion of not just looking at a movie, but allows for novel objects to appear as part of the movie itself, and even cast shadows.
