Over the time, artworks have become more and more accessible to everyone interested. First, people learned to cover large distances to see well-known masterpieces. Then, the narrowing gap between the rich and the poor-made museum visits more affordable to the general public. And, finally, the evolution of the Internet and digital museums has eliminated any distance-related obstacles and ended the epoch of an elite-only access to cultural heritage. Today, we are leveraging virtual reality (VR) solutions to achieve a brand-new level of relationships with art and artwork comprehension.

Recently, CROC, one of Russia’s largest IT integrators and key NNTC partners, completed a long-term innovative project that brought the entire hall of the State Hermitage Museum into virtual reality. We would like to share a story by Ilya Simonov, CROC’s VR Center Director, describing the whole journey, from engagement to solution search.

HOW IT ALL STARTED

The State Hermitage Museum, located in St. Petersburg (Russia), decided to initiate its own digital transformation by moving its building with all exhibits into virtual reality in order to retain the cultural heritage and make the museum virtually accessible from anywhere. To prove the concept, the customer first decided to virtualize one of the State Hermitage Museum halls — the Jupiter Hall called so because of a huge ancient god sculpture being exhibited there.

Accurate artwork digitization and reproduction in 3D provides the following advantages:

  • Interactivity. Virtual museums are free from limitations people face when visiting real-life museums. Users can now zoom in any exhibit and view it as close as they want to, rise above the Jupiter’s statue and look at the back of its head, fly around and look behind its back, discovering the tiniest details: dress pleats, hair curls, and stone texture.
  • Safety. Many artworks, which were not exhibited before, can now come back to galleries and museums in VR and AR format. Moreover, exhibit digitization is crucial for the retention of the cultural heritage: if a real-life original is damaged, its digital copy will persist.
  • Wide coverage. Exhibits can be demonstrated worldwide and accessed by a wider audience.

HOW TO ADDRESS THE CHALLENGE

Initially, we believed that available solutions would be enough to bring the museum hall into virtual reality, but, in fact, we had to find and pioneer a new combined approach. During the project, we considered three technologies: modeling, laser scanning, and photogrammetry.

  • Laser scanner.Laser scanning returned good sculpture geometry but poor colors.
  • Modeling. A modeler could make a sculpture model from scratch, like in computer games, but we knew beforehand that fine art experts wouldn’t be happy with the outcome as they needed not a piece by a contemporary artist but a masterpiece by a Greek sculptor accurately reproduced with all the tiniest details.
  • Photogrammetry. Photogrammetry analyzes photos, identifies the same patterns and shapes a point cloud from them. For this purpose, the exhibit is shot from different angles. Afterwards, the point cloud is converted into a polygonal model with texture. That very technology helped us digitize the State Hermitage Museum’s hall.

HOW WE DID IT

During the project, we tried different cameras, from a common DSLR camera to a medium-format 100 MP PhaseOne, and finally revealed that it didn’t make sense to opt for a high-end camera. Because of long exposure shooting, even the slightest vibration resulted in blurred pictures, thus reducing texture sharpness and adding unwanted noise to geometry. Moreover, it took too much time even for the server to compute the data obtained.

We used a specially built GPU server to compute a point cloud to be then adapted for various devices. It took two months for four people to optimize all the virtual sculptures and the hall.

The optimization is required to then run a program on devices of different capacity and, more importantly, achieve optimal perception being close to real life. The design was tailored for three types of devices:

  • Home game PCs and museum VR systems (max quality and resolution)
  • School computers (optimization, which causes little degradation, while being effort- and time-consuming)
  • Mobile devices (the most optimized versions)

OUTCOME

We have done it, and now any user can closely explore the Jupiter Hall exhibits: walk across the hall, listen to an audio guide, and even climb to the ceiling to view the design of hall walls. The State Hermitage Museum successfully competes with the Louvre Museum and features the world’s only hall that is scanned for virtual reality with such a precision. We presented the project at FMX, a conference on animation, effects, games and immersive media.

Given a trial and error approach, we completed the first hall in four months. Now, powered by a proven and fine-tuned technology, we can simulate the next hall in just 1.5 – 2 months.