Researchers being researchers
This post describes the process of designing and building a holographic projector from an old laptop to enable people to interact with the research we develop in the Computer Science department, at Birkbeck University. You’ll find a short video at the end of the post if you want to skip ahead.
A common requirement of funding for research projects is to provide some benefit to society. It is often difficult to envisage how we can get our research out to the public, especially when the output is often targeted at commerce or other researchers.
Our research focuses on developing search and text mining tools that can operate with any collection of texts in any language, the result of which was Samtla (Search And Mining Tools for Labelling Archives). Search engines often store documents, such as web pages, as collections of words. Users enter keywords into a search engine, which locates the web pages containing mentions of those keywords. This is an oversimplification, but the point is that systems using words as the identifiers of what web pages are about require a lot of work to develop. We have to preprocess the text of the web pages, dividing them into words and removing any parts of the word related to the grammar of the language. This allows us to find web pages mentioning the same topic without caring about whether authors used a different tense when describing the same topic. However, by removing the grammar of a text, you often lose some information about its context or interpretation. Samtla allows people to search their text collections without any preprocessing. This is only one benefit, we also use Statistical Language Models to rank the documents, which captures to some degree, a level of semantics encoded in the text.
The idea came from my research on Samtla and my personal interest in hacking electronics, combined with a recent British Library Labs event at the London School of Economics. There they mentioned the release of a new collection of 3D scanned objects on SketchFab. We also used a 3D scan of an Aramaic Magic Bowl by Richard Webb, which was kindly shared by the University of Exeter Digital Humanities Lab. The Aramaic Magic Bowl texts were some of the first corpora used to develop the original Samtla system.
The holographic projector could be situated in a room and experienced by a group of people since you can view it from four different angles.
Holographic Projector ahoy
A holographic projector is based on the principle of Pepper’s Ghost, which in reality means its does not generate a true hologram as those seen in your favourite SciFi movie.
If you do a Google search you’ll find plenty of tutorials, including a DIY version for your mobile phone or tablet:
The core idea was to allow users to search and browse the collection of 3D objects in a way that was more intuitive than using a mouse and keyboard. I decided on a kiosk-style holographic projector. The main restriction on the implementation of Samtla HoloUX was that it had to be cheap to create and use materials that I had readily available since I had a budget of zero.
I opted to recycle my laptop, as well as the materials (perspex, MDF) from a poster frame that was being recycled by the department. The overall cost of the project was £74, which was mainly spent on an Airbar for a 15.6 inch display to enable the use of gesture-based controls (I was originally planning on turning the laptop into an oversized tablet). The Airbar is listed as being compatible with Windows 10, but the great thing is that I have found it to work with Linux (Mint and Ubuntu) out of the box!!
Bill of materials
The materials needed to build the a holographic projector include:
- 1 x old laptop or LCD screen (mine was an Acer Aspire 5551).
- 1 x Airbar for a 15.6 inch display (or whichever size fits the dimensions of your display, 15.6″ is the max.)
- Clear acrylic or perspex: used to create an acrylic sandwich to house the laptop parts to create a lower profile, as well as the construction of a 3D pyramid to render the projection from the laptop display.
- Wood or plastic to construct a frame to support the laptop.
- Neopixels for adding a futuristic element to the display (this is part of future work and will be housed at the bottom of the projector.
Software tools include:
- Samtla for indexing and searching the metadata.
- ThreeJS for rendering the models.
- Blender3D to transform the SketchFab models. for threejs to read.
- Google speech API
- Responsive Voice JS Text to Speech library
Users interact with the Samtla HoloUX using voice or hand gestures. Voice allows them to activate certain functionality such as the browser and the search tool, whilst hand gestures enable users to manipulate the 3D objects in situ either by swiping to move to the next one, or zooming and rotating to get a better view of the current object.
Building a holographic projector from an old laptop
The project is pretty easy to setup. The main idea is that we project an image onto each of the four sides of the pyramid to give the impression of a 3D object materialising in front of the user.
ThreeJS allows us to divide the display space into four different 3D projections taken from the front, left, right, and back of the object. This allows more than one person to view the object, or allows the same viewer to walk around the object to view it from all sides as if the object was physically present.
In terms of the back-end, we use Samtla to index, search, and rank the objects based on the stored metadata (see our recent paper to learn more). The voice queries are converted to text using the Google Speech API and dispatched to the Samtla retrieval engine, which uses a character-based n-gram language model rather than the conventional word-based one so as to achieve greater flexibility in language agnostic query processing, as well as compensating for errors in the query resulting from user error, or errors generated by the recognition process. A set of ranked documents (3D objects) are then returned to the user, where they can then browse through the search results by using the swipe gesture to move through them.
Despite its simplicity in terms of construction, the effect is quite mesmerising. Naturally you can use a much bigger screen, such as an old monitor or TV to increase the display size, however, since this is a prototype, and I’d quite like to be able to transport it on the tube to demonstrate, I decided to keep to the current form factor.
I see this new physical interface replacing those cards you find in Museum exhibits that state the object has been removed for photography or research. If these missing objects were replaced by a holographic display, visitors would still be able to get something closer to the missing exhibit, than simply some text or a static photograph.
A short video demonstrating the use of voice and gestures to interact with the prototype:
Over the next few days I plan to optimise the code to improve the overall performance, particularly in terms of rendering the 3D models.