2022 Retreat Hackathon Projects

Among other things, we organized a one-day internal hackathon to experiment with our platform and see what interesting applications we could make with it. Here are the projects that came out of the hackathon👇:

Speckle Functions

Team: Connor, Iain

Tools/Tech Used: OpenFaas, vue.js, node, flux, kubernetes

Speckle can do a lot more than connect two applications together, it also hosts your 3D model on the web and allows applications to integrate with this data. This can bring a whole host of extra features and power to your 3D data. In the future we’d love to see Speckle being used to create and automate all sorts of actions, from automatically generating documentation and drawings whenever a 3D model is changed, to automatically analyzing the 3D model for compliance with specifications or regulations.

We’ve previously published a number of tutorials, videos, and blog posts about how to build rich applications on top of Speckle, using Speckle’s API and webhook. A common piece of feedback about the webhook and API is that the experience could be improved, both for application developers and for Speckle users. We’d love to see a future where it is as simple and intuitive as possible to add additional functionality and customization to Speckle. To solve this, we prototyped Speckle Functions during our hackathon.

The prototype allowed application developers to publish their application to Speckle, then make it available to others in a marketplace of applications. Users could then select the application from the marketplace and connect it with their Speckle stream. In our prototype, whenever a user created a commit to their stream, the application immediately got to work analyzing the data they had added.

There are endless possibilities for what could be created and connected to Speckle, and we’d love to hear what you would like to see!

Speckle Corrupted

Team: Fabians, Alex, Kate

Tools/Tech Used: THREE.js & @speckle/viewer for rendering, cannon.js for physics

One of the core features of Speckle is the 3D Web Viewer. It allows users to inspect their 3D data visually, right from the comfort of their web browser. While we have spent a lot of effort already making the Viewer user experience great, one of the features that we feel is missing is being able to load the 3D data inside a physics-based world in which users could walk around and experience the things they’ve built similarly to how they would in real life.

This problem is what we decided to explore in the recent Speckle Hackathon. To make this exploration more fun we decided to take a step further and implement a “first-person shooter mode” in the viewer - with guns, baddies to shoot, and everything! The original Doom game was a revolutionary technological experience at the time of its release and a classic in the first-person shooter genre, so we also wanted to pay tribute to it in our new Speckle first-shooter experience we so affectionately called - “Speckle Corrupted” 🔥 ☠️

The result was a quirky shooter experience you could enable by setting the following URL query string parameter - “doom=true”. Unlike the standard viewer, this one loaded the underlying Speckle Stream as a 3D level with real physics and collision detection. We had the ability to walk around with WASD keys, a nice skybox, and even non-player characters sprinkled around the map. Most importantly - you could even listen to the original Doom theme while playing!

Getting this far wasn’t easy, however, because even with our resident viewer expert Alex by our side, hacking the Speckle Viewer to support actual physics and ensuring that the physics are smooth was a struggle. 3D objects in Speckle streams sometimes have collision boundaries that don’t make sense within a traversable game environment. We’ve used the cannon.js library for all our physics needs since it is more user-friendly, especially in a hackathon scenario. We chose to go with a standalone physics library rather than using a three.js pre-integrated one because it is much easier to use and avoids unnecessary bloating. Additionally, at this point in time Speckle streams don’t support textures and animations, so introducing them required setting up maps between arbitrary Speckle Stream object values and specific textures and animations.

In the future, we’d like to use the lessons we learned to build a traversable first-person experience into the core Speckle viewer so that anyone could enable it in their streams. While technically challenging, it would make the visual review & demo process much nicer and more fun for Speckle users.

Speckle Diffr (diffing, for real)

Team: Jedd, Claire, Alan

Tools/Tech Used: Speckle SDK, Speckle Rhino+GH Connector

Speckle’s original mission statement  - to be ✨ the GitHub of AEC ✨ - was never truly formalized into a feature that lets you compare differences in 3D models and validate changes, object by object. This was a keystone in our platform’s ambitions because current workflows for tracking changes as a project develops rely heavily on drawing annotation and long coordination meetings, and most painfully, they require the reviewers to tediously find every model element that has changed before the coordination can even begin. This is becoming an increasingly impossible task with the emergence of hyper-complex building information models, often with a level of detail that is extraordinarily time and money expensive to review.

We hacked together a proof of concept for actual diffing in our Rhino and Grasshopper connectors, relying on a combination of an object’s id and applicationId to determine if it has been added, deleted, modified, or remains unchanged when comparing two commits:

We learned that true 3D object diffing is really hard both technically (detached properties make our immutable ids not so immutable) and semantically (which object property changes do users actually care about?), but essential to improve the quality of life for any professional working with large and complex 3D models. Hopefully, this hackathon hack will be released as an alpha feature in the near future!

Project SINeRF

(Speckle Integrated Neural Radiance Field)

Team: Bilal, Rey, Gergő

Tools/Tech Used: SpecklePy, Nvidia’s NeRF, plyfile

A year ago in our first retreat hackathon, we went from taking a simple image to generating a point cloud.  This year we decided to go one step further in this hackathon by trying to generate a mesh from a video. There were several components involved in the creation of this hack.

1. Creating an app to host capture the video and host it on a server
2. An appropriate machine learning model to host and generate a Mesh from a video
3. Using the python Speckle SDK to convert the native format of the Mesh into an acceptable speckle Mesh

The main benefits that can come from an app like this come with data collection for further data analysis;  the following are just some of the many benefits.

• Generating mesh data sets from video
• Segmentation and machine learning on meshes
• Having mesh objects of real-world objects for modeling purposes and coordination

In going through this hack, we’ve managed to successfully complete tasks 2 and 3 during the timespan of the hackathon.  For the machine learning model, one of the main obstacles we encountered during the hackathon was a lack of hardware resources to support the amount of training required to properly train a model for speed and compression. We ended up using Nvidia’s NeRF for the model. This offered an advantage in not requiring positional data that’s often required in other models that are often captured and preprocessed with specific hardware requirements.

Above is a generated mesh from a video of the mailbox near the hotel. The output was a ply file which we converted using Speckle Python’s API into an acceptable mesh resulting in something you can interact with on the SpeckleViewer.

In the future, we hope to be able to support people to be able to deploy models and use Speckle as part of the machine learning pipeline for 3D data. In combination with its powerful viewer, Speckle offers a great deal of functionality and has the potential as a future tool kit for data scientists and machine learning enthusiasts.

Conclusion

These are the projects we come up with during our 1-day hackathon! We are planning to take next steps to implement some into our platform. If you are interested in hacking your way into Speckleverse, feel free to let us know in our community!