DistART

24 hours, this is the time we were allowed to design and build a highly scalable platform for image processing tools. This is an idea we had during the 2016 Microsoft Imagine Hackaton at EPFL and we won as the best project of the event! The platform is executing heavy image processing code on GPU (the currently famous deep learning Neural Style in this case) that were running on Amazon Web Services (GPU VMs were not available on Azure at that time). The tool we built is three-fold:

Providing an API for uploading images, keeping jobs metadata and keeping a queue of operations.

Executing the image processing code on multiple worker machines and environment, all synchronising with the API's scalable tools to distribute the list of jobs.

Providing users with a nice interface to upload and receive images: for this we chose to implement a Telegram Bot to be able to interact directly from a messaging app.

Each part has been implemented using Node.js that helps quickly experimenting concepts of web services.
The API is running under multiple scalable APP API instances. They run a RESTfull API by greedily storing job metadatas in a Storage Account Table, job images in Storage Account Blob storage to finally put the job at the end of a Storage Account Queue. These 3 entites are highly scalable and helps every actors of the system to interact seamlessly.

The telegram bot is the only non-scalable bit. For the sake of the demo it was running on a RaspberryPI at home, connected to internet under a DynDNS. The idea to build a telegram bot was to demonstrate the wide range of applications unlocked by building a RESTFull API.

The DEMO had to be shut down as GPU machines are costly when running round the clock. However you can check the presentation that we gave at the end of the 24 hours. The project is also open source and available on Github.

Links:

• Neural Style paper
• Azure storage account and App Api
• GITHUB Repository

Technology used: Node.js Azure Cloud Telegram Bot RaspberyPI AWS GPU VMs

BACK