High-performance Distributed Computing Engines

Home / Portfolio / High-performance Distributed Computing Engines
High-performance Distributed Computing Engines

A computing engine is a generic system designed to execute a large number of processing tasks, typically defined by dynamic logic such as a workflow or a set of business rules, on an arbitrary set of input data. The output of the processing is fed back into the system or collected and used for its final purpose. The big data term is usually heard when the processing tasks are relatively trivial compares to the (very) large data sets involved. The emergence of commoditised processing frameworks such as Apache Spark, Facebook Presto or Basho Riak have encouraged the adoption of scalable, distributed computing for stand-alone applications.

However, these out-of-the-box platforms are typically hard to configure, deploy and maintains, and command a significant integration cost when they are made part of a larger architecture. A more popular alternative, cloud computing services such as Amazon EC2 and Microsoft Azure are easier to configure and use, but have limited flexibility and rely on a commercial cloud hosting model that may not be palatable to companies who want tighter control and ownership across their software ecosystems.

Custom calculation engines are a viable option in most scenarios that involve bespoke application architectures and specialised types of processing and data flows, and require tight cohesion with existing components and high performance (throughput, processing speed, latency, etc.). These scenarios are common in data or process-centric industries, such as financials, government, healthcare, telecoms, manufacturing and logistics. A well designed bespoke computing engine is the ideal solution when high-performance, flexibility, robustness, and deterministic timing/latency are required for a specific class or classes of tasks and data. Users of a healthcare monitoring platform, or an algorithmic trading application, would want to take action as soon as possible after a new input is presented to the system.

One of Gluon’s main areas of expertise is in the engineering of bespoke, mission-critical computing platforms. We have written several instance of computing engines, from multi-threaded processing components for monolythic applications to large-scale distributed computing frameworks, complete with high-performance service hosting and communications infrastructure. Depending on business and functional requirements, some of them relied on tightly optimised calculation graphs, some were entirely rule-based, others were based on dynamically configurable workflows. We pioneered, refined and adopted solutions as diverse as actor based parallelism, location-independent, metadata-based service discovery, zero-configuration service orchestration, advanced result caching, Command Query Responsibility Segregation.

Our Clients for this types of systems included three top-tier investment banks, one top-tier broker, and two large hedge funds. Business applications ranged from pre and post-trade risk analytics and reporting, to portfolio valuation, interactive rule-based compliance, and middle-office processing.

Technologies used were: .NET, Java and Scala with bespoke or standard concurrency libraries, high-performance serialisation, high-performance in-memory and socket communications, and assorted middleware (ZeroMQ/NanoMsg, Memcached, RabbitMQ, etc.).

This entry was posted in . Bookmark the permalink.