< Back
Thumbnail Sam
Educational

Building FX-as-a-Service - Balancing Tech Risk and Velocity

FX risk management
Fintech
New technologies
Sam Hunt Mini

Posted by Sam Hunt at MilltechFX

'3 min

16 September 2022

16 September 2022

In 2012, Knight Capital had a catastrophic technical error. A legacy for loop within their trading system sent out 4 million unwanted trades within 45 minutes. This error ended up costing the company $440m. Within the Financial Services world, this story is well known and illustrates how a single mistake can bring the downfall of a company.

At the same time, in the tech world, companies like Facebook and Airbnb were promoting the mantra of “move fast and break things.” Employing techniques such as error budgets (the number of allowable errors for a product or service) to encourage fast shipping of features, even if this led to errors. The argument here was that it was cheaper and more efficient to recover services and fix issues when errors occurred, rather than prepare for every eventuality and slow delivery.

One of the challenges that lies at the heart of building a successful fintech, is blending these two attitudes into a single effective culture. This is something that we work hard on at MillTechFX and have had success with. Whilst this will always be a work in progress, here are some of the techniques that are working well for us.

There be dragons

Within any product or service, there are areas of higher risk. These are the areas where the dragons live and extra caution is required. Identifying these areas, openly discussing them in the team and creating a set of design principles for these areas ensure that everyone working in these areas understands the risk. For us, this included principles for validating early and often, minimising transforms of orders and being very clear where retries are permitted.

Commandos not Cowboys

A mantra one of our board members shared with us, that we used a lot, was “Commandos not Cowboys.” This phrase helped us to identify and discuss activities we were doing that may bring excessive risk. It encouraged individual and collective responsibility within the team and reminded us to invest time in activities that reduced risks such as simulations, post-mortems and documentation.

DORA is your friend

We established the DORA (DevOps Research and Assessment) metrics of deployment frequency, change lead time, change failure rate and failure recovery time. We pull this information from our ticketing system and CI/CD tooling and share it with the team. This enables us to have discussions, supported by data, on whether we are balancing velocity vs risk. It also allows us to articulate the value and track the impact of improvement initiatives.

Automation first

We employ a microservices architecture, and are firm believers that the only way to deliver quality at pace is with an automation-first approach to testing. We prioritise automation testing activities and ensure releases do not go live if they do not have sufficient automation test coverage.

Shift security left, far left

Shifting security left within development pipelines is a core principle of DevSecOps. It’s a good principle that encourages early security reviews and automated guardrails. We choose to take this further and embrace the idea of shifting security left on the organisation timeline, as well as in our development pipeline. There comes a point for most organisations where security certifications are required to build trust with customers, partners, and auditors. We chose to bring this forward, building out our security capability and certifications before they were required, to help increase our own confidence in our security.

Don’t be ashamed of your tech debt

Due to the high pace of growth of scaleup businesses, you are always having to make tough product decisions. Do you build a lean version of a feature that solves your current need, or a more generalised version that could support future needs? As your product discovery capabilities are also being established, you are often making decisions with incomplete information. Creating tech debt is an inevitable outcome. This tech debt can quickly saddle you with legacy before your time. It’s important to be open about it and set expectations for the areas that will need to be revisited. Work with your product people and functional stakeholders to prioritise this work in, as there will never be a good time otherwise.

We hope this was useful and will continue to share what we learn along the way as part of the building FX-as-a-Service blog series.

Sam Hunt Mini

Sam Hunt, CTO

With 13 years of technology experience, Sam's previous roles included establishing the Emerging Technology practice of Capgemini UK and technical leadership of the FinTech Practice of Star Global. As a true technologist at heart, he continues to explore and champion new technologies, frameworks, approaches and cultures.

Posts by tag

Other posts you may be interested in