Migrating our background processing workflow to .NetCore
The software development industry has undergone major changes in recent years, with the adoption of .NET Core being a standout shift. As we in Expensya, consider ourselves a top innovative startup, we must embrace this cross-platform technology for its numerous advantages over the traditional .NET framework.
However, the migration to .NET Core requires intermediate steps, as the current Azure Cloud Service (classic) servers used by Expensya are not compatible with the relatively new technology. To accommodate this change, Expensya will transition first towards a new infrastructure: App Services.
Why Migrate? Why these choices?
At Expensya we’re firm believers in Simon Sinek’s “Start with why”, but you need a bit of context in this case for the why to make sense.
We embarked on these migrations for multiple reasons:
- Cloud service is being deprecated: this is the least convincing argument, but Cloud Services are Microsoft’s first venture into the cloud, and they will be retired in favor of newer Azure options. The sunset date for Cloud Service is in 2024 so we have plenty of time, but we wanted to accelerate this work for the reasons cited below.
- Perf gain: as .NET continues to evolve, Microsoft has made it a priority to improve the performance of the newer versions. Every millisecond that we can shave off our response time is a chance to delight users during their Expensya journey.
- Reduced deployment time: with App Service the deployment process is streamlined. While this makes for a quality-of-life improvement for the dev teams deploying daily, it also puts us in a better place for reducing the MTTR (mean time to resolution) for future incidents since we’re not spending so much time waiting for cloud infrastructure once we have a fix ready.
- Advanced monitoring/diagnostics: Microsoft has made lots of investments in App Services and enables new monitoring/diagnostic things that aren’t available in Cloud Services. One of those things is a health probe which pings the servers continuously and if at some point an instance stops responding, recycle it automatically instead of having to wait for human intervention. That’s something that can be added manually with Cloud Services but comes out of the box for App Services
It is worth mentioning that Expensya has already undergone a first migration effort this past summer for our backend servers. As such, migrating the remaining pieces is a must, knowing what it has offered us already!
Impacts on the migrated backend servers:
At Expensya we’ve been driving a cultural change to incorporate data and data-driven decisions into our DNA. Therefore, this document wouldn’t be complete without some data.
Here’s an overview of the server-side response times before and after the migration:
Average response time before the migration: 293 ms
Average response time after the migration: 157 ms
That’s a 136 ms or 46% improvement!
Before crying victory too fast, we should look at the load on the servers during that time as we know there’s the summer effect, and servers that have less work to do often time have better response times.
Despite seeing a 1.5x increase in traffic (when comparing the July/August periods with September) we’ve managed to cut our response times by nearly 50%.
In terms of deployment velocity, previously with our Cloud Service infrastructure, we had to spin up new machines every time we wanted to do a new deployment. That operation alone could take up to 10 minutes and in instances where we needed to deploy a hotfix that’s precious time lost. Now with App Service the machines are always running, so we simply need to push the latest version of the code, and we’re all set.
You might be wondering what we will be migrating during this phase of the project! Last summer we migrated our Backend Server, and this February we will be migrating our Worker!
What is a Worker?
A worker is a separate process designed to perform asynchronous tasks, such as background jobs and online services. By executing these operations in a separate process, workers prevent the main application from being blocked and ensure that it remains responsive. Workers are especially useful for handling resource-intensive, time-consuming, or long-running operations, freeing up the main application to attend to other requests.
But what about Expensya? What do our workers do?
Nearly all automated tasks in Expensya are handled by our worker, which includes OCR, automatic accounting export, and transaction imports. This makes the worker a crucial component of our infrastructure. The upcoming project aims to enhance the performance of our worker, leading to faster processing of OCR, smoother accounting integrations, and an overall improved user experience for our clients.
When will be migrating?
The migration to new services is a crucial step, and this project is no exception. On February 9th, the migration officially begins with the switch from Cloud Service Classic to App Service in production. This will be followed by the transition from .NET Framework to .NET Core, which is set to be completed by the end of March.
Here’s a more in-depth timeline of the migration:
- February 9th: Our OCR will be running on the new platform.
- February 10th: Our BI module will follow.
- February 13th: Client integrations will no longer be on the old platform.
- February 14th: Migration complete! Our platform will be shining and new!
Migrations come with challenges, and this one is no exception. One of the challenges is adapting our testing module to maintain backward compatibility with both the new and old platforms. Our team is taking the time to ensure a seamless transition, focusing on delivering with no disruptions.
Caution:
If you are a customer of Expensya and have a custom integration in place, it is important to ensure that your server has IP filtering set up to allow access from certain IP addresses. In this case, the IP addresses 20.223.189.45 and 94.245.95.138 need to be whitelisted in your firewall in order to ensure seamless operation of your custom integration with Expensya. Please note that this information has already been communicated to all impacted customers during the first phase of the migration project. Failing to whitelist these IP addresses may result in disruptions to your integration with us.
Conclusion:
Expensya is going through a major transformation to keep pace with the rapidly evolving software development landscape. Our team is dedicated to providing you with the best possible experience, and that’s why we’re taking the time to upgrade our technology. The end result will be a more streamlined, efficient, and innovative platform that is better equipped to handle your needs now and in the future.
We’re confident that this transformation will result in a better experience for you and for us. We’re excited to continue delivering even better services in the future and we look forward to your continued support.
