Adfin | On call at Adfin: how we keep things running smoothly

At Adfin, we help businesses get paid faster and take the hassle out of credit control. That means our systems need to be available and responsive around the clock because invoices are sent, payments are received, and cash flow is chased at all hours of the day.

We want to offer the best possible service to our customers, but every software has issues and bugs. Therefore, we aim to identify and fix any issues before our users even notice them - and if they do notice and report an issue, our definition of awesome is saying that “Thanks for reporting it, we’ve already identified it and the team is shipping a fix right now”.

This post is a behind-the-scenes look at how we handle on-call, ensuring we're always ready to respond to issues and keep everything running smoothly, day or night. It is written to be digestible for all levels, whether you're an engineer or someone simply curious about how we stay online 24/7.

What it means to be on‑call

At Adfin, our backend engineers rotate weekly in the on‑call guardian role. This means being ready to respond to incoming alerts, whether it’s a sudden spike in errors, a failed payout, or a customer flagging something that doesn’t look right. We subscribe to the “you build it, you run it” mindset, meaning that those who write the code support it in production. This creates tight feedback loops, faster resolution times, and a strong sense of ownership across the team.

How incidents reach us

When something goes wrong in our systems, like a payment not going through or a screen not loading properly, we rely on a few key tools to help us spot and fix it quickly.

Every time our software does something, it leaves a digital trail called a log. These logs give us an audit of what was happening under the hood, such as whether an action was successful or not. We collect and monitor these logs using Datadog, which gives us a real-time view into the health of our systems. It works much like a car dashboard, showing us warning signs, performance metrics, and unusual activity so we can spot and respond to issues before they turn into bigger problems.

When Datadog detects an issue, it automatically raises an alert through incident.io, which we use to manage and respond to incidents. incident.io is our incident management tool. It creates a central place for us to respond to issues when things go wrong. It tells us who’s on call, what needs to be investigated, and gives us a running timeline of everything that’s happened. Because it connects directly to Slack, we can coordinate our response in real time, without switching between tools. That means less confusion, faster resolutions, and a clear record of what happened and how we fixed it. Having everything in Slack, from assigning ownership to sharing updates, keeps the entire team in the loop and helps us move quickly when things go wrong.

Even if the problem only affects one customer or seems minor, we take it seriously. We dig in, figure out what went wrong, and fix it properly so it doesn’t happen again.

What happens during an incident

Every incident is different, but there are some typical steps we follow to make sure we stay calm, respond quickly, and resolve things properly.

When an incident is raised, the on-call guardian acknowledges it in incident.io, which lets the team know someone is on it.

We then begin to investigate. This means diving into logs and metrics, checking what the system was doing at the time, and looking at recent changes or related issues. Our aim is to identify the root cause, not just the surface error.

Next, we try to understand the impact. Is this affecting just one customer or many? Is money movement involved? Are downstream systems affected? Knowing the scope helps us decide how urgent the response needs to be.

Throughout the process, we communicate in Slack. Thanks to incident.io, there’s a dedicated Slack channel per incident where we share updates and coordinate the response. The on-call guardian is typically the incident lead, but in certain cases, the right people need to be involved and the incident lead could be assigned to someone else.

If we find a straightforward, safe solution, we’ll often apply a hotfix. This is a small, focused code change that can be deployed immediately, rather than waiting for the next scheduled release. This helps us fix urgent issues fast without compromising stability.

Finally, once everything is back to normal, we close the incident. We write a short summary on what happened, what we did to resolve it, and whether any long-term changes are needed. In more serious cases, we’ll do a post-incident review so the whole team can learn from what happened.

It’s not a rigid process as every incident is unique, but having this structure helps us stay organised and responsive, even when things go wrong unexpectedly.

How we prevent issues from reaching production

Each backend engineer takes a one‑week on‑call shift. In the week before their shift, they act as the Dev Guardian, with a focus on preventing issues before they reach customers.

We work across three environments: dev, staging, and production. Dev is used for everyday coding and early testing, staging is our pre-production test environment that is also used by our platform integrations, while production is our live system.

The Dev Guardian reviews both the dev and staging environments for errors or unusual activity, and shares anything suspicious with the team. They assign owners to issues that need fixing, so bugs are dealt with before we release them to production. They're also in charge of checking our end-to-end (E2E) test results. These are automated tests that simulate how a real user would interact with the product. These tests are crucial for catching unexpected issues, and the Dev Guardian ensures they're running smoothly and highlighting any genuine problems.

This early focus on quality helps us keep production stable and our customers happy.

Release cadence

We release new changes once a week, and the on-call shift is timed to support this cycle.

On Monday, the on-call guardian starts their shift by preparing the system for release. This includes a brief code freeze (a pause on merging new changes) to make sure we’re testing a stable version of our product. They then perform a regression test of our key flows, checking that core features such as invoice creation and payments are working as expected.

Once everything looks good, we deploy to staging, our pre-production environment. This gives us a final opportunity to catch bugs or regressions before promoting to production on Tuesday, when the release goes live for customers.

Why this works

What sets Adfin apart is how much effort we put into preventing issues before they happen.

Proactive mindset: We don’t wait for incidents to escalate. We treat every exception, failed test, or minor complaint as a signal worth investigating. This helps us catch problems early and fix them before they affect customers.
Structured ownership: Our rotating Dev Guardian and On-Call Guardian roles give each engineer dedicated time to focus on quality and stability. Just as importantly, they offer exposure to the full platform. Every engineer gets hands-on experience across all parts of the system, building a deep understanding of how everything fits together.
Built-in escalation: By using incident.io and Slack together, we can respond to problems quickly and clearly, without confusion or duplicated effort.
Disciplined releases: Our weekly release rhythm, with staging checks, manual tests, and hotfix options, gives us confidence that what we ship is safe and reliable.

In summary

Keeping the lights on at Adfin isn’t just about reacting to fires. It’s about building systems, habits, and culture that make fires less likely in the first place. We work together, share what we learn, and treat reliability as something we all own.

At Adfin, we care deeply about delivering the best possible experience for our customers. That means every alert, test failure, and minor issue gets our attention.

On call at Adfin: how we keep things running smoothly

Read more

Why the new Fair Payment Code is a game-changer for small businesses - but there's still more to do

Meet Lena Gryn, our new Head of Product Design

Neil Criddle: betting on Adfin early