In my previous blog posts from this series I talked about background processing in Hangfire and Azure Functions, now I'm going to talk about Quartz.NET and my experience with it. As I said before, these are not just "hello world with < insert technology name here >" type of posts, but real "war stories" from my day to day projects.
Open-source job scheduling system for .NET
https://www.quartz-scheduler.net/
Quartz.NET is a C# port of a very popular Java framework for job scheduling called Quartz. I only recently played with it when I had to implement some recurring jobs in a .NET desktop app, and my first impressions are pretty good. Let's see how it works!
Problem to be solved
We had a desktop app installed on multiple Windows machines, which allowed users to create orders. Our new feature allowed them to schedule orders whenever they wanted. This means that they didn't had to choose between a fixed schedule but they could create their own, and the customization level was pretty high. They could schedule an order daily at a certain hour, in certain days of the week, every few weeks or months, on the Xth day of the month, etc.
Solution
The solution I picked was based on Quartz.NET because it was easy to convert a schedule defined by the user into something that this library could understand and execute.
This was only the first issue, because I had to also find a way to send the schedule from my desktop app to the server. Of course, there are multiple ways to do this, ranging from the communication protocol(send over HTTP, sockets, etc.) to the way we store our job definition(schedule, execution point, etc.)
After a bit of research I decided that I will not reinvent the wheel and I'll use something that Quartz.NET offers by default: the JobStore.
What is a JobStore?
In order to execute a piece of code with Quartz.NET you need to specify two things: the schedule and the job that needs to be executed (a class that implementes IJob). Once you do this, Quartz.NET will save this info in a JobStore. The schedule will be saved as ITrigger and the information that gets executed will be stored as IJobDetail.
The most popular JobStore is the RAMJobStore, which as you might have guessed already, it stores the job data in memory. We already used the RAMJobStore in our Windows service, but the downside is that they are not persistent. In some cases this is just fine, we for example stored our jobs in XML files that were read at startup, so even if the service crashed, the next time it would be back up it would schedule those jobs again. However, in this new feature request our schedules are defined by users, so they are not fixed. This means we could use the RAMJobStore, but after a restart the schedule is gone and we would have to ask our user to create the job again. Obviously this is not a good idea, so I decided to use the AdoJobStore.
AdoJobStore
This type of store allows us to store our data in a database like SQL Server, MySQL, SQLite, etc. For me this was perfect because I already had a database which was used by both the desktop app and the Windows service. Now all I had to do was to install Quartz.NET in the desktop app, give it the database credentials, and use its API to schedule jobs. What's cool about this is that you don't have to restart the remote service every time a job is created. Users can create a new job and Quartz.NET will pick it up immediately on the remote server.
There are plenty of tutorials about this so I won't cover it in detail, but I do want to give you an overview with a few samples of code. You'll see how I used the Quartz.NET API to store the job information in the database and how I scheduled the job in the Windows service.
The advantage of this solution is that I don't have to worry about communication between my app and the remote service. Everything is done by Quartz.NET once I give it access to my database. I should also point out that Quartz.NET will create a few tables where it will store triggers and job information, but this wasn't an issue for me. I just had to make sure that those tables were created with a migration, because Quartz.NET won't create them by default. For SQL Server, you can use this script.
Error handling
If my job has failed I have to run it again using exponential back-off. To do this in Quartz.NET you need to register a listener for your job that detects when it has failed and reschedules it accordingly. The flow would be like this:
- Register a listener for the job
- Detect when the job has failed and inspect the error
- If we want to execute again, we calculate the time for the next execution and reschedule our job
- We do this until the maximum number of executions was exceeded or the job ran successfully
In order to implement this I followed this tutorial which had almost everything I needed, except one thing. The author was using as example one-time jobs. This meant that the job had to execute only once, so it was ok to reschedule it a few minutes later in case of failure.
Let me give you an example to understand this better. One-time job is like saying that I want to execute this job next Monday at 3 PM and that's it. If it fails, then I'll reschedule from Monday 3 PM to Monday 3:15 PM. If it fails again, I'll schedule it for 3:45 PM, etc.
A recurring job would execute EVERY Monday at 3 PM if we use the same example. If I alter the schedule and say that I want to execute 15 minutes later, then it will only execute at 3:15 PM and stop. If I make it execute every Monday at 3:15 PM then this means I've altered the initial trigger, and my customer won't be happy because they want 3 PM, not 3:15 PM.
So my solution was to leave the initial trigger alone, and create another one because you can have multiple triggers for the same job. In this way I kept the "every Monday at 3 PM" trigger and I created another one-time trigger at Monday 3:15 PM. If this one fails, I'll create another one at 3:45 PM, and then maybe I'll stop, but the initial one remains unchanged and the next week it will execute again on Monday at 3 PM.
By doing this I also had to store the number of retries in the JobDataMap of the trigger, not the one of the job because it wouldn't persist, even if I used the [PersistJobDataAfterExecution] attribute. I don't know why this happened and I wasted a few hours, but it worked when I stored the data in the trigger as I did below. I hope that if you landed here because you have the same issue then you won't waste as much time as I did.
Issues
Before I go on I should mention that I used Hangfire in production for almost 2 years, but the solution I rolled out with Quartz.NET has just reached production a few months ago, so it's not a fair comparison. It should come as no surprise that I haven't had any major issues with Quartz.NET
Conclusion
I have to admit that the Hangfire setup was easier, and the fact that Quartz.NET doesn't have a built-in dashboard means that I have to look directly in the database to see which triggers fired or not, but these are not major issues. In the end it got the job done and I would happily use it again if I have the chance.
