Update operators in MongoDB

MongoDB is a popular database choice for projects that need flexibility and good horizontal scaling. While newer versions support transactions, MongoDB still works the best without them. To achieve data consistency, we can use MongoDB update operators. They can make our writes much smarter, especially if we learn a couple of patterns. In this article, we will write a small service for granting rewards to the users for their activity. We will use MongoDB and update operators to track the user activity. The complete example code in Spring and Micronaut is included for self-study.

What are MongoDB update operators?

In a typical CRUD application, we usually modify whole documents with a single operation. For example, if we want to update a document, we just send a new version to replace the old one. This programming model is very simple. We shouldn’t be surprised that it has a strong framework support. Sadly, in the naive form it has one drawback. When two people try to edit the same document at the same time, they can overwrite each other’s work. We can mitigate that with techniques such as optimistic locking or transactions.

Update operators deal with consistency in another way. They can perform more advanced updates than simple assignments on a document. In the previous approach, we build the new document version in memory. With update operators, we build a recipe of how to modify a document, and then the database applies it on the most recent server-side document version. Why does it work? Because in MongoDB, an operation on a single document is atomic. If two clients request an update at the same time, they will be applied in a sequence.

We can find the full list of update operators in MongoDB manual. Here we will mention only a couple of them that we will use in our example:

• $set: the basic assignment
• $inc: increments the value in a field
• $setOnInsert: in upsert operations, it sets the value, but only if we create a new document (does nothing when we update an existing document)
• $addToSet: adds a value to a set

Practical example

To show how to use MongoDB update operators, we are going to write a simple microservice that grants rewards to the users after meeting certain conditions. The microservice would track the user activity and determine when they met the conditions for being granted a reward. Let’s imagine that our business offers on-line ordering some service (e.g. cleaning). We want to reward people who bought something in 3 different days, and spent at least $100 in total on their purchases. We are going to write our app in Kotlin. Regarding framework, we will show the code snippets in both Micronaut and Spring Framework. The integration tests would use Kotest Framework.

Step 1: the domain model

To code the reward logic, we need to track two pieces of information per user:

1. spent amount of money,
2. the days with at least one purchase.

For tracking days, we will simply store dates in a set. Sets guarantee that the elements will be unique. In this way, multiple purchases on the same day would not count for reward calculation. Each of them would try to insert the same date into a set, which would not increase the element count.

data class Reward(
    val userId: UserId,
    override val spentAmount: Money,
    override val purchaseDays: Set<String>
) : PurchaseDaysRewardable {
    override fun isGranted(visitor: RewardVisitor) =
       visitor.visitPurchaseDaysRewardable(this)

    companion object {
        fun emptyReward(userId: UserId) = Reward(
            userId = userId,
            spentAmount = Money.zero(),
            purchaseDays = setOf()
        )
    }
}

The example code uses the visitor pattern to decouple the actual condition checking from both the entity and the reward calculation service. We’re not going to dive into it here, because probably everyone is already waiting for those MongoDB update operators to appear! So… we’ll use Reward entity only for reading data. With update operators, we are going to send an update recipe to the database. We need another entity to represent our modifications:

data class RewardUpdate(
    val userId: UserId,
    val amountIncrement: Money = Money.zero(),
    val newPurchaseDays: Set<String> = setOf()
)

Such a class clearly describes our intent. Here we actually hardcode what kind of modification we do on each field. If you need something more flexible, it’s also possible, but requires writing some extra code.

Step 2: calculating rewards

The next step is writing RewardCalculator service. It would implement the actual flow: fetching the existing reward document, and determining what to update.

class RewardCalculator(
    private val repository: RewardRepository,
    private val rewardVisitor: RewardVisitor,
    private val timeProvider: TimeProvider
) {
    fun rewardPurchase(command: PurchaseCommand) {
        logger.info {
            "Updating reward information due to a service purchased " +
                "by user: '${command.userId.raw}'"
        }
        val item = repository.findReward(command.userId) ?:
            Reward.emptyReward(command.userId)
        if (item.isGranted(rewardVisitor)) {
            repository.updateReward(
                RewardUpdate(
                    userId = command.userId,
                    amountIncrement = command.totalAmount
                )
            )
        } else {
            repository.updateReward(
                RewardUpdate(
                    userId = command.userId,
                    amountIncrement = command.totalAmount,
                    newPurchaseDays = setOf(currentDate())
                )
            )
        }
    }

    private fun currentDate(): String = LocalDateTime
        .ofInstant(timeProvider.now(), ZoneId.systemDefault())
        .format(DateTimeFormatter.ISO_DATE)

    fun findReward(userId: UserId): RewardDetails {
        // ...
    }

    companion object {
        val logger = KotlinLogging.logger { }
    }
}

We want to prevent purchaseDays set from growing infinitely. For this reason, we do not add new elements to it, if the user has already been granted a reward. Let’s also notice how RewardUpdate() works in practice. Actually, we don’t have to know the existing field values to create it. We put only the new data to append. Later, it’s the database job to figure out how to modify it with the document stored in the collection. If another service instance also tries to write something, the database will gracefully merge both updates. We do not need to use any kind of locking or transactions here.

Step 3: MongoDB document

So far, we were writing only the domain code which should be free of storage-specific or framework code. Now it’s time to write the MongoDB document representation (example for Micronaut):

@MappedEntity("rewards")
data class RewardDocument(
    @field:Id
    @field:AutoPopulated
    var id: ObjectId,
    @field:BsonRepresentation(BsonType.STRING)
    var userId: String,
    var spentAmount: BigDecimal,
    var purchaseDays: Set<String>
)

Here’s the Spring version:

@Document("rewards")
data class RewardDocument(
    @field:Id
    var id: ObjectId,
    @field:BsonRepresentation(BsonType.STRING)
    @field:Indexed(unique = true)
    var userId: String,
    @field:Field(targetType = FieldType.DECIMAL128)
    var spentAmount: BigDecimal,
    var purchaseDays: Set<String>
)

Although we see that both frameworks have the same approach, the actual API-s are slightly different. Also, the level of support varies. Spring allows us declaring indexes directly on document DTO-s. In Micronaut 3.5, we have to write a custom index creator executed at application startup. On the other hand, Spring requires to provide an extra hint for handling BigDecimal values due to the backward compatibility with older MongoDB versions. Without this hint, an attempt to use $inc update operator would fail.

Step 4: repository and MongoDB update operators

The final part is writing the repository code for our collection. Let’s start with Micronaut example. Until recently, this framework did not have any support for declarative repositories. This feature appeared only recently and is not very mature yet. In fact, I had some hard-to-explain trouble with proper type handling during deserialization. For this reason, we’ll continue writing the repository by hand:

@Singleton
class RewardMongoRepository(private val mongoClient: MongoClient) {
    fun findByUserId(userId: String): RewardDocument? {
        return collection().find(eq(USER_ID_FIELD, userId)).firstOrNull()
    }

    fun update(
        userId: String,
        spentAmountIncrement: BigDecimal,
        newPurchaseDays: Set<String>
    ): Long {
        val result = collection().updateOne(
            eq(REWARDS_COLLECTION, userId),
            Updates.combine(
                Updates.setOnInsert(USER_ID_FIELD, userId),
                Updates.inc(SPENT_AMOUNT_FIELD, spentAmountIncrement),
                Updates.addEachToSet(PURCHASE_DAYS_FIELD, newPurchaseDays.toList())
            ),
            UpdateOptions().upsert(true)
        )
        return result.matchedCount
    }

    private fun collection(): MongoCollection<RewardDocument> {
        return mongoClient
            .getDatabase(DATABASE_NAME)
            .getCollection(REWARDS_COLLECTION, RewardDocument::class.java)
    }
}

Now let’s take a look at Spring version:

interface RewardMongoRepository :
    Repository<RewardDocument, ObjectId>,
    CustomRewardMongoRepository
{
    fun findByUserId(userId: String): RewardDocument?
}

interface CustomRewardMongoRepository {
    fun update(
       userId: String,
       spentAmountIncrement: BigDecimal,
       newPurchaseDays: Set<String>
    ): Long
}

@Component
class CustomRewardMongoRepositoryImpl(
    private val client: MongoTemplate
) : CustomRewardMongoRepository {
    override     fun update(
       userId: String,
       spentAmountIncrement: BigDecimal,
       newPurchaseDays: Set<String>
    ): Long {
        val result = client.upsert(
            Query.query(Criteria.where(USER_ID_FIELD).`is`(userId)),
            Update()
                .setOnInsert(USER_ID_FIELD, userId)
                .inc(SPENT_AMOUNT_FIELD, spentAmountIncrement)
                .addToSet(PURCHASE_DAYS_FIELD).each(newPurchaseDays),
            RewardDocument::class.java
        )
        return result.matchedCount
    }
}

Both tools provide a very similar API for using MongoDB update operators. Since we don’t care whether the document existed before or not, we also must mark the operation as upsert (insert or update). Note that future versions of Micronaut may reduce the amount of boilerplate. The new declarative repositories allow building custom updates in a single annotation. Once the new API matures, it may be an excellent choice.

MongoDB Update operators: when to use?

We can see that update operators offer an alternative way for dealing with consistency. The question arises: when to use them? I think that they are not suitable for CRUD operations. When we have a big business entity, most of the fields are modified by humans anyway. The best way for handling them are plain assignments and… optimistic locking.

On the other hand, we may have a business process where the users do not modify the data directly. Instead, we have distributed algorithms that need some persistent storage for the current state. They usually make specific, tailored modifications. This is the perfect use case for MongoDB update operators. Optimistic locking would be an overkill there. Similarly, multi-document transactions have performance penalty and restrict the horizontal scaling.

Summary

The key takeaway from this article is that MongoDB update operators help us making consistent updates without optimistic locking or transactions. To use them, we need to:

• create a separate entity for representing an update,
• implement a custom update operation in our repository,
• learn how upserts work in MongoDB.

Currently, we have to write custom updates by hand both in Micronaut and Spring. Micronaut hopefully will change that soon, once the new API for declarative repositories matures enough. Except that, both frameworks use a very similar approach. If we know one, we can quickly switch to the other.