Easy-peasy serverless workflow management. Part 1: Schedule updates to FaunaDB using Pipedream
Imagine you have a Car Listing website where users can post their adverts in the hope of selling their beloved though no longer-needed, car. Using Stripe as our billing system and FaunaDB as our database, here is how this flow could work (imagine some Lambda functions as well in between steps):
That’s all well and good — we’ve got an advert saved in Fauna which we can easily retrieve on our Front End web app. It will remain in public view for the next 30 days (which is what the user has paid for).
But what happens on day 30? How will we instruct Fauna that the advert should no longer be published? Now a Lambda function could still help us implement the un-publication logic in Fauna, but most Lambdas are state-less, i.e. they have no awareness of time and can’t reason with monthly schedules.
When building Lambda functions, you should assume that the environment exists only for a single invocation. It should not rely on any existing data structures or temporary files, or any internal state that would be managed by multiple invocations (such as counters or other calculated, aggregate values). From AWS
This is where a cron job shines and can help us schedule any interaction with our database. Cron is a software utility that can schedule repetitive tasks to take place at specific time intervals. It’s mostly used to schedule scripts; however, when you schedule a cron job within platforms like Pipedream, you can use it to schedule pretty much anything — from emails to HTTP calls.
On Pipedream’s secure serverless environment, we can interact with Fauna directly if we provide a Fauna key with the necessary permissions. That’s the kind of plug-and-go simplicity one would expect from interacting with a DBaaS such as Fauna in the first place. Let’s have a look at what the workflow we need to implement looks like.
Setting up a Pipedream workflow trigger
Pipedream’s workflow comprises a series of user-defined steps that can be executed in sequence to complete the flow. As a first step, we need a trigger. In our case, the trigger is our daily schedule, during which we should check if any adverts have expired and need to be unpublished. Step requires only a few clicks:
Setting up FaunaDB
Create an index
We want to read from a specific index called “all_listings_by_status_expiresOn”, which fetches only those adverts that have a status of “Published” and an “expiresOn” property that we can match against today’s date. Head to your database’s Indexes tab and create a new index:
- Name the index in a meaningful way. I opt for describing very explicitly what the index is by naming it in a way that includes: a) the collection on which the index will operate and b) a reference to the terms we want it to search for (think of these as arguments that we need to pass to a search query). To keep things simple, let’s call this one “all_listings_by_status_expiresOn”, where listings are the collection where our car listings are stored, and status and expression are the terms we want to search on.
- Choose a collection; in this case, the listings collection.
- Add two terms, status and expiresOn. a) Make sure these terms exist as keys in the data object in each of your listings and are named similarly. b) Note that the order in which these terms are created will affect the way we query our index (more on this later)
The importance of terms
At this point, you might be asking about the data type of our terms. For this example, we will be using a string type for both like so:
Note that we are using a yyyy-mm-dd format for our expiresOn string. It’s essential to do so for two reasons:
a) this follows part of the ISO8601 format, allowing us to standardise our dates. A complete timestamp in ISO8601 can include more info, like the local time, but this isn’t necessary for our example. We will assume that a listing is expired if the first time we check for it on a given date, the expiresOn property equals the current date, regardless of time.
b) yyyy-mm-dd is lexicographically sortable as a string, which means we can compare it for equality with any similarly formatted date string in Node.js. However, if you need to deal with date and time, you’ll want to use Fauna’s native Date type.
Create a Fauna key to authenticate a Fauna client
Head to Fauna dashboard at dashboard.fauna.com/keys to generate a key:
As a best practice, instead of creating a key with a Server role which will provide unrestricted access to the particular database, try to minimise the actions the key will be able to undertake. To do so, you must first create a new role at dashboard.fauna.com/roles. In my case, I only want this role to allow anyone using the key (in our case, Pipedream) to:
- Write to only one specific collection (remember we need to update any expired adverts)
- Read from a specific index called “all_listings_by_status_expiresOn” — This is an index we will create in the next step to fetch only those adverts that have a status of “Published” and an expiresOn property that matches today’s date.
Let’s call this key FAUNA_PIPEDREAM_READ_AND_UPDATE_EXPIRED_LISTINGS to make sure its scope is clear and give it the following privillages:
Connect to FaunaDB in a Pipedream workflow
This is where the Pipedream simplicity kicks in. As the next step, search for a FaunaDB action. You’ll likely see two options: select “Use any Fauna API in Node.js”. This is because we will not only be reading documents from our db but also include logic to update any expired adverts.
More on the “check and update logic” in a bit; first, we need to create and authenticate a Fauna Client (this is where the pleasant feeling of realising you won’t have to make another HTTP call to an intermediate API kicks in).
As soon as you add this action to your workflow, you’ll see it’s made of two parts: configure and code. To authenticate, you’ll first be asked to provide a key.
Configure
Add the FAUNA_PIPEDREAM_READ_AND_UPDATE_EXPIRED_LISTINGS secret of the key we created earlier. You can provide the same nickname in Pipedream as the Fauna key’s name for simplicity and to ensure everybody in your team knows which key and what privileges this gives Pipedream.
Implement the flow
Code
Step 1: Instantiate FaunaDB client and the current date
Firstly, we need to instantiate the FaunaDB client and pass it the secret for our key. When we previously added the secret in the Configure step, Pipedream saved that as an environment variable for us. We can reference anywhere in our code using this.faunadb.$auth.secret, which means we must pass this variable as an argument to faunadb.Client without needing to expose the actual secret in our code:
async run({steps, $}) {
const faunadb = require('faunadb')
const q = faunadb.query
// instantiate the client and pass the env variable for our FaunaDB key's secret
const client = new faunadb.Client({ secret: this.faunadb.$auth.secret })
}
})Then we need to instantiate today’s date to compare it with the expiresOn property of the listings we will retrieve later. To do so, we must ensure our date is created in the ISO format using toISOString. Since ISO timestamps are created with both date and time, we need to split the string to only include the first part, which is the date in yyyy-mm-dd format (just like the format of the expiresOn property in our listings):
async run({steps, $}) {
const faunadb = require('faunadb')
const q = faunadb.query
const client = new faunadb.Client({ secret: this.faunadb.$auth.secret })
// instantiate a "today" date to check against each listing's expiry date
const today = new Date().toISOString().split('T')[0];
}
})Add a console log to check ‘today’ is created in the right format and test your workflow before moving on.
Step 2:
We can now retrieve listings from FaunaDB using the query function of the client. To achieve this we will pass the Map FQL function as an argument. Map itself, takes 2 arguments. A set of results and a lambda function to operate on them.
async run({steps, $}) {
const faunadb = require('faunadb')
const q = faunadb.query
const client = new faunadb.Client({ secret: this.faunadb.$auth.secret })
// 1. instantiate a "today" date to check against each listing's expiry date
const today = new Date().toISOString().split('T')[0];
// 2. get all listings that are currently published but expire today
let res = await client.query(
q.Map(
q.Paginate(
q.Match(
q.Index('all_listings_by_status_expiresOn'),
['Published',today]
)
),
q.Lambda('ref', q.Get(q.Var('ref')))
)
)
console.log("res is ", res)
}
})Let’s break it down. Starting from our second argument Our lambda function, will for now simply extract the reference (i.e. the unique id in FaunaDB) from each result . The first argument is a series of nested FQL functions that do the following:
- Paginate, simply allows us to manage large result sets and comes first.
- We then pass Match (yet another FQL function) to Paginate. Match allows us to retrieve the actual results by using two more arguments: a) the index we want to retrieve from (in this case ‘all_listings_by_status_expiresOn’) and b) the terms we want to match for. In this case this is the array of values [ ‘Published’, today ] where ‘Published’ is the value of the status property we are looking for in each document and today, is today’s date we instantiated earlier.
In other words we want to retrieve all Published car advertisements that expire today. As we mentioned earlier, the order of these two values in the array matters. The order should be the same as the order of terms when we originally created the index, i.e. status should come first and expiresOn second.
Click on Test to try the workflow out and check the Logs tab in the Results card in Pipedream. You should be able to get a result back by this stage:
Step 3:
Now that we can match and retrieve all car advertisements that expire today, let’s update their status so that they are no longer “Published” but become “Expired” instead:
async run({steps, $}) {
const faunadb = require('faunadb')
const q = faunadb.query
const client = new faunadb.Client({ secret: this.faunadb.$auth.secret })
// 1. instantiate a "today" date to check against each listing's expiry date
const today = new Date().toISOString().split('T')[0];
// 2. get all listings that are currently published but expire today
let res = await client.query(
q.Map(
// Paginate through the results of the index match
q.Paginate(
q.Match(
// Use your index that's sorted by expiredAt and status
q.Index('all_listings_by_status_expiresOn'),
// Match all documents where expiredAt == today and status == 'published'
['Published',today ]
)
),
//3. For each ref returned by the index match, update the status to 'expired'
q.Lambda('ref', q.Update(q.Var('ref'), { data: { status: 'Expired' } }))
)
)
console.log("res is ", res)
}
})The second argument to Map — the lambda function — was only retrieving the reference of each result by the previous step. We are now expanding this function passing the Update FQL function which will update each matched listing with a new “expired” status”.
Need to use another Serverless Database instead of FaunaDB? Check out this article where we explore serverless workflow management with Supabase instead.
