This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

J

Job Scheduling

We often have repetitive tasks to perform such as checking internet connection every few hours, maybe checking stock prices every few hours, or checking if that one product has gone on sale on an e-commerce site. Instead of having to do this manually, you can schedule these tasks to run at regular intervals.

For example, consider this check_internet.sh Bash script to check if you are connected to the internet or not by pinging Google’s webpage:

echo -e "GET http://google.com HTTP/1.0\n\n" | nc google.com 80 > /dev/null 2>&1

if [ $? -eq 0 ]; then
    echo "Online"
else
    echo "Offline"
fi

Now, to run this script say, every 2 hours to check for internet connection, we use the crontab utility in Linux.

# enters the configuration file that contains cron jobs to be scheduled
crontab -e
# run this script every 2 hours
0 */2 * * *  check_internet.sh

Let’s not get ahead of ourselves and first see what is a cron job and what does this terrible syntax 0 */2 * * * mean.

What is a cron job?

Derived from the Greek God of Time, Chronos, a cron job is a task that is scheduled by a job scheduler such as crontab on Linux-based operating systems and by cron on a Unix-like operating system.

To learn more about the uncommon asterisk syntax, refer crontab.guru

Though checking the internet connection is a trivial example, we have seen in health checks how important it is to check if a server or an application is healthy (alive) or not.

Why do you need cloud for scheduling cron jobs?

From health checks to thousands of repetitive tasks such as fetching data from several APIs, to scheduling a job based on an event trigger, manually managing so many jobs becomes impossible. Hence, to manage jobs at scale, cloud is at our behest.

Job scheduling tools such as Google Cloud’s Cloud Scheduler enable automation of execution of tasks based on date-time scheduling or other methods of execution such as event-based triggers. It eliminates the need for manual kick-offs, reducing delays and avoiding repetitive tasks.

1 - Cloud Scheduler

Schedule virtually everything

Cloud Scheduler is a fully managed enterprise-grade cron job scheduler. It allows you to schedule virtually any job, including batch, big data jobs such as ETL, and cloud infrastructure operations. You can automate everything, including retries in case of failure to reduce manual toil and intervention.

Cloud Scheduler logo

Leveraging Cloud Scheduler

Cloud Scheduler can be used in the following ways:

  • Reduce minimal manual effort by scheduling repetitive Big Data tasks such as fetching and preprocessing of data for a data pipeline

  • Downscale or upscaling cloud infrastructure when needed in a reliable manner

  • Automate health checks, trigger a Cloud Pub/Sub pipeline, and a lot more with integration with our GCP services

Learn