A deep dive into Asynchronous Processing
Dealing with long time processing for a single transaction or waiting too long when downloading something from an application and causing an upstream timeout error due to heavy loads makes the server slow to respond to requests.
We decide to use asynchronous processing.
Our existing process downloads CSV files from the server, including parsing, extracting records from the database, generating it to CSV, and running on a single thread process. In that process, we encountered an upstream timeout error every time the records got bigger. Our first approach is still to do it all synchronously but optimize the parsing and extracting of records, but still, an upstream timeout error occurs when the records get bigger.
In that situation, we tried to find another solution to solve the problem, then found the asynchronous processing and started looking at it.
Flow diagram before the asynchronous process
https://drive.google.com/file/d/1vaYYiVPjjfNQcQU0-9ezTcJQ2Q6oA9EY/view?usp=sharing (image URL)
Dive into asynchronous programming
There are many different options for asynchronous programming, and this is one of the subsets of libraries out there.
Redis
AWS S3 bucket
To start, let’s set up our library to implement asynchronous processing:
Setting up Resque
Setup Redis
You’ll need Redis installed. Assuming you’re on MacOS and using Homebrew:
| brew redis install |
Follow the instructions in the notes to start Redis on boot, or start it manually with redis-server
Starting and stopping Redis in the foreground
To test your Redis installation, you can run the redis-server executable from the command line:
| redis server |
Setup Resque
Install Resque by adding gem ‘resque’ to the Gemfile and running bundle install
| gem ‘resque’ |
Setting up AWS S3
| gem ‘aws-sdk-s3’ |
S3 Configuration
To set it up, go to config/storage.yml First, uncomment the section headed by Amazon. Next, fill in your bucket’s name and region. For my application, the Amazon section of config/storage.yml looks like this:
For details on configuring region and credentials, see the developer guide.
| def client Aws::S3::Client.new( region: ‘us-east-2’, credentials: credentials, ) end |
| def upload client.put_object(bucket: ‘bucket_name’, key: ‘file-name’, body:’data’) end |
| def download client.get_object(bucket: ‘bucket_name’, key: ‘file-name’) end |
Now we can perform asynchronous programming by creating the controller and job for the export CSV controller.
Writing Jobs
Each job should be a separate Ruby class that extends ApplicationJob and defines perform method.
| class GenerateCSVJob < ApplicationJob queue_as :default def perform # define dummy text to seed the CSV records text =<<-EOS id,first name,last name,age 1,taro,tanaka,20 2,jiro,suzuki,18 3,ami,sato,19 4,yumi,adachi,21 EOS csv = CSV.generate(text, headers: true) do |csv| csv.add_row([“5”, “saburo”, “kondo”, “34”]) end client.put_object(bucket: ‘bucket_name’, key: ‘file-name’, body:data) end private def client Aws::S3::Client.new( region: ‘us-east-2’, credentials: credentials, ) end end |
Writing Controller
To run the execution of the job, initialize the instance class and call the method performed to the controller.
| class ExportCSVController < ApplicationController # This method will upload the csv file to s3 through background job def upload GenerateCSVJob.new.perform_now() end def download client.get_object(bucket: exports_bucket, key: @filename) end end |
To understand asynchronous programming, see the flow diagram from request to response.
https://drive.google.com/file/d/1CoTKxPHLJGMtrRvVXwcZuGcsTrhzjAr5/view?usp=sharing (image URL)
An application can handle heavy tasks without waiting for the result. When the result is ready, a separate call will request the server download or present it to the user and this is how asynchronous programming works in background processes.
Conclusions
If your application is designed for high/heavy load requests, it will require more than synchronous processes. There are several benefits associated with asynchronous programming, including:
- It provides an improved user experience. Asynchronous programming can improve the overall user experience since it helps systems run more efficiently. It helps reduce wait times, which often inconvenience the users.
- It helps improve an application’s performance. This type of programming can also help improve an application’s speed, the user interface of an application and make it easier for users to do their target.
- It’s possible to apply it to many programming languages. While asynchronous programming can make coding more complex, using the technique in various languages with different syntaxes is possible.
