API's, Microservices and JavaScript are the digital glue of the modern Internet.
This blog post shows how you can build a fast serverless API for a large data set. We'll use a 144MB file with 500K famous quotes to create an efficient NoSQL datastore using code hooks. We'll build and deploy a public serverless API to serve random quotes from the NoSQL database to clients.
As a demonstration of the quotes API we'll create a small React app that displays a slideshow with famous quotes and authors. The finished React demo app looks something like the screen shot shown below.
You can check out the live demo app here.
Download 500K quotes from kaggle.com
Kaggle is a fantastic source for learning and experimenting with data and insights.
Not surprisingly, they have a large dataset with 500K quotes from famous people.
Link to dataset: https://www.kaggle.com/datasets/manann/quotes-500k
The dataset is a comma separated (CSV) file format and contains three columns: the quote, the author of the quote and the category tags for that quote.
The snippet below shows the raw file content:
quote,author,category
"I'm selfish, impatient and a little insecure. I make mistakes, I am out of control and at times hard to handle. But if you can't handle me at my worst, then you sure as hell don't deserve me at my best.",Marilyn Monroe,"attributed-no-source, best, life, love, mistakes, out-of-control, truth, worst"
"You've gotta dance like there's nobody watching,Love like you'll never be hurt,Sing like there's nobody listening,And live like it's heaven on earth.",William W. Purkey,"dance, heaven, hurt, inspirational, life, love, sing"
You know you're in love when you can't fall asleep because reality is finally better than your dreams.,Dr. Seuss,"attributed-no-source, dreams, love, reality, sleep"
...
We'll download the complete dataset to a local 144MB file quotes.csv that we'll import to our NoSQL datastore later.
But before we can start to build our quotes application, we need a new project for the source code and stuff.
The server (less)
Create a new Codehooks project with a skeleton API
To host our serverless API and data we create a new Codehooks project. If you're new to Codehooks.io you can read more about getting started in the docs.
From the command line using the CLI we'll create a new project with the coho create command:
coho create quotes
cd quotes
In the index.js source file we add the following code for our first skeleton API.
/*
* Random quotes API. Codehooks (c) example.
*/
import app from 'codehooks-js'
// routehook function
function randomQuote(req, res) {
console.log("Quote coming up soon");
res.json({
"quote": "The true sign of intelligence is not knowledge but imagination.",
"author": "Albert Einstein"
});
}
// REST hook
app.get('/quote', randomQuote);
// bind to serverless runtime
export default app.init();
Before we can deploy our application we also need to add dependent NPM packages.
npm i codehooks-js -s
Next we can deploy the serverless API to the Codehooks cloud.
coho deploy
Project: quotes-q04p Space: dev
Deployed Codehook successfully! 🙌
To access the API from clients we need to create a read only API key.
coho add-token --readonly
Created new token 121a4850-549e-4be1-9b1e-61d8c6ef7f21 with access READ
curl -X GET \
'https://quotes-q04p.api.codehooks.io/dev/quote' \
--header 'x-apikey: 121a4850-549e-4be1-9b1e-61d8c6ef7f21'
{
"quote": "The true sign of intelligence is not knowledge but imagination.",
"author": "Albert Einstein"
}
Awesome, our skeleton API works, next we can continue to create the real quotes API, and finally our React app.
Import the dataset csv to a data collection in the NoSQL datastore
The CLI command coho import can import CSV and JSON files to create data collections in the NoSQL datastore. Notice that we've added the --rowcount parameter, this injects an incremental numeric field rowcount for each object imported in the datastore. This will be used later to create a search index for fast lookups and finally the random quotes query.
coho import ./quotes.csv quotes --rowcount
Finished import of 499709 objects
Please notice that you can experiment on the free plan with a smaller version of the dataset in the GitHub source here.
Check that we got all 500K quotes
From the import command we saw that the output reported 499709 objects.
We can also double check that the total number of objects in the datastore is correct using the coho count CLI command.
coho count quotes
{ count: 499709 }
Cool, the total count shows 499709, which is correct.
Using the coho query command we can also execute a test query to fetch a small slice of e.g. 2 records skipping 250K records into the dataset.
coho query quotes --limit 2 --offset 250000 --pretty
The output below shows that the csv file has been successfully imported and converted to a NoSQL data collection.
{
quote: 'I swear that while I live I will do what little I can to preserve and to augment the liberties of man, woman, and ',
author: 'Robert G. Ingersoll, The Liberty of Man, Woman and Child',
category: 'equality, freedom, honesty, justice, liberty, mercy, rights, virtue',
rowcount: 250041,
_id: '183cff13185-08fcnc76koz5ns'
}
{
quote: 'It was possible to love life, without loving your life.',
author: 'Christopher Coe',
category: 'gratitude, honesty, life',
rowcount: 250045,
_id: '183cff13185-0czojrvb5heddj'
}
Yay, the quotes data collection looks great, we're all good to continue 🙌
Add index for fast lookup
Performing operations on large datasets can slow down any application, we don't want that. In our case, we need to fetch a random object from the quotes dataset that has almost 500K objects.
Indexing is a powerful mechanism that creates fast lookups for a given key that exists in the indexed key space. As you will see in the real API code in the next section, the indexed rowcount field will be used to fetch our random quote - fast. Use the coho createindex command to create the index for the rowcount field.
coho createindex --collection quotes --index rowcount
Create the API for random quotes
The code for the final random quotes API is shown below. Notice the logic that uses a random number between 0-499709, i.e. total number of quotes in our dataset. By using the index field rowcount it's super fast to fetch an object directly from the database and return a random quote.
/*
* Random quotes API
* Codehooks (c) example.
*/
import app from 'codehooks-js'
// routehook function
async function randomQuote(req, res) {
// pick a random index number to fetch by the rowcount index field
const randomPos = Math.floor(Math.random() * 499709) + 1;
// open the NoSql Datastore
const conn = await Datastore.open();
// tell engine to use index field rowcount, and just start and end at the same pos
const options = {
useIndex: "rowcount",
startIndex: randomPos,
endIndex: randomPos
}
// run the query, returns a stream of 1 item ;)
conn.getMany('quotes', options)
.on('data', (data) => {
const { quote, author } = data;
res.json({ quote, author })
})
}
// REST hook
app.get('/quote', randomQuote);
// bind to serverless runtime
export default app.init();
Deploy the API again with coho deploy.
With the real API in place, we can move on to create a simple React app to integrate with our quotes API.
The client
Creating the quotes React app
We can use the brilliant create-react-app utility to kick-start the development of our quotes React app.
And we've deployed the app to GitHub pages by following this tutorial https://github.com/gitname/react-gh-pages.
Start with the initial create-react-app command and go with the flow, it could not be any easier.
npx create-react-app react-quotes
Check out the complete source code for the React quotes app at GitHub https://github.com/RestDB/codehooks-io-react-quotes.
The App.js source file contains the main display logic. It's really simple and does little more than to render the quote and the author from the state object. Also check out the App.css for the unimpressive app design 🤦🏽♂️
render() {
const { advice } = this.state;
return (
<div className="App">
<header className="App-header">
<img src={logo} className="App-logo" alt="logo" />
<h2>
React and Codehooks.io quotes
</h2>
<div className="card">
<h4 className="heading">{this.state.quote}</h4>
<p className="author">{this.state.author}</p>
</div>
<p><a className="App-link" href="#" onClick={this.nextQuote}>Next quote 👉</a></p>
</header>
</div>
);
}
The state object is populated from the API call using the built in fetch API.
fetchQuote = async () => {
try {
const resp = await fetch("https://quotes-q04p.api.codehooks.io/dev/quote", {
method: "GET",
headers: { "x-apikey": API_KEY }
})
const { quote, author } = await resp.json();
this.setState({ quote, author });
} catch (ex) {
console.error(ex);
}
}
You can test the app locally with the npm start command.
This starts a local server at http://localhost:3000 that loads the React quotes app in your local browser.
Source code: https://github.com/RestDB/codehooks-io-react-quotes.
Live demo here: https://restdb.github.io/codehooks-io-react-quotes/.
Summary
API's and microservice architecture plays a fundamental role in modern application development. In this blog post we've shown how the Codehooks serverless architecture can act as a complete backend and API for a React app.
Key takeaways:
- Kaggle.com has some very useful datasets
- Large datasets are easily imported to the NoSQL datastore using the CLI
- Fast development and deployment of your API
- Database indexes speeds up your API
- Deploying your React app to GitHub pages is really cool
Thank's for reading this, and I hope you've learned something new and useful.
"Einstein may have discovered the theory of relativity - Chuck Norris invented the reality of kicking ass."