Lecture5 - Develop Server-side Apps of MongoDB (quiz).pdf

Review – Write Node.js APPs (server-side apps) COMPS381F 3 Call back functions in Node.js ❑ A Node.js file includes a collection of callback functions, ❑ so making it an asynchronous event-driven program, ❑ able to prevent I/O blocking, ❑ and handle a large number of requests. 4 Call back fun...

Review – Write Node.js APPs (server-side apps) COMPS381F 3 Call back functions in Node.js ❑ A Node.js file includes a collection of callback functions, ❑ so making it an asynchronous event-driven program, ❑ able to prevent I/O blocking, ❑ and handle a large number of requests. 4 Call back functions in Node.js ❑ A Callback function is passed as an argument into another function, which can then be invoked (called back) inside the outer function. ❑ Common parameters in a callback function • • err: failure status of the last operation, results: additional results or information from the last operation (such as a file handle, database connection, rows from a query, and so on) 5 Call back functions in Node.js Node expects - almost - all callback functions to accept an Error object as the first argument. • If no error occurred, the first argument should be null. • If you use inline anonymous functions, this is a typical code snippet that you will encounter using node: 1. function asyncOperation ( a, b, c, callback ) { of hard work ... 2. 3. // ... lots if ( /* an error occurs */ ) { return callback(new Error("An error has occurred")); 4. } // ... more work ... 5. callback(null, d, e, f); 6. } 7. asyncOperation ( params.., function ( err, returnValues.. ) { //This code gets run after the async operation gets run }); 6 Outer function is defined Outer function is invoked, callback is (results) => {} 7 Callback function is passed as an argument into the Outer function 8 Run the outer function If no error 9 Run the outer function If error hits 10 There may be different program structures when using call-functions in Node.js, depending on your app requirements. Ref: https://gist.github.com/sunnycmf/b2ad4f80a3b627f04ff2 The left-hand side is a typical Node.js program with call-back functions that use • Waterfall pattern It lets you waterfall through function chains and leaves handling the error it up to the original invoking function. It is likely that you do not want to throw uncaught errors all the time. 11 Outer function is defined Outer function is invoked, filename is `/some/file`, callback is function (err, result) {…} 12 Callback function is passed as an argument into the Outer function 13 Run the outer function If error hits 14 Run the outer function If no error 15 Use Node.js to write app programs A typical Node.js app should prov ide the following serv ices (Applicable to any server apps) : ❑ Listen on a particular TCP port • Await client requests ❑ Determine client’s request (what do clients want?) Identify different requests, for example: localhost:8099/greetings Print “Hello There!”” localhost:8099/greetings/sayHello Print “Hello There!” localhost:8099/greetings/sayHelloWithTime Print “Hello There! It is now XXXX” Parse request parameters (if any) Others: Returns 404 ❑ Process the request ❑ Respond to the client • Return data in plain text, HTML or JSON format 16 An app program sample Listen on a particular TCP port 17 An app program sample Process the (GET) request 18 An app program sample Determine client’s request 19 An app program sample Respond to the client 20 Two Request Methods ❑ Two most common HTTP methods for a requestresponse between a client and a server are: • GET - Request data from a specified resource The query string (name/value pairs) is sent in the URL of a GET request: https://www.w3schools.com/action_page.php?fname=Yalin&lname=Liu • POST - Send data to a server to create/update a resource The data sent to the server with POST is stored in the request body of the HTTP request: POST /test/demo_form.php HTTP/1.1 Host: w3schools.com name1=value1&name2=value2 21 Node.js handles GET Request A standard HTTP URL 22 Node.js handles GET Request A standard HTTP URL 23 Node.js handles POST Request 24 Node.js handles POST Request 25 Prepare basic Node.js app files ❑ A folder (directory) for storing code ❑ A package.json file that contains App name, description, version, author details, App dependencies, startup script and etc. ❑ Run your Node.js APP (appname.js) ❑ “node appname ” ❑ "npm install" and "npm start" to run "node appname" 26 Develop Node.js apps How many steps to develop an app? 1. Create an app folder 2. Prepare app files - Write a Node.js program: server.js - Prepare package.json (node dependencies) 3. Run the app, use `npm install` and `npm start` 4. Test the app by one of the following methods - Web browser – send HTTP requests - New terminal – curl GET requests - Write a HTML file to send HTTP requests 27 Review – Node.js Driver of MongoDB COMPS381F 28 MongoDB Node.js Driver ❑ The official MongoDB Node.js driver • • https://docs.mongodb.com/ecosystem/drivers/node/ Optimized for simplicity (and speed) The driver allows Node.js to write a program to drive MongoDB! 29 Install MongoDB Node.js Driver ◾ Do one of the followings: 1. Use npm to install the MongoDB Node.js driver. npm install mongodb 2. Use package.json { " name": " server", " description": "A very simple Node.JS MongoDB app", " author": "Raymond SO <[email protected]>", " version": "0.0.1", " dependencies": { " mongodb": ”3.1" }, " engine": " node >= 0.10.0", "scripts": {"start": " node server"} } Remember to run npm install to download and install the dependencies. Reference: http://docs.mongodb.org/getting -started/node/client/ 30 Write Node.js to connect to MongoDB const const const const MongoClient = require('mongodb').MongoClient; assert = require('assert'); url = ''; dbName = ‘’; Callback runs when connection is established const client = new MongoClient(url); client.connect((err) => { assert.equal(null,err); console.log("Connected successfully to server"); const db = client.db(dbName); /* * CRUD Operations */ client.close(); }); Reference: http://docs.mongodb.org/getting -started/node/client/ 31 Insert Data - Algorithm 1. Connect to MongoDB server 2. When the connection to MongoDB server is established, ( via callback ) ▪ ▪ Check connection error(s) Insert document(s) 3. When the insertion is completed, ( via callback ) ▪ ▪ Check insertion error(s) Close MongoDB server connection Reference: http://docs.mongodb.org/getting -started/node/client/ 32 Insert Data 2 1 const insertDocument = (db, callback) => { db.collection('books').insertOne( { "name" : "Introduction to Node.js", "author" : "John Dole", "price" : 75.00, “stock“ ：0 }, (err, result) => { 3 assert.equal(err, null); console.log("Inserted one document into the books collection."); callback(result); }); }; const client = new MongoClient(url); client.connect((err) => { assert.equal(null,err); console.log("Connected successfully to server"); const db = client.db(dbName); insertDocument(db, () => { client.close(); 4 }); }); 33 Query For All Documents in a Collection (1) const findRestaurants = (db, callback) => { let cursor = db.collection('restaurant').find() cursor.forEach((doc) => { console.log(JSON.stringify(doc)); }); callback(); }; const client = new MongoClient(url); client.connect((err) => { assert.equal(null,err); console.log("Connected successfully to server"); const db = client.db(dbName); }) findRestaurants(db,() => { client.close(); }) To return all documents in a collection, call the find method without a criteria object. 34 Query For All Documents in a Collection (2) const findRestaurants = (db, callback) => { let cursor = db.collection('restaurant').find().limit(10) cursor.toArray((err,docs) => { assert.equal(err,null); callback(docs); }); }; const client = new MongoClient(url); client.connect((err) => { assert.equal(null,err); console.log("Connected successfully to server"); const db = client.db(dbName); findRestaurants(db,(restaurants) => { client.close(); console.log("Disconnected MongoDB server"); console.log(restaurants); }) }) 35 Check Update Result const updateRestaurants = (db, callback) => { db.collection('restaurant') .updateOne( { "restaurant_id" : "41156888" }, { $set: { "address.street": "East 31st Street" } }, (err, results) => { assert.equal(err,null); console.log(results); if (results.result.nModified == 1) { console.log('Update Succeed!'); } else { console.log('Update failed!!'); } callback(); }); }; 36 Use ObjectID in Query How to use _id in read operations var ObjectID = require('mongodb').ObjectID; … var criteria = {}; criteria['_id'] = ObjectID("59e6dff691371fbe8690155b"); … var cursor = db.collection("restaurants").find(criteria); 37 MongoDB Object ID (_id) ❑ MongoDB adds a unique name-value pair "_id": {"$oid": "xxxxx"} to every new document ❑ _id can be used as the primary key 38 Update Data With Node.js Criteria const updateRestaurants = (db, callback) => Matching { db.collection('restaurant') one document .updateOne( { "restaurant_id" : "41156888" }, { $set: { "address.street": "East 31st Street" } }, (err, results) => { assert.equal(err,null); console.log(results); callback(); }); }; 39 Knowledge Integration – Develop Node.js (server) Apps of MongoDB COMPS381F 40 Write Node.js Apps to Handle MongoDB Node.js based MongoDB Server (Apps) ❑ Should be able to handle CRUD operations of MongoDB ❑ Should provide the request-response services (Applicable to any server apps): ❑ ❑ ❑ ❑ Listen on a particular TCP port Determine client’s request (what do clients want?) Process the request Respond to the client 41 MongoDB App – Example 1 Write a Node.js app to allow user send GET requests for finding/remove MongoDB documents const http = require('http'); const url = require('url’); const server = http.createServer((req,res) => { var parsedURL = url.parse(req.url, true); switch(parsedURL.pathname) { case '/find': handle_Find(res, parsedURL.query); break; case '/delete': handle_Delete(res, parsedURL.query); break; default: res.writeHead(404, {'Content-Type': 'text/plain'}); res.end(`${parsedURL.pathname} - Unknown request!`); } }) server.listen(process.env.PORT || 8099); 42 MongoDB App – Example 1 Write a Node.js app to allow user send GET requests for finding/remove MongoDB documents const MongoClient = require('mongodb').MongoClient; const assert = require('assert’); const mongourl = ''; const dbName = 'test'; const findDocument = (db, criteria, callback) => { let cursor = db.collection('bookings').find(criteria); cursor.toArray((err,docs) => { assert.equal(err,null); callback(docs); }); } 43 MongoDB App – Example 1 Write a Node.js app to allow user send GET requests for finding/remove MongoDB documents const handle_Find = (res, criteria) => { const client = new MongoClient(mongourl); client.connect((err) => { assert.equal(null, err); console.log("Connected successfully to server"); const db = client.db(dbName); findDocument(db, criteria, (docs) => { client.close(); console.log("Closed DB connection"); res.writeHead(200, {"content-type":"text/html"}); res.write('<html><body><ul>'); for (var doc of docs) { //console.log(doc); res.write(`<li>Booking ID: ${doc.bookingid}, Mobile: ${doc.mobile}`); } res.end('</ul></body></html>'); }); }); } 44 MongoDB App – Example 2 Write a Node.js app to display HTML forms and allow user send GET requests for finding/remove MongoDB documents. const http = require('http'); const url = require('url’); const server = http.createServer(function (req,res) { var today = new Date(); console.log(today.toTimeString() + " " + "INCOMING REQUEST: " + req.connection.remoteAddress + " " + req.method + " " + req.url); var parsedURL = url.parse(req.url,true); //true to get query as object if (parsedURL.pathname == '/search') { const client = new MongoClient(mongourl); client.connect((err) => { assert.equal(null, err); console.log("Connected successfully to server"); const db = client.db(dbName); /*CRUD operations*/ res.writeHead(200, {"Content-Type": "application/json"}); res.write(JSON.stringify(restaurants)); res.end(); console.log(today.toTimeString() + " " + "CLOSED CONNECTION " + req.connection.remoteAddress);});}); } else { res.writeHead(404, {"Content-Type": "text/plain"}); res.write("404 Not Found\n"); res.end(); }}); server.listen(process.env.PORT || 8099); 45 Mongoose – ODM of Node.js COMPS381F 46 Node.js & Mongoose ❑ Mongoose • An ODM for building MongoDB object modeling in Node.js • Builts using the official MongoDB Node.js driver • Translates data in MongoDB to JavaScript objects in your application • Provides a scheme-based solution to modeling your application data and includes built-in type casting, validation, query building, business logic hooks and more high-level data modeling functions ❑ ODM • stands for Object Document Mapper • a tool that maps the structure of the databases in objects (JSON objects in the case of Node.js) Mongoose official platform: http://mongoosejs.com/index.html Mongoose official documents: https://mongoosejs.com/docs/guide.html 47 Mongoose ❑ Mongoose is open source, licensed under MIT ❑ Install Mongoose by npm command line: npm install mongoose ❑ Or include in the dependencies section of package.json { "name": "server", "description": "ODM App", "author": "Raymond SO <[email protected]>", "version": "0.0.1", "dependencies": { "mongoose": "^4.10.8" }, "engine": "node >= 0.10.0", "scripts": { "start": "node server" } } 48 Basic Concepts of Mongoose ❑ Schemas ▪ Describe the data construct of a document ▪ Define the name and type of data (in documents) ❑ Models ▪ A compiled version of the schema ▪ One instance of the model will map to one document in the database Schema Model 49 Connecting to MongoDB ❑ Connect() take a mongodb:// URI, or the parameters host, database, port, options. const mongoose = require('mongoose’); mongoose.connect('mongodb://’, {useMongoClient: true,} ); 50 Connecting to MongoDB Replace <username>and <password> 51 GET Notified ❑ We have a pending connection to the test database ❑ We now need to get notified if we connect successfully or if a connection error occurs ❑ Once our connection opens, our callback will be called. let db = mongoose.connection; db.on('error’, console.error.bind(console,'connection error:’) ); db.once('open', (callback) => { // This function will be executed once when the db connection // is opened. // CRUD here!!! // }); 52 Schema ❑ With Mongoose, everything is derived from a SCHEMA ❑ We've got a schema with one property, name, which will be a String. var kittySchema = mongoose.Schema({ name: String, age: Number }); ❑ The permitted Schema Types are: ▪ ▪ ▪ ▪ ▪ ▪ ▪ ▪ String Number Date Buffer (for storing binary information, for example images stored in Mongo DB) Boolean Mixed (any type of data) ObjectId Array 53 Re fe re n c e : h ttp s: //m o n g o o se j s.c om /do c s/sc h e m atype s.h tm l Schema – eg 1 var mongoose = require('mongoose'); var Schema = mongoose.Schema; var blogSchema = new Schema({ title: String, author: String, body: String, comments: [{ body: String, date: Date }], date: Date, hidden: Boolean, meta: { votes: Number, favs: Number 0..* } Blog Comment }); 54 Schema – eg 2 Publishers Books ID:String Name:String Address:String ISBN:String Title:String Author:String Price:Number Stock:Number Available:Boolean 0..* var mongoose = require('mongoose'); var publisherSchema = mongoose.Schema({ id: mongoose.Schema.ObjectId, name: String, address: String, books: [{isbn: String, title: String, author: String, price: Number, stock: Number, available: Boolean}] }); module.exports = publisherSchema; 55 Create Collection ❑ A model is a class with which we construct documents. ❑ In this case, each document will be a kitten with properties and behaviors as declared in our schema. ❑ One instance of the model will map to one document in the database var Kitten = mongoose.model('Kitten', kittySchema); Create a MongoDB collection called name ‘Kittens’ 56 Create Document const mongoose = require('mongoose'); mongoose.connect('mongodb://’, {useMongoClient: true,}); Step 1: Connect to MongoDB Step 2: Create an instance Step 3: Save the instance Step 4: Close MongoDB connection const kittySchema = mongoose.Schema({ name: String, age: Number}); const db = mongoose.connection; db.on('error', console.error.bind(console, 'connection error:')); db.once('open', (callback) => { let Kitten = mongoose.model('Kitten', kittySchema); let fluffy = new Kitten({name: 'fluffy', age: 0}); fluffy.save((err) => { if (err) throw err console.log('Kitten created!’) db.close(); }); }); 57 Create Document – Externalized the Model var mongoose = require('mongoose'); mongoose.connect('mongodb://localhost/test'); var kittySchema = require('./models/kitty'); var db = mongoose.connection; db.on('error', console.error.bind(console, 'connection error:')); db.once('open', function (callback) { var Kitten = mongoose.model('Kitten', kittySchema); var fluffy = new Kitten({name: 'fluffy', age: 0}); fluffy.save(function(err) { if (err) throw err console.log('Kitten created!') db.close(); }); }); var mongoose = require('mongoose'); var kittySchema = mongoose.Schema({ name: String, age: Number }); module.exports = kittySchema; 58 Find Documents db.once('open', function (callback) { var Kitten = mongoose.model('Kitten', kittySchema); Kitten.find({name: /^flu/}, results function(err,results) { if (err) return console.error(err); console.log(results); db.close(); }); }); 59 Update db.once('open', function (callback) { var Kitten = mongoose.model('Kitten', kittySchema); Kitten.findOne({name: "fluffy"}, function(err,result) { if (err) return console.error(err); console.log(result); Change name result.name = "lion"; result.save(function(err) { if (err) throw err console.log("Name changed"); db.close(); }); }); }); 60 Validation ❑ How can we ensure 0 >= age >= 30 in all documents? ❑ How can we ensure name must be provided? ❑ How can we ensure the length of name must be > 0 and less than 20? var kittySchema = mongoose.Schema({ name: String, age: Number }); var fluffy = new Kitten({age: -1}); 61 Built-in Validators ❑ Mongoose has several built-in validators . ▪ All SchemaTypes have the built-in required validator. ▪ Numbers have min and max validators. ▪ Strings have enum, match, maxlength and minlength validators. Ref: https://mongoosejs.com/docs/validation.html#built-in-validators 62 Required Validator var kittySchema = mongoose.Schema({ name: {type: String, required: true} age: Number }); 63 Min, Max Validators var kittySchema = mongoose.Schema({ name: {type: String, required: true} age: {type: Number, min: 0, max: 20} }); 64 Minlength, Maxlength Validators var kittySchema = mongoose.Schema({ name: {type: String, required: true, minlength: 1, maxlength: 50 } age: {type: Number, min: 0, max: 20} }); 65 Enum Validator name can only be assigned one of the values in the array var kittySchema = mongoose.Schema({ name: {type: String, enum: ['fluffy','john','mary','peter'] } age: {type: Number, min: 0, max: 20} }); 66 Match Validator var kittySchema = mongoose.Schema({ name must begin name: {type: String, with an upper-case match: /^[A-Z]+/ alphabet } age: {type: Number, min: 0, max: 20} }); Regex (regular expression) 101: https://regex101.com Regex 101: Everything you need to know: https://blog.mergify.com/regex101-everything-you-need-to-know/ 67 Default var mongoose = require('mongoose'); var blogSchema = mongoose.Schema({ title: String, author: String, body: String, comments: [{ body: String, date: Date }], date: { type: Date, default: Date.now }, hidden: Boolean, meta: { votes: Number, favs: Number } }); 68 Validation Example db.once('open', (callback) => { let Kitten = mongoose.model('Kitten', kittySchema); let fluffy = new Kitten({name: 'fluffy', age:90}); fluffy.validate((err) => {console.log(err);}); fluffy.save((err) => { if (err) throw err console.log('Kitten created!’) db.close();}); }); var mongoose = require('mongoose'); var kittySchema = mongoose.Schema({ name: String, age: {type: Number, min: 0, max: 20} }); module.exports = kittySchema; 69 MongoDB & Binary Data COMPS381F 70 Storying Binary Data in MongoDB ◾ Base64 is used commonly in a number of applications including email via MIME and storing complex data in XML. ◾ The Base64 term originates from a specific MIME content transfer encoding. ◾ Base64 is a generic term for a number of similar encoding schemes that ◾ encode binary data by ◾ treating it numerically and ◾ translating it into a base 64 representation . ◾ Base64 encoding schemes are commonly used when there is a need to ◾ encode binary data that needs to be stored and transferred ◾ over media that are designed to deal with textual data . ◾ This is to ensure that the data remains intact without modification during transport. BASE64 Decode and Encode: https://www.base64decode.org 71 Base64 table from RFC 4648 • Use 64 digits to encode information • Each base64 digit needs exactly 6 bits Source: https://en.wikipedia.org/wiki/Base64 72 Encode Base64 in Node.js fs.readFile(filename, function(err,data) { data contains MongoClient.connect(mongourl,function(err,db) { var new_r = {}; some JPEG data new_r['title'] = title; new_r['mimetype'] = mimetype; new_r['image'] = new Buffer(data).toString('base64'); insertPhoto(db,new_r,function(result) { db.close(); res.writeHead(200, {"Content-Type": "text/plain"}); res.end('Photo was inserted into MongoDB!'); }) }); }) function insertPhoto(db,r,callback) { db.collection('restaurants').insertOne(r,function(err,result) { assert.equal(err,null); console.log("insert was successful!"); callback(result); }); } 73 Decode Base64 in Node.js db.collection('photos').findOne({"key": target}, function(err,doc) { assert.equal(err,null); if (doc != null) { bfile = new Buffer(doc.data,'base64'); mimetype = doc.mimetype; } callback(bfile,mimetype); doc.data contains }); some JPEG data encoded using Base64 74 Online Photo Albums Upload photos, album details & etc. Clients Node.js Server (HTML5, iOS, Android) Filesystem DB Download photos, album details & etc. Storage Options: 1. Album details in DB, photos in filesystem 2. Album details and photos in DB What're the pros and cons of each option? 75 Limitations of MongoDB ◾Recall that MongoDB uses BSON format to store documents ◾The maximum size of BSON document is 16MB ◾ Given this limitation, the largest (binary) data that can be stored in MongoDB is 16MB ◾ GridFS should be used for storing and retrieving objects that exceed the BSON-document size limit ◾Reference • MongoDB Manual GridFS: https://docs.mongodb.com/manual/core/gridfs/ • Tutorial of GridFS: https://mongodb.github.io/node-mongodb-native/2.2/tutorials/gridfs/ 76 GridFS ◾ GridFS is a specification for storing and retrieving files that exceed the BSON-document size limit of 16 megabytes. ◾ Instead of storing a file in a single document, GridFS divides a file into parts, or chunks, and stores each of those chunks as a separate document. By default, GridFS limits chunk size to 255 kilobytes. ◾ GridFS uses two collections to store files: ◾ the chunks collection which stores the file chunks, and ◾ the files collection which stores the file metadata. ◾ When you query a GridFS store for a file, the driver or client will reassemble the chunks as needed. ◾ GridFS is useful not only for storing files that exceed 16 megabytes but also for storing any files that you want to access without having to load the entire file into memory. 77 GridFS File GridFS Chunk Chunk Chunk Chunk Chunk Chunk 78