Lecture4 - Develop Node.js Apps.pdf

COMPS381F (2023 Autumn) Unit04 – Develop Node.js Apps Alin Liu 1 Outline ❑ Script Structure of Node.js 1. Asynchronous Programming ❑ Develop Node.js Apps 2. A Typical Node.js APP 3. HTTP Methods: GET VS. POST 4. Use Node.js to handle HTTP services 5. TCP PORTS* 2 Script Structure of Node.js: A...

COMPS381F (2023 Autumn) Unit04 – Develop Node.js Apps Alin Liu 1 Outline ❑ Script Structure of Node.js 1. Asynchronous Programming ❑ Develop Node.js Apps 2. A Typical Node.js APP 3. HTTP Methods: GET VS. POST 4. Use Node.js to handle HTTP services 5. TCP PORTS* 2 Script Structure of Node.js: Asynchronous JS ON Programming COMPS381F 3 References ❑ Event Loops in NodeJS – Beginner’s Guide to Synchronous and Asynchronous Code: https://www.freecodecamp.org/news/nodejs-eventloop-tutorial/ ❑ Node.js: What it is, when and how to use it, ,and why you should: https://www.freecodecamp.org/news/node-js-what-when-where-why-how-ab8424886e2/ 4 Asynchronous Programming in Node.js ❑ NodeJS is an asynchronous event-driven JavaScript runtime environment designed to build scalable network applications. ❑ Asynchronous programming makes Node.js able to handle a large amount of requests. Why and How? 5 Short for I/O (input/output) ❑ I/O operations refer primarily to the program’s interactions with the system’s disk and network. For examples: • reading/writing data from/to a disk, • making HTTP requests, • and talking to databases. ❑ I/O operations are very slow compared to accessing memory (RAM) or doing work on the CPU. Services requests to a server are actually I/O operations! 6 The OLD way of doing things ❑ Synchronous programming (a way to execute codes) • • The code runs in the sequence. A function (in a code line) is called and has returned some value, only then the next line will be executed. ➢ The program has to wait for the line/function to finish its work before going on to the next line/function. ➢ I/O recourses are “blocked” because of doing nothing when waiting for the interacting/query responses 7 Blocking I/O Issues ❑ Traditional way to handle a large amount I/O requests via Synchronous programming Issues: ➢ Overhead when creating new stacks and execution environment ➢ Time consuming for completion of I/O ➢ Relatively more difficult to scale up 8 Asynchronous Programming ❑ Asynchronous Programming • It refers code that doesn’t execute in sequence. • Functions inside a program are performed not according to the sequence but only when certain conditions are met. How? Call back functions! 9 The Callback Function ❑ A Callback function is passed as an argument into another function, which can then be invoked (called back) inside the outer function . ❑ Common parameters in a callback function • • err: failure status of the last operation, results: additional results or information from the last operation (such as a file handle, database connection, rows from a query, and so on) 10 Example: Error Handling 11 Example: Reading A File Synchronous Blocked Blocked Asynchronous Call this function when ‘open’ is completed Call this function when ‘read’ is completed 12 Reading A File: Complete Example Callback err – status of fs.open() handle – file handle of the opened file err – status of fs.read() length – number of characters read Callback 13 Asynchronous Programming in Node.js ❑ Node.js achieves asynchronous programming by ➢ Call-back Function ➢ Callback function are called at the completion of a given task; this prevents any blocking, and allows other code to be run in the meantime. 14 Typical convention of callback functions in Node.js ❑ Error call-back convention: • Error is the first parameter, call-back is the last parameter 1. The function hits an error, call the callback with the first parameter being an Error object. 2. The function cleanly exits, call the callback with the first parameter being null and the rest being the return value(s). 1. function asyncOperation ( a, b, c, callback ) { 2. // ... lots of hard work ... 3. if ( /* an error occurs */ ) { 4. return callback(new Error("An error has occurred")); 5. } // ... more work ... 6. callback(null, d, e, f); 7. } 8. asyncOperation ( params.., function ( err, returnValues.. ) { //This code gets run after the async operation gets run }); More examples: https://gist.github.com/sunnycmf/b2ad4f80a3b627f04ff2 15 Callbacks in Node.js ❑ Call-back are the foundation of Node.js ➢ Callbacks give you an interface with which to say, "and when you're done doing that, do all this." ➢ This allows you to have as many IO operations as your OS can handle happening at the same time. ➢ In a web server with hundreds or thousands of pending requests with multiple blocking queries, performing the blocking queries asynchronously gives you the ability to be able to continue working and not just sit still and wait until the blocking operations come back. 16 In-class Exercises Ex 1: What is Node.js primarily used for? A. B. C. D. Building static websites Creating desktop applications Developing server-side and networking applications Designing mobile applications 17 In-class Exercises Ex 2: Which of the following is NOT a feature of Node.js? A. B. C. D. Single-threaded Non-blocking I/O model Synchronous programming Uses JavaScript 19 In-class Exercises Ex 3: What is the main advantage of asynchronous programming in Node.js? A. It allows multiple operations to be handled simultaneously B. It simplifies the coding process C. It reduces the memory usage of the application D. It increases the speed of single operations 21 In-class Exercises Ex 4: What is the purpose of the callback function in Node.js asynchronous programming? A. It is used to pause the execution of the program B. It is used to handle the result of an asynchronous operation C. It is used to repeat a set of operations D. It is used to throw errors in the program 23 Develop a typical JS ON Node.js App COMPS381F 25 Required Files for Node.js APPs ❑ A folder (directory) for storing code ▪ Express, a framework for writing Node.JS app (introduced later in the course), uses a specific directory structure storing different types of resources ❑ A package.json file that contains ▪ ▪ ▪ ▪ App name, description, version, author details & etc. App dependencies, startup script and etc. Other info (see: https://docs.npmjs.com/files/package.json) Example: { "name": "simple_server", "description": "A very simple Node.JS app", "author": "Raymond SO <[email protected]>", "version": "0.0.1", "dependencies": { "express": "*" }, "engine": "node >= 0.10.0", "scripts": {"start": "node simple_server"} } ❑ Run "npm install" to download and install dependencies ❑ Run "npm start" to run "node simple_server" 26 Write Node.js Scripts A typical Node.js APP should prov ide the following serv ices (Applicable to any server apps): 1. Listen on a particular TCP port • Await client requests 2. Identify different requests (what do clients want?) localhost:8099/greetings Print “Hello There!”” localhost:8099/greetings/sayHello Print “Hello There!” localhost:8099/greetings/sayHelloWithTime Print “Hello There! It is now XXXX” Parse request parameters (if any) Others: Returns 404 3. Process the request 4. Respond to the client • Return data in plain text, HTML or JSON format 27 A Node.js APP Sample ❑ Reference command lines in an Ubuntu terminal: git clone https://github.com/yalin-liu/comps381-2023.git cd comps381-2023/lab03/httpsamples/rectangle gedit server.js gedit package.json npm install npm start Check the files and run it! 28 Source code: https://github.com/yalin-liu/comps381-2023/tree/main/lab04/httpsamples/rectangle/server.js How to process the request/response from/to users? --- HTTP 29 HTTP Methods: JS ON GET VS. POST COMPS381F 30 What is HTTP? ❑ HTTP stands for Hypertext Transfer Protocol • • enable communications between clients and servers a request-response protocol between a client and server ❑ Examples of client and server: ▪ Clients: A web browser, a mobile app (written in any languages on any platforms), kiosk, interactive voice response system or yet another server! ▪ Server: A database server (such as Oracle, MySQL, MSSQL, MongoDB), internet server (Apache, MS IIS), application servers (Tomcat, IBM WebSphere, ASP.NET), PHP and Node.JS scripts 31 The HTTP Protocol 1. A client submits an HTTP request to the server 2. Then the server returns a response to the client ❑ The response contains (server-side apps) • • • HTTP Headers status information about the request the requested content, etc. 32 HTTP Headers May contain Message Body (as in GET/POST request) HTTP headers carry information about the client browser, the requested page, the server and more. HTTP 1.1 Spec defines the message format for HTTP request and response (https://tools.ietf.org/html/rfc2616) a.k.a Message Body Request resources (such as HTML document) are stored in the message body 33 Source: http://code.tutsplus.com/tutorials/http-headers-for-dummies--net-8039 Tools For HTTP Headers & Responses ❑ Live HTTP Headers • https://addons.mozilla.org/en-US/firefox/addon/http-header-live/ ❑ Firebug • http://getfirebug.com ❑ Postman (useful when writing RESTful Web services) • https://www.getpostman.com ❑ CURL (command line program) • Type the following in an Ubuntu terminal: curl -v -X GET http://hkmu.edu.hk 34 Check HTTP Headers ❑ Type a CURL line “curl -v -X GET http://hkmu.edu.hk ” in an Ubuntu terminal: • CURL stands for “client command line tool”, used to transfer data to and from a server. 35 Reference: https://curl.se/, https://www.geeksforgeeks.org/curl-command-in-linux-with-examples/ HTTP Header – Example 1 When you type a url such as: http://net.tutsplus.com/tutorials/other/top-20-mysql-best-practices in your address bar, your browser sends an HTTP request and it may look like this: GET /tutorials/other/top-20-mysql-best-practices/ HTTP/1.1 Host: net.tutsplus.com User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Connection: keep-alive Cookie: PHPSESSID=r2t5uvjq435r4q7ib3vtdjq120 Pragma: no-cache Cache-Control: no-cache Headers other than Host, under most circumstances, are optional. 36 HTTP Header – Example 2 After your request is sent, your browser receives an HTTP response that may look like this: HTTP/1.x 200 OK Transfer-Encoding: chunked Date: Sat, 28 Nov 2009 04:36:25 GMT Server: LiteSpeed Connection: close X-Powered-By: W3 Total Cache/0.8 Pragma: public Expires: Sat, 28 Nov 2009 05:36:25 GMT Etag: "pub1259380237;gz" Cache-Control: max-age=3600, public Content-Type: text/html; charset=UTF-8 Last-Modified: Sat, 28 Nov 2009 03:50:37 GMT X-Pingback: http://net.tutsplus.com/xmlrpc.php Content-Encoding: gzip Vary: Accept-Encoding, Cookie, User-Agent Status Line HTTP headers Content <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtm <html xmlns="http://www.w3.org/1999/xhtml"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <title>Top 20+ MySQL Best Practices - Nettuts+</title> <!-- ... rest of the html ... --> When developing a server-side app, you need to prepare the above HTTP responses. 37 Two HTTP Request Methods: GET & POST ❑ Two most common HTTP methods for a requestresponse between a client and server are: • GET - Request data from a specified resource The query string (name/value pairs) is sent in the URL of a GET request: /test/demo_form.php?name1=value1&name2=value2 • POST - Send data to a server to create/update a resource The data sent to the server with POST is stored in the request body of the HTTP request: POST /test/demo_form.php HTTP/1.1 Host: w3schools.com name1=value1&name2=value2 38 The GET Method ❑ GET is used to retrieve data from html, images, JavaScript, CSS, etc. ❑ Most data that loads in your browser was requested using this methods. Example: When we request a HTML document • The very first line of the HTTP request may look like this: GET /tutorials/other/top-20-mysql-best-practices/ HTTP/1.1 • Once the html loads, the browser will start sending GET request for images, that may look like this: GET /wp-content/themes/tuts_theme/images/header_bg_tall.png HTTP/1.1 39 The GET Method – Example What’ll happen when the submit button is pressed? For example: Yalin Liu 40 Try it by accessing: http://www.w3schools.com/tags/att_form_method.asp The GET Method – Example ❑ The GET method appends the form-data to the URL in the form of a query string (name/value pairs) https://www.w3schools.com/action_page.php?fname=Yalin&lname=Liu 41 The GET Method: Notes ❑ Some notes on GET requests: • • • • • • GET requests GET requests GET requests GET requests GET requests GET requests Not secure can be cached remain in the browser history can be bookmarked should never be used when dealing with sensitive data have length restrictions should be used only to retrieve data 42 The POST Method ❑ Post sends data to a server to create/update a resource. Notes: GET can also be used to send data by the query string, but • GET has length limitations, cannot send large amounts of data • GET is not secure to send sensitive information (such as user id and passwords) ❑A POST request sends the data (e.g., the query string - name/value pairs) in the HTTP message body POST /test/demo_form.asp HTTP/1.1 Host: w3schools.com name1=value1&name2=value2 43 The POST Method - Example What’ll happen when the submit button is pressed? For example: Yalin Liu 44 The POST Method - Example ❑ The POST method will not append the form-data to the URL The form data is sent in the HTTP message body in the following form: POST /tags/demo_form_method_post.asp HTTP/1.1 Host: www.w3schools.com User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:38.0) Gecko/20100101 Firefox/38.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate Referer: http://www.w3schools.com/tags/tryit_view.asp?x=0.38527487521064185 Cookie: _ga=GA1.2.2010567626.1444090130; __gads=ID=ea8311b4fcaf5724:T=1444090130:S=ALNI_MbB7XreQjoBag9hERpzpzmpNnWYDw; _gat=1; ASPSESSIONIDACRCAACC=PAOIDNFBMCFIFGIHGCOJHECK Connection: keep-alive Content-Type: application/x-www-formurlencoded Content-Length: 22 fname=Yalin&lname=Liu 45 The POST Method: Notes ❑ Some notes on POST requests: • • • • POST POST POST POST requests requests requests requests are never cached do not remain in the browser history cannot be bookmarked have no restrictions on data length 46 GET VS POST 47 Source: https://www.w3schools.com/tags/ref_httpmethods.asp Other HTTP Request Methods 48 Reference: https://www.geeksforgeeks.org/different-kinds-of-http-requests/ HTTP Status Codes ❑ 200's are used for successful requests. • 200 OK • 206 Partial Content ❑ 300's are for redirections. ❑ 400's are used if there was a problem with the request. • 404 Not found • 401 Unauthorized • 403 Forbidden ❑ 500's are used if there was a problem with the server. • 505 Internal Server Error See complete List at http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10 49 Use Node.js to handle JS ON HTTP services COMPS381F 50 A Node.js APP Sample: greeting0.js var http = require('http'); var url = require('url'); Imports core modules See https://nodejs.org/api/ for a complete list of available modules var server = http.createServer(function (req,res) { var greetingMsg = "Hello there!"; console.log("INCOMING REQUEST: " + req.method + " " + req & res objects will be used to communicate with the client req.url); var parsedURL = url.parse(req.url,true); //true to get query as object if (!( parsedURL.pathname == '/greetings' || parsedURL.pathname == '/greetings/sayHello’ || parsedURL.pathname == '/greetings/sayHelloWithTime’ )) { res.writeHead(404, {"Content-Type": "text/plain"}); res.write("404 Not Found\n"); res.end(); } else { res.writeHead(200, {"Content-Type" : "text/html"}); res.write('<html><head><title>sayHello</title></head>'); greetingMsg + '</H1>'); res.writeHead() sets the HTTP header res.write() writes the HTTP message body res.end() closes the HTTP connection res.write('<body><H1>' + if (parsedURL.pathname == '/greetings/sayHelloWithTime') { var today = new Date(); res.write('<p>It is now ' + today.toTimeString() + '</p>'); } res.end('</body></html>'); } }); server.listen(process.env.PORT || 8099); Await client requests 51 Passing Request Parameters ❑ Let’s improve greeting0.js to accept GET parameters: • • localhost:8099/greetings/sayHello?name=raymond localhost:8099/greetings/sayHelloWithTime?name=john • If this name/value pair is provided, it’ll be used in the greeting message ❑ Turns the query string to JSON object: var parsedURL = url.parse(req.url,true); var queryAsObject = parsedURL.query; ❑ Retrieve the name/value pair: if (queryAsObject.name != null) { greetingMsg = "Hello " + queryAsObject.name; } else { greetingMsg = "Hello there!"; } 52 Parsing Request Parameters: greeting1.js var http = require('http'); var url = require('url'); var server = http.createServer(function (req,res) { var greetingMsg = "Hello there!"; console.log("INCOMING REQUEST: " + req.method + " " + req.url); var parsedURL = url.parse(req.url,true); //true to get query as object var queryAsObject = parsedURL.query; if (queryAsObject.name != null) { greetingMsg = "Hello " + queryAsObject.name; } if (!( parsedURL.pathname == '/greetings' || parsedURL.pathname == '/greetings/sayHello' || parsedURL.pathname == '/greetings/sayHelloWithTime' )) { res.writeHead(404, {"Content-Type": "text/plain"}); res.write("404 Not Found\n"); res.end(); } else { res.writeHead(200, {"Content-Type" : "text/html"}); res.write('<html><head><title>sayHello</title></head>'); res.write('<body><H1>' + greetingMsg + '</H1>'); if (parsedURL.pathname == '/greetings/sayHelloWithTime') { var today = new Date(); res.write('<p>It is now ' + today.toTimeString() + '</p>'); } res.end('</body></html>'); } }); server.listen(process.env.PORT || 8099); 53 Parsing Request Parameters: rectangle.js ❑ Write a Node.js server to calculate area of a rectangle: Example: http://localhost:8099/?width=10&length=20 var http = require('http'); require('url'); var url = function Rectangle(width,length) { this.width = width; this.length = length; this.area = this.width * this.length; } function handle_incoming_request(req, res) { console.log("INCOMING REQUEST: " + req.method + " " + req.url); var parsedURL = url.parse(req.url,true); //true to get query as object queryAsObject = parsedURL.query; var var obj = new Rectangle(queryAsObject.width, queryAsObject.length); res.writeHead(200, {"Content-Type" : "application/json"}); res.end(JSON.stringify(obj));; } var s = http.createServer(handle_incoming_request); s.listen(process.env.PORT || 8099); 54 HTTP Header – Content Type ❑ "Content-Type" in the HTTP Response Header must be set to allow clients properly handle the server's response data Example: Server sending JPEG files ("Content-Type": "image/jpeg") fs.exists(fname, function(exists) { http://localhost:8099/GET%20?fname=ouhk-logo.jpg if (exists) { console.log('Opening file:' + fname); fs.readFile(fname, function(err,data) { res.writeHead(200, {'Content-Type': 'image/jpeg'}); res.write(data); res.end(); }); // end fs.readFile() } // end if (exists) else { console.log('File not found: ' + fname); res.writeHead(404, {'Content-Type': 'text/plain'}); res.write('404 Not Found\n'); res.end(); } }) // end fs.exists() Source at: https://github.com/yalinliu/httpsamples/blob/94785b19881e5b970ea61b3c75d00dd35789d907/serveJPGFile.js 55 HTTP Samples via Node.js ❑ https://github.com/yalin-liu/httpsamples.git http://localhost:8099/ http://localhost:8099/login http://localhost:8099/login?id=xxx&password=yyy http://localhost:8099/?name=xxx&age=13 56 Process URL ❑ A standard HTTP URL url.parse(urlStr, [parseQueryString], [slashesDenoteHost]) • urlStr – a valid URL string • parseQueryString • true - parses the query string portion of the URL into an object literal • false – default • slashesDenoteHost • true – processes URL expressed using //host/path format • false – default Reference: https://nodejs.org/api/url.html 57 Properties of the Object Returned by URL.PARSE() 58 Process GET Requests var url = require('url'); console.log( url.parse( 'http://user:[email protected]:8080/p/a/t/h?query=string#hash', true )); Returns the following object: { href: 'http://user:[email protected]:8080/p/a/t/h?query=string#hash', protocol: 'http:', host: 'user:[email protected]:8080', auth: 'user:pass', hostname: 'host.com', port: '8080', pathname: '/p/a/t/h', search: '?query=string', query: { query: 'string' }, hash: '#hash', slashes: true } 59 Process POST Requests ❑ Post requests transmit their data as the body of the request. ❑ To access the data, you can buffer the data to a string in order to parse it. ❑ The data is accessible through the “data” events emitted by the request. ❑ When all the data has been received, the “end” event is emitted: var qs = require('querystring'); var data = ''; req.on('data', function(chunk) { data += chunk; }); req.on('end', function() { var post = qs.parse(data); console.log(post); }); 60 TCP PORTS JS ON (*optional) COMPS381F 61 What are TCP PORTs? ❑ Let's assume we have two applications running on one PC that require TCP/IP (Internet) communications. ❑ Assume one is a web browser and the other is an email client. Both applications send and receive packets with the same IP address, so how does the Transport layer differentiate a web browser packet from an email packet? ❑ The answer is port numbers . 62 Source: http://microchip.wikidot.com/com4104:tcp-ip-ports What are TCP PORTs? 63 Source: http://microchip.wikidot.com/com4104:tcp-ip-ports TCP/IP “Well-known” PORTs ❑ "Well-Known" ports are port numbers that have been reserved for common applications, typically server applications. ❑ Clients know that servers will be listening for their requests at these reserved port numbers. Example: Web browsers (clients) connect to Web servers listening on port 80 (by default) mongo connects to MongoDB servers listening on port 27107 (by default) Try connect to port 28017 via a web browser… 64 Source: http://microchip.wikidot.com/com4104:tcp-ip-ports Server Creates Socket And Listens server.listen(process.env.PORT || 8099); Your Node.js server listens on port 8099 or a port assigned by cloud vendors Source: http://microchip.wikidot.com/com4104:example-use-sockets-to-create-a-tcp-connection 65 Client Create a Socket and Connects Web browsers, curl and mobile apps connect to your Node.js sever that listens on port 8099 Source: http://microchip.wikidot.com/com4104:example-use-sockets-to-create-a-tcp-connection 66 Transport Layer Delivers Message To Server Source: http://microchip.wikidot.com/com4104:example-use-sockets-to-create-a-tcp-connection 67 Server Create Socket & Process Source: http://microchip.wikidot.com/com4104:example-use-sockets-to-create-a-tcp-connection 68 Transport Layer Delivers Message to Cient Source: http://microchip.wikidot.com/com4104:example-use-sockets-to-create-a-tcp-connection 69 Sockets Closed Source: http://microchip.wikidot.com/com4104:example-use-sockets-to-create-a-tcp-connection 70 Review 1. Asynchronous Programming in Node.js: • Call-Back functions 2. Develop A Typical Node.js APP: • Use `npm` to install dependencies and run Node.js app 3. HTTP Methods: • GET (not secure, only to request data) • POST (send data to the server) 4. Use Node.js to handle HTTP services: • Passing Request Parameters 5. TCP PORTS* 71 The coming tutorials! P04, Fri. P02, Fri. P01, Next P05, Next P03, Next 9:00~11:00, in D0625 11:00~13:00, in D0627 Mon. 14:00~16:00, in D0626 Mon. 16:00~18:00, in D0627 Tue. 14:00~16:00, in D0627 Tutorial tasks: • • • • Asynchronous Programming in Node.js Use Node.js to handle HTTP Requests Write Node.js Apps Practices after-class 72 Q&A 73 FYI: HTTP VS. HTTPS ❑ All HTTP requests and responses are then encrypted with public session keys, so that anyone who intercepts communications can only see a random string of characters, not the plaintext. ❑ If a website uses HTTP instead of HTTPS, all requests and responses can be read by anyone who is monitoring the session. Essentially, a malicious actor can just read the text in the request or the response and know exactly what information someone is asking for, sending, or receiving. ❑ The S in HTTPS stands for "secure." HTTPS uses TLS (or SSL) to encrypt HTTP requests and responses, so an attacker would see a bunch of seemingly random characters. 74