How Node.js Works Plus Its Key Uses In Web Apps Development

Node.js is a very popular JavaScript runtime environment. And it’s continuing to have a positive impact on various users too particularly around developer productivity and satisfaction. When asked to describe Node.js, respondents use positive terms like – “fast”, “easy”, “awesome”, “powerful”, “flexible,” etc. Thus, it’s one of the most popular programming languages.

Likewise, JavaScript is also one of the most universal software development technologies. Traditionally used as a web frontend development tool, it has also become a major cross-platform mobile development tool as a basic technology for a large number of platforms. Such as Apache Cordova/PhoneGap, React Native, NativeScript, Appcelerator Titanium, etc.

But, the areas of application for JavaScript do not end here. Lately, there has been a lot of buzz around the use of JavaScript for server-side programming. One of the tools that indicated this shift in web development was Node.js. Realistically, Node.js is actually not a framework or a library. Instead, it’s a runtime environment, based on Chrome’s V8 JavaScript engine.

What Node.js Is All About

Node.js is a platform built on Chrome’s JavaScript runtime for easily building fast and scalable network applications. The technology was first introduced back in 2009 by Ryan Dahl at the annual European JSConf and was immediately recognized as “the most exciting single piece of software in the current JavaScript universe”.

It uses an event-driven, non-blocking I/O model that makes it lightweight and efficient. It’s also perfect for data-intensive real-time applications that run across distributed devices. Additionally, it also provides a rich library of various JavaScript modules. And as a result, which simplifies the development of web-based applications.

Node.js = Runtime Environment + JavaScript Library

The main idea of Node.js: use non-blocking, event-driven I/O to remain lightweight and efficient in the face of data-intensive real-time applications that run across distributed devices.

In other words, Node.js is an open-source, cross-platform runtime environment for developing server-side and networking applications. Most if not all its applications are written in JavaScript. And can be run within the Node.js runtime on OS X, Microsoft Windows, and Linux.

How Node.js Environment Works

In simple terms, Node.js is an open-source backend JavaScript runtime environment. It is used as a backend service where javascript works on the server side of the app. This way JavaScript is used on both the front-end and the back-end environments. Node.js runs on the Chrome V8 JavaScript Engine which converts JavaScript code into machine code.

On one side, of it’s key significant working features is that it’s highly scalable, lightweight, fast, and data-intensive. On the other side, Node.js accepts the request from the clients and sends the response, while working with the request node.js handles them with a single thread. To operate I/O operations or requests node.js use the concept of threads.

Whilst, bearing in mind, that a thread is a sequence of instructions that the server needs to perform. It runs parallel on the server to provide the information to multiple clients. Node.js is an event loop single-threaded language. It can handle concurrent requests with a single thread without blocking it for one request. Basically, Node.js works on two concepts. Such as follows: The non-blocking I/O concept side and the asynchronous concept side:

The Non-Blocking I/O Environment Concept

To enumerate, the non-blocking i/o concept means working with multiple requests without blocking the thread for a single request. I/O basically interacts with external systems such as files, and databases. Node.js is not used for CPU-intensive work means for calculations, and video processing because a single thread cannot handle the CPU works.

The Asynchronous Environment Concept

In the same fashion, the asynchronous concept means executing a callback function. The moment we get the response from the other server or database it will execute a callback function. Callback functions are called as soon as some work is finished and this is because Node.js uses an event-driven architecture. The single thread doesn’t work with the request instead it sends the request to another system which resolves the request and it is accessible for another request.

Be that as it may, to implement the concept of the system to handle the request node.js uses the concept of Libuv.

What Is Libuv, And Why Do Node.js Developers Need It?

By definition, Libuv is an open-source library that in terms of the Programming Languages concept, it’s a built-in C program. Simply put, Libuv is an open-source library that handles the thread pool, signaling, inter-process communications all other magic needed to make the asynchronous tasks work at all.

Originally, Libuv was developed for Node.js itself as an abstraction of libev. However, as of now, multiple projects are already using it. Most people think Libuv is the event loop itself, but this is not true! For one thing, Libuv implements a full-featured event loop and it’s also the homespace of several other key parts of Node in app design and webdev projects.

Some of these key parts include:

TCP and UDP sockets of the net package
Async DNS resolutions
Async file and file system operations (like the one we’re doing here)
File System events
IPC
Child processes and shell control
Thread pool
Signal handling
High-resolution clock

This is mainly why Node.js uses it, it’s a full abstraction around several key parts of every OS, and it is necessary for the whole runtime to interact with it’s surrounding environment.

In essence, it has a very strong focus on asynchronous and I/O. Whilst, giving Node access to the underlying computer operating system, file system, and networking. Libuv implements two extremely important features of node.js:

What An Event Loop Entails

Specifically, the event loop contains a single thread and is responsible for handling easy tasks like executing callbacks and network I/O. When the program is to initialize all the top-level code is executed, the code is not in the callback function.

All the application code that is inside callback functions will run in the event loop. Event Loop is the heart of node.js. When we start our node application the event loop starts running right away. Most of the work is done in the event loop.

Nodejs use event-driven-architecture. We may consider the following key stages: And, as such, during the looping process, firstly, the events are emitted, secondly, the event loop picks them up, and then, lastly, the callbacks are called.

What An Event Queue Entails

As soon as the request is sent the thread places the request into a queue. It is known as an event queue. The process like the app receiving an HTTP request or a server or a timer will emit events as soon as they are done with the work and the event loop will pick up these events and call the callback functions that are associated with each event and response is sent to the client.

The event loop is an indefinite loop that continuously receives requests and processes them. It checks the queue and waits for the incoming request indefinitely.

The Main Thread Pool Function

It’s, important to realize, that even though Node.js is single-threaded it internally maintains a thread pool. When non-blocking requests are accepted there are processed in an event loop, but while accepting blocking requests it checks for available threads in a thread pool, assigns a thread to the client’s request which is then processed and send back to the event loop, and response is sent to the respective client. For your information, the thread pool size can easily be changed.

Consider this line of a Node pool size:

process.env.UV_THREADPOOL_SIZE = 1;

Moving on, the next thing is understanding the way a Node.js design concept in application platforms usually runs. First of all, Node JS is not a server. In this sense, Node JS doesn’t serve anything directly. One needs to create an application server instance using other JS libraries.

Hence, it basically is Javascript with added features to support a majority of the core backend needs like access to the filesystem, OS system calls, and the like. The magic happens in the Chrome V8 JavaScript Engine which runs the node program.

A Sample Node.js Process Model In Applications

Just like we aforementioned, Node JS follows event-based programming. This is the absolute core of Node. It’s a single-threaded program that acts upon callbacks and never blocks on the main thread. Hence, instead of spawning a new thread for every new user, the Node loops through the events of all the users in a single thread itself.

This avoids thread creation overhead. So, ideally, the main thread only listens for events and nothing else. It never gets blocked because of the way the Node forces you to write code. Technically, the code you write should always be asynchronous and callbacks are driven. To have an idea, take a look at the diagram below:

What About The CPU-Intensive Applications?

Well, CPU-intensive apps are best suited for multi-threaded systems, obviously because they will be performed by multiple cores simultaneously. But turns out the node doesn’t work that way. Node-based apps are single-threaded.

Hence, a single thread will need to perform all the CPU-intensive jobs which will make it very slow. This is the reason if you have ever tried building image-processing apps on the node, you would have been pretty disappointed at the speed.

Learn More: Node.js Under The Hood #3 – Deep Dive Into the Event Loop

While, in the case of I/O operations, the node works really well. This is because I/O operations in the node are not carried out in the main thread. Most of the I/O is written as nonblocking asynchronous.

But, certain background I/O are insanely difficult to write in an asynchronous manner. For those I/O APIs are executed in separate threads and processes for DB access and process execution. Also, certain file operations are performed by OS. Internally, Node JS relies on libev to provide the event loop.

Essentially, this’s supplemented by libeio which uses pooled threads to provide asynchronous I/O. Thus, the main event loop only focuses on handling the events and not the associated operations, making the response time very fast.

Why & When The Hell Should You Use The Node.js Framework?

First of all, JavaScript’s rising popularity has brought with it a lot of changes. And the face of web development today is dramatically different. Definitely, there’re so many things that we can do on the web nowadays.

With JavaScript running on the server as well as in the browser. Some of them were totally hard to imagine just several years ago. Or were they just encapsulated within sandboxed environments like Flash or Java Applets? Before digging into Node.js solutions, you might want to read up on the benefits of using JavaScript across the stack in detail first.

Secondly, as Wikipedia states: “Node.js is a packaged compilation of Google’s V8 JavaScript engine, the libuv platform abstraction layer, and a core library, which is itself primarily written in JavaScript.” Beyond that, it’s worth noting that Ryan Dahl, the creator of Node.js, was aiming to create real-time websites with push capability, “inspired by applications like Gmail.”

The Main Node.js Features To Consider

When discussing Node.js, one thing that definitely should not be omitted is built-in support for package management using NPM. A tool that comes by default with every Node.js installation. The idea of NPM Modules is quite similar to that of Ruby Gems: a set of publicly available, reusable components. All are available through a simple and easy installation.

In particular, through an online repository, with version and dependency management. A full list of packaged modules can be found on the npm website or accessed using the npm CLI tool that automatically gets installed with Node.js.

The module ecosystem is open to all, and anyone can publish their own module to be listed in the NPM repository.

Some of the most useful NPM Modules today are:

Express.js:– A Sinatra-inspired web framework for Node.js. It’s the de facto of many standard apps.
Hapi:– a very modular and simple-to-use configuration-centric framework for building web and services apps
Connect:– An extensible HTTP server framework for Node.js, providing “plugins” known as middleware.
Socket.io /Sockjs:– Server-side component of the two most common WebSockets components today.
Pug.js (formerly Jade.js):- It’s one of the top templating engines, inspired by HAML, a default in Express.js.
MongoDB and Mongojs:– MongoDB wrappers to provide the API for MongoDB object databases in Node.js.
Lodash (underscore, lazy.js):– The JavaScript utility belt. It’s Underscore who initiated the game.
Forever:– Probably the most common utility for ensuring that a given node script runs continuously.
Bluebird:– A full-featured Promises/A+ implementation with exceptionally good performance
Moment.js:– A JavaScript date library for parsing, validating, manipulating, and formatting dates.
Redis:– Provides the Redis client library.

By far, NPM Modules are the most widely used package manager, but Yarn is also gaining popularity in various segments.

Other Important Features You Should Know About

There are tons of really useful packages out there, available to all (no offense to those that I’ve omitted here). For example, it uses a single-threaded program and the same program can provide service to a much larger number of requests.

Not to mention, this number of requests is more than traditional servers like Apache HTTP Server. That said, the following are some of the other important and memorable features that make it the first choice for many software architects.

Asynchronous, Event-Driven:−

All APIs of the Node.js library are asynchronous, that is, non-blocking. It essentially means a server-based one never waits for an API to return data. The server moves to the next API after calling it and a notification mechanism of Events of Node.js helps the server to get a response from the previous API call.

Single-Threaded, Highly Scalable:−

Node.js uses a single-threaded model with event looping. The event mechanism helps the server to respond in a non-blocking way and makes the server highly scalable as opposed to traditional servers which create limited threads to handle requests.

Very Fast, No Buffering;−

Being built on Google Chrome’s V8 JavaScript Engine, the Node.js library is very fast in code execution. Since Node.js is released under the MIT License, the applications never buffer any data. These applications simply output the data in chunks.

But, the above list of its features is not all! For one thing, it goes on and on!

Who Uses the Node.js Application?

As an open-source project, Node.js was sponsored by Joyent, a cloud computing and hosting solutions provider. The company invested in a number of other technologies such as the Ruby on Rails framework while providing cloud hosting services.

Particularly, for giant platform applications such as Twitter and LinkedIn. And, for its mobile application backend, the latter also became one of the first companies to use Node.js. The technology was later adopted by a number of technology leaders, such as Uber, eBay, Walmart, and Netflix, among others. The following is a link to the GitHub wiki to borrow ideas.

It contains an exhaustive list of projects, applications, and companies that are using Node.js. This list includes eBay, General Electric, GoDaddy, Microsoft, PayPal, Uber, Wikipins, Yahoo!, and Yammer to name a few. So, having said that, you can learn more about Projects, Applications, and Companies Using Node by GitHub in more detail.

In the past, the CPU-expensive operations were managed client-side with Microsoft Silverlight. But, in our future set-up, they would be done on the server. Surprisingly, I have found several resources on the net pointing to possible problems that an application with these requirements might face in Node.js due to its architecture.

Related Toolkit: Spectoos | The No #1 Platform To Add Website Testimonials

Of course, there are many other resources claiming that those problems do not exist. Or that they’re easily solvable with a certain module or offloading heavy tasks to another server. The last solution, however, would be difficult to implement for us due to several reasons. So, now the question we need to ask and answer is this:

Can Node.js manage applications required to perform CPU-intensive tasks as described? Unfortunately, it is not advisable to use Node.js for CPU-intensive applications for various key reasons as well. The following are the areas where Node.js is proving itself as a perfect technology partner.

The main examples are:

I/O bound Applications
Data Streaming Applications
Data-Intensive Real-time Applications (DIRT)
JSON APIs based Applications
Single Page Applications

It wasn’t until recently that the wide adoption of server-side JavaScript with Node.js started. Notably, its interest peaked in 2017, as per Google Trends, and remains high. Below are more examples of where to use it, or even, where you can find it in use:

1. Chats

Chat is the most typical real-time, multi-user application. From IRC (back in the day), through many proprietary and open protocols running on non-standard ports, to the ability to implement everything today in Node.js with WebSockets running over the standard port 80.

The chat application is really the sweet-spot example for Node.js: it’s a lightweight, high-traffic, data-intensive (but low processing/computation) application. That runs across distributed devices. It’s also a great use case for learning too, as it’s simple, yet it covers most of the paradigms you’ll ever use in a typical Node.js application.

2. API on Top of an object DB

Although Node.js really shines with real-time applications, it’s quite a natural fit for exposing the data from object DBs (e.g. MongoDB). JSON-stored data allow Node.js to function without impedance mismatch and data conversion.

For instance, if you’re using Rails, you would convert from JSON to binary models. And then expose them back as JSON over the HTTP. That’s whenever the data is consumed by Backbone.js, Angular.js, etc. Or even plain jQuery AJAX calls. With Node.js, you can simply expose your JSON objects with a REST API for the client to consume.

Additionally, you don’t need to worry about converting between JSON. Or whatever else when reading or writing from your database (if you’re using MongoDB). In sum, you can avoid the need for multiple conversions. By using a uniform data serialization format across the client, server, and database.

3. Queued Inputs

If you’re receiving a high amount of concurrent data, your database can become a bottleneck. As depicted above, Node.js can easily handle the concurrent connections themselves. But, because database access is a blocking operation (in this case), we run into trouble.

Fortunately, the solution is to acknowledge the client’s behavior before the data is truly written into the database. With that approach, the system maintains its responsiveness under a heavy load. Particularly, very useful when the client doesn’t need firm confirmation of the successful data written.

Typically, some good examples include the logging or writing of user-tracking data, processed in batches and not used until a later time. As well as operations that don’t need to be reflected instantly.

Like updating a ‘Likes’ count on Facebook where eventual consistency (so often used in the NoSQL world) is acceptable. Data gets queued through some kind of caching or message queuing infrastructure. Like RabbitMQ or ZeroMQ — and then digested by a separate database batch-write process.

Or computation-intensive processing backend services, written in a better-performing platform for such tasks. Similar behavior can be implemented with other languages/frameworks too. But, not on the same hardware, with the same high, maintained throughput.

4. Data Streaming

In short, with Node, you can push the database writes off to the side. And then deal with them later, proceeding as if they succeeded. Likewise, in more traditional web platforms, HTTP requests and responses are treated as isolated events.

In fact, they’re actually streams of key observations that can we can utilize in Node.js to build some cool features. For example, it’s possible to process files while they’re still being uploaded. And as the data comes in through a stream, we can then process it in an online fashion. This could be done for real-time audio or video encoding and proxying between different data sources.

5. Proxying

Node.js is easily employed as a server-side proxy. Where it can handle a large number of simultaneous connections in a non-blocking manner. It’s especially useful for proxying different services with different response times. Or else, collecting data from multiple source points.

As an example, let’s consider a server-side application communicating with third-party resources. And then, pulling in data from different sources. Or even storing assets like images and videos to third-party cloud services.

Although dedicated proxy servers do exist, using Node instead might be very helpful. That’s if your proxying infrastructure is non-existent. Or if you need a solution for local development. By this, I mean that you could build a client-side app.

More so, using a Node.js development server for assets and proxying/stubbing API requests. And while in production you’d handle such interactions with a dedicated proxy service (Nginx, HAProxy, etc.).

6. Application Monitoring Dashboards

In terms of brokerage, we can easily relate to stock traders’ dashboards. So, let’s get back to the application level. Whereby, this is another example where desktop software dominates but with a very easy replacement possibility with a real-time web solution.

A broker’s trading software is used in tracking stock prices. Or even perform calculations/technical analysis, and create graphs/charts. Switching to a real-time web-based solution would allow brokers to easily switch workstations or working places.

Another common use case in which Node-with-web-sockets fits perfectly is tracking website visitors. As well as visualizing their interactions in real time. Whereby, you could be gathering real-time stats from your web users.

Or even moving it to the next level by introducing targeted interactions with your visitors. By opening a communication channel when they reach a specific point in your funnel. (If you’re interested, this idea is already being productized by CANDDi.)

Imagine how you could improve your business if you know what your visitors were doing in real time. But only if you could visualize their interactions. With the real-time, two-way sockets of Node.js, now you can.

7. System Monitoring Dashboards

Now, let’s visit the infrastructure side of things. Imagine, for example, a SaaS provider that wants to offer its users a service-monitoring page, like GitHub’s status page. With the Node.js event loop, we can create a powerful web-based dashboard.

Basically, that checks the services’ statuses in an asynchronous manner and pushes data to clients using WebSockets. Both internal (intra-company) and public services’ status reports can be live and in real-time using this technology.

On the other hand, we can try to push that idea even a little further. And try to imagine a Network Operations Center (NOC) monitoring applications in a telecommunications operator. Or even a cloud/network/hosting provider, or some financial institution.

Well, they all run on the open web stack backed by Node.js and WebSockets instead of Java and/or Java Applets. However, don’t try to build hard real-time systems in Node (i.e., systems requiring consistent response times). Erlang is probably a better choice for that class of application.

8. Dedicated Server-Side Web Applications

In conjunction with Express.js, Node.js can also come in handy to create classic web applications on the server side. However, while possible, this request-response paradigm in which Node.js would be carrying around rendered HTML is not the most typical use case.

If your application doesn’t have any CPU-intensive computation, you can build it in Javascript top-to-bottom. Even if it’s down to the database level if you use JSON storage Object DB like MongoDB. This eases development (including hiring) significantly.

Crawlers receive a fully-rendered HTML response, which is far more SEO-friendly than, say, a Single Page Application or a WebSockets app run on top of Node.js. Any CPU-intensive computation will block Node.js responsiveness, so a threaded platform is a better approach. Alternatively, you could try scaling out the computation.

9. Heavy Server-Side Computation/Processing

When it comes to heavy computation, Node.js is not the best platform around. No, you definitely don’t want to build a Fibonacci computation server in Node.js. In general, any CPU-intensive operation annuls all the throughput benefits Node offers.

With its event-driven, non-blocking I/O model. Simply, because any incoming requests get a block while the thread is in occupation with your number-crunching. Assuming you’re trying to run your computations in the same Node instance you’re responding to requests with.

And, as we aforementioned, Node.js acts as a single thread meaning that it uses only a single CPU core. When it comes to adding concurrency on a multi-core server, the Node core team always makes sure that the work is successful. More so, in the form of a cluster module |ref:|

10. Nginx Reverse Proxy

By the same token, you can also run several Node.js server instances pretty easily behind a reverse proxy via Nginx. While with clustering, you should still offload all heavy computation to background processes written in a more appropriate environment for that.

And then, as a result, have them communicate via a message queue server like RabbitMQ. Initially, even though your background processing may run on the same server, such an approach has the potential for very high scalability.

In terms of your worker servers, they can also easily separate them out through distribution. Without the need to configure loads of front-facing web servers. Of course, you’d use the same approach on other platforms too. But, with Node.js you’ll get all that high reqs/sec throughput we’ve talked about.

More Quick Facts

The largest percentage of users deploy on Amazon Web Services followed by on-premise infrastructure.
A good number of users deploy to Heroku, DigitalOcean, Google Cloud Platform, and Microsoft Azure too.
Web Apps remain the most popular use case for Node, with 85% of respondents indicating they use Node.
43% of respondents indicate they use Node for enterprise applications.
Integration with databases, front-end frameworks, and load balancing is at the forefront too.
Most users expect to increase their use of Node.js, particularly in Latin America and EMEA.
Babel is the leading transpiler, but back-end, full-stack, and ‘other’ developers are increasing their use too.
Among Module Bundlers, Webpack seems to be consolidating its lead across most developed regions.
It’s also becoming increasingly important to users as they manage different packages for multiple environments.

Node.js continues to see its popularity grow on every continent and in a very broad set of use cases due to its flexibility and utility for a wide variety of use cases.

Node.js with Express.js against Ruby on Rails

Let’s compare Node.js with Express.js against Ruby on Rails, for example. There used to be a clear decision in favor of the latter. Especially, when it came to accessing relational databases. Like PostgreSQL, MySQL, and Microsoft SQL Server.

Relational DB tools for Node.js were still in their early stages. On the other hand, Rails automatically provides a data access setup right out of the box. Coupled with DB schema migrations support tools and other Gems (pun intended).

Other Useful Resource References Worth Mentioning

On the other hand, using Node.js with a relational database is still quite a pain. So, do yourself a favor and pick up any other environment like Rails, Django, or ASP.Net MVC if you’re trying to perform relational operations. An alternative to these CPU-intensive computations is to create a highly scalable MQ-backed environment.

Not to mention, with back-end processing to keep Node as a front-facing ‘clerk’ to handle client requests asynchronously. And, of course, your next question is: How do I start with Node.js after I installed it? Perse, once you’ve installed Node.js, now it’s time to build your first web server. Whereby, you’ll create a file named app.js containing the following contents:

const http = require('http');

const hostname = '127.0.0.1';
const port = 3000;

const server = http.createServer((req, res) => {
  res.statusCode = 200;
  res.setHeader('Content-Type', 'text/plain');
  res.end('Hello World');
});

server.listen(port, hostname, () => {
  console.log(`Server running at http://${hostname}:${port}/`);
});

Now, run your web server using node app.js. Visit http://localhost:3000 and you’ll see this message: “Hello World!”

Node.js Download

Generally, you can Install Long-Term Support and the latest versions of Node.js for Windows and MacOS here. Not forgetting, you can see the docs and getting started guides at this link. You can also refer to the Introduction to Node.js for a more comprehensive guide.

Node.js IDEs

Almost any popular code editor has support and plugins for JavaScript and Node.js, so it only matters how you customize your IDE to your coding needs. But, many developers highly praise special tools from VS Code, Brackets, Atom, and WebStorm.

Node.js Frameworks

Using middleware over pure Node.js is a common practice that makes developers’ lives easier. We have a separate article comparing popular Node.js frameworks, where we look at Express.js, Meteor, Sales.js, Koa.js, Keystone.js, and Loopback.js. For more JavaScript ecosystem tools used with Node.js, you can see their dedicated article.

It’s Node.js’ key strengths and notable weaknesses that make it the subject of a heated discussion. To set the record straight, we have analyzed both – the main Node.js pros and cons. In an attempt to find out what projects can benefit from this technology choice.

Final Thoughts:

In one sentence: Node.js shines in real-time web applications employing push technology over WebSockets. What is so revolutionary about that? Well, it’s after over 20 years of a stateless web based on the stateless request-response paradigm. And now, we’ve web apps with real-time, two-way connections for clients and servers to initiate communication.

Whilst, allowing them to exchange data freely. This is in stark contrast to the typical web response paradigm. Where the client always initiates communication. In addition, it’s all based on the open web stack (HTML, CSS, and JS) running over the standard port 80. For sure, one might argue that we’ve had this for years in the form of Flash and Java Applets.