Web REST API Benchmark on a Real Life Application

Working as a web freelancer I am interested in how different frameworks and technologies perform, but the majority of the benchmarks found over the internet are considering just the Hello World example.

When you are building a real life application there are more aspects to take into consideration, so I decided to run a complete benchmark between the most popular frameworks and technologies.

Alongside the performance, I was interested in how easy is to achieve specific tasks in each framework and what is the cost of scaling the application performance.

Who are the candidates

  • Laravel 5, PHP 7.0, Nginx
  • Lumen 5, PHP 7.0, Nginx
  • Express JS 4, Node.js 8.1, PM2
  • Django, Python 2.7, Gunicorn
  • Spring 4, Java, Tomcat
  • .NET Core, Kestrel

What are we testing

We are interested in the number of requests per seconds each framework achieves on different server configurations alongside how clean or verbose the code looks like.

How server configurations look

We are also interested in how each framework scales its performance and what is the price of achieving such performance. This is why we’ll test them on 3 different server configurations using DigitalOcean:

  • 1 CPU, 512 MB – $5 / month
  • 4 CPU, 8 GB – $80 / month
  • 12 CPU, 32 GB – $320 / month

What are we building

We want to test a real life application example so we’ll basically build a Web REST API that exposes 4 endpoints, each one having a different complexity:

  1. Hello World  simply respond with a JSON containing Hello World string.
  2. Computation – compute the first 10.000 Fibonacci numbers.
  3. Simple Listing – we have a MySQL database containing a countries table and we’ll list all the countries.
  4. Complex Listing – we add a users table alongside a many-to-many mapping between users and countries and we want to list all users that visited France, alongside all countries each one visited.

For building the last two endpoints we’ll use the tools each framework provides to achieve our goal in the easiest possible way.

How are we testing them

For testing them we’ll use both wrk and ab HTTP benchmarking tools in order to check if we get similar results and also variate the concurrency of the requests so each technology can achieve its maximum potential.

These tools will run on their own droplet created on DigitalOcean so they don’t compete on server resources with the actual API application.

Also, the server used for placing the test requests and the one used to run the application are linked using their private IP so there won’t be any noticeable network latency.

Benchmark results

Below you can see the results grouped by each endpoint and also you can check on a single chart how each framework scaled on different server configurations.

How the API is built

Below are the actual controllers used for each framework to make an idea about how the code looks like. You can also check the whole code which is available on Github.

Laravel and Lumen with PHP

<?php
namespace App\Http\Controllers;
use Illuminate\Routing\Controller as BaseController;
use App\User;
use App\Country;
class Controller extends BaseController
{
    public function hello() 
    {
        return response()->json(['hello' => 'world']);
    }
    public function compute()
    {
        $x = 0; $y = 1;
        $max = 10000 + rand(0, 500);
        for ($i = 0; $i <= $max; $i++) {
            $z = $x + $y;
            $x = $y;
            $y = $z;
        }
        return response()->json(['status' => 'done']);
    }
    public function countries()
    {
        $data = Country::all();
        return response()->json($data);
    }
    public function users()
    {
        $data = User::whereHas('countries', function($query) {
                        $query->where('name', 'France');
                    })
                    ->with('countries')
                    ->get();
        return response()->json($data);
    }
}

Express JS with Node.js

const Country = require('../Models/Country');
const User = require('../Models/User');

class Controller 
{
    hello(req, res) {
        return res.json({ hello: 'world' });
    }

    compute(req, res) {
        let x = 0, y = 1;
        let max = 10000 + Math.random() * 500;

        for (let i = 0; i <= max; i++) {
            let z = x + y;
            x = y;
            y = z;
        }

        return res.json({ status: 'done' })
    }

    async countries(req, res) {
        let data = await Country.fetchAll();
        return res.json({ data });
    }

    async users(req, res) {
        let data = await User.query(q => {
                q.innerJoin('UserCountryMapping', 'User.id', 'UserCountryMapping.userId');
                q.innerJoin('Country', 'UserCountryMapping.countryId', 'Country.id');
                q.groupBy('User.id');
                q.where('Country.name', 'France');
            })
            .fetchAll({
                withRelated: ['countries']
            })

        return res.json({ data });
    }
}

module.exports = new Controller();

Django with Python

from django.http import JsonResponse
from random import randint
from models import Country, User, UserSerializer, CountrySerializer

def hello(req):
    return JsonResponse({ 'hello': 'world' })

def compute(req):
    x = 0
    y = 1
    max = 10000 + randint(0, 500)

    for i in range(max):
        z = x + y
        x = y
        y = z

    return JsonResponse({ 'status': 'done' })

def countries(req):
    data = Country.objects.all()
    data = CountrySerializer(data, many=True).data

    return JsonResponse({ 'data': list(data) })

def users(req):
    data = User.objects.filter(usercountrymapping__countries__name='France').all()
    data = UserSerializer(data, many=True).data

    return JsonResponse({ 'data': list(data) })

Spring with Java

package app;

import java.util.List;
import java.util.Random;
import java.util.concurrent.atomic.AtomicLong;

import org.hibernate.Criteria;
import org.hibernate.SessionFactory;
import org.hibernate.criterion.Restrictions;
import org.json.JSONException;
import org.json.JSONObject;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

import model.Country;
import model.User;

@RestController
public class Controller {
    private SessionFactory sessionFactory;
    
    public Controller(SessionFactory sessionFactory) {
        this.sessionFactory = sessionFactory;
    }

    @RequestMapping(value = "/hello", produces = "application/json")
    public String hello() throws JSONException {
        return new JSONObject().put("hello", "world").toString();
    }
    
    @RequestMapping(value = "/compute", produces = "application/json")
    public String compute() throws JSONException {
        long x = 0, y = 1, z, max;
        Random r = new Random();
        max = 10000 + r.nextInt(500);
        
        for (int i = 0; i <= max; i++) {
            z = x + y;
            x = y;
            y = z;
        }
        
        return new JSONObject().put("status", "done").toString();
    }
    
    @RequestMapping(value = "/countries", produces = "application/json")
    @Transactional
    public List<Country> countries() throws JSONException {
        List<Country> data = (List<Country>) sessionFactory.getCurrentSession()
                .createCriteria(Country.class)
                .list();
        return data;
    }
    
    @RequestMapping(value = "/users", produces = "application/json")
    @Transactional
    public List<User> users() throws JSONException {
        List<User> data = (List<User>) sessionFactory.getCurrentSession()
                .createCriteria(User.class)
                .createAlias("countries", "countriesAlias")
                .add(Restrictions.eq("countriesAlias.name", "France"))
                .list();
        return data;
    }
}

.NET Core

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Microsoft.EntityFrameworkCore;
using dotnet_core.Models;
using dotnet_core.Data;

namespace dotnet_core.Controllers
{
    public class MyController : Controller
    {
        private readonly ApplicationDbContext context;

        public MyController(ApplicationDbContext context)
        {
            this.context = context;
        } 

        [HttpGet]
        [Route("/hello")]
        public IEnumerable<string> hello()
        {
            return new string[] { "hello", "world" };
        }

        [HttpGet]
        [Route("/compute")]
        public IEnumerable<string> compute()
        {
            int x = 0, y = 1, z, max;
            Random r = new Random();
            max = 10000 + r.Next(500);

            for (int i = 0; i <= max; i++) {
                z = x + y;
                x = y;
                y = z;
            }

            return new string[] { "status", "done" };
        }

        [HttpGet]
        [Route("/countries")]
        public IEnumerable<Country> countries()
        {
            return context.Country.ToList();
        }

        [HttpGet]
        [Route("/users")]
        public object users()
        {
            return context.UserCountryMapping
                    .Where(uc => uc.country.name.Equals("France"))
                    .Select(uc => new {
                        id = uc.user.id,
                        firstName = uc.user.firstName,
                        lastName = uc.user.lastName,
                        email = uc.user.email,
                        countries = uc.user.userCountryMappings.Select(m => m.country)
                    })
                    .ToList();
        }
    }
}

Conclusions

Having in mind that in a real world application almost all the requests interact with the database, none of the choices are bad and all of them could handle the requirements of most web applications.

However, the performance of the Node.js with Express JS is quite remarkable. It competes with technologies such as Java and .NET Core or even outperforms them and combined with the simplicity of Javascript ES6 which you can natively use with Node.js 8, it provides so much power.

Regarding application scalability, the best performance per cost was gained on the middle size server. Adding 12 cores and 32 GB of memory didn’t help too much. Maybe, in this case, the bottle neck is somewhere else or it requires fine tunings to unlock the full server potential.

What do you think?

If you found the results interesting please click the 💚️ button so other can see them, too. You can find all the source code here:

https://github.com/mihaicracan/web-rest-api-benchmark