1. Intro
MongooseIM is a fork of ejabberd 2.1.x server, maintained by Erlang Solutions Ltd. It has proven many times to be robust, reliable software capable of running on big clusters.
Since it implements open protocol – XMPP – its applications can vary from chat servers for online games, through mobile software, up to serving as a control channel in apps unrelated to messaging.
MongooseIM might not be as feature-rich as some other XMPP servers, but its main goals are scalability and performance. New functionalities are introduced only when sure it won’t become a bottleneck or at least is optimized in the best possible way.
2. Key features
What we’ve changed first, was the main data type. Ejabberd 2.1.x operates on lists and we have replaced them with binaries, which reduces memory usage.
Over time we “converted” some of the extension modules to be compatible with MIM and checked them for bottlenecks and ways of eliminating them.
Our most recent effort was to improve the XML parser engine. The test results will be published in the next part of this article.
MongooseIM is:
* Written in Erlang with regards to OTP principles
* Fault tolerant
* Scalable – having 10 machines in a single cluster is not a problem
* Flexible – with pluggable modules, it is possible to customize the server and gain performance by disabling unnecessary add-ons
* Ready for the modern Web - Supports WebSockets
* Supports multiple domains within a single cluster
* Offers Redis support for session storage
* Has a very good test coverage.
3. Load Tests
The load tests we present were executed on the latest release of MongooseIM and ejabberd.
Software
OTP R15B02
ejabberd 2.1.11 (most recent version on Github)
MongooseIM 1.1
MySQL 5.5.24-0ubuntu0.12.04.1
Tsung 1.5.0a
Server node
Ubuntu 12.04 LTS, kernel 3.2.0-23-generic
AMD FX-8150 8x3.3GHz
32GB RAM
XMPP server configuration
Sessions: in Mnesia
User credentials: in MySQL
Rosters: in MySQL
Both ejabberd and MongooseIM used the same configuration file. Following Erlang VM tweaks were used:
+K true
+A 5
+P 10000000
-env ERL_MAX_PORTS 1000000
-env ERL_FULLSWEEP_AFTER 2
-env ERL_MAX_ETS_TABLES 100000
Also following modules were enabled in all tests:
mod_muc
mod_muc_log
mod_disco
mod_adhoc
mod_last
mod_privacy
mod_private
mod_register
mod_roster_odbc
mod_sic
4. “Max users” test
User arrival rate: 150/s
User count: 400k
Roster size: 100
Users don’t fetch rosters
No message exchange
The aim of this test was to check resource usage with maximum possible count of online users connected to server.
MongooseIM
During user arrival phase server CPU usage was about 50-70% per each core. After all 400k users connected, memory usage was 66%. It is probable that such setup would be able to handle about 500k-600k online users.
ejabberd
ejabberd uses slightly more CPU (about 70-80% usage) and about 30% more memory (while having 50k users less online than in case of MongooseIM!). Server behaviour is also significantly less predictable, because connect time often reaches 3.5sec and login time can rise up to 90sec! Due to these problems, the number of connected users might increase slower and with less stability.
Conclusion
MIM can handle higher user arrival rate and total user count.
5. “Max messages” test
User arrival rate: 100/s
User count: 75k
Roster size: 100
~21k messages per second
The aim of this test was to check resource usage under heavy load and another important parameter: message latency.
Comments
Under heavy load and with exactly the same configuration both servers show similar performance in terms of CPU. MongooseIM’s advantage shrinks to about 300MB at the end of the test but a sudden increase in message rate results in lower memory peak in case of MIM and memory usage becomes stable quicker.
Message delivery latencies are also comparable and never higher than 350ms.
6. Erlang Halfword emulator
Erlang Halfword emulator mode was officially introduced in R14B02 and can only be used when erts is compiled as 64-bit application. Here is the information from R14B02 README:
From this release, the previously experimental halfword emulator is now official. It can be enabled by giving the –enable-halfword-emulator option to the configure script.
The halfword emulator is a 64-bit application, but uses halfwords (32-bit words) for all data in Erlang processes, therefore is using less memory and is faster than the
standard 64-bit emulator. The total size of all BEAM code and all process data for all processes is limited to 4Gb, but ETS tables and off-heap binaries are only limited by the amount of available memory.
Unfortunately in this mode both MongooseIM and ejabberd crashed quite often.
7. Multi User Chat test
MUC was tested in 3 variants: 2 “big rooms” and “many small rooms”.
Big room 1
User arrival rate: 10/s
Room size: 500
Message rate: ~60/s
In both cases message latency was virtually the same and always lower than 20ms.
MongooseIM used more CPU (45% opposed to ~37%) but less memory. Since user count was low, the difference in RAM usage was about 100MB.
Big room 2
User arrival rate: 3/s
Room size: 3000
Message rate: ~10/s
In both cases message latency was virtually the same and always lower than 150ms. As for MIM we could observe higher peaks though.
Both servers performed the same in terms of CPU usage but MIM uses almost 300MB less memory.
Many small rooms
User arrival rate: 60/s
Users: 50k
Rooms: 2000
Avg. room size: 25
Message rate: ~1650/s
Avg. message rate per room: 0.8/s
Message latency was always lower than 600ms but in general MIM achieved better or at least not worse delivery times.
During arrival phase MIM CPU usage was slightly better (as in “Max messages”) and during message exchange there wasn’t much difference between MIM and ejd.
Just like in the “Max messages” test, ejd uses significantly more memory and MIM advantage shrinks during message exchange but it is still about 750MB.
8. WebSockets vs. BOSH
Ejabberd features BOSH support but we wanted to check if actually adding WebSockets support is a better alternative. Perhaps WS are not supported by some older software but it might be a fair price for higher performance.
User count: 10k
User arrival rate: 90/s
Message rate: 6k/s
WebSockets use only slightly more CPU than ordinary XML stream but we can’t say the same about BOSH. The difference is very large: ~40% of CPU for WebSockets and 60-70% for BOSH.
Memory usage is also much higher. During user arrival phase it can be ~4.2GB in case of MIM WS and 11.5GB in case of ejabberd BOSH.
During message exchange phase the advantage shrinks to about 1GB.
9. Conclusion
MIM and ejabberd are both winners but in various categories. In all cases MIM uses less RAM. Sometimes difference is small, sometimes it becomes significant.
CPU usage was slighly smaller during arrival phase but during message exchange the difference is negligible. Sometimes ejd is at advantage.
Message latencies are sometimes better for MIM, sometimes for ejd and there are tests that show no difference.
The important conclusion is that WebSockets support promises to be much more efficient than BOSH.
