Benchmarking is the practice of measuring the performance of a system to identify its limits. Benchmarking is an integral part of our service’s development process and we use it both for regression testing (verifying performance has not been reduced between releases) and also as an aid for optimizing database performance. It is generally considered good practice to measure the performance of your system periodically as well as after every change made to ensure it is achieving maximal performance and to potentially uncover relevant issues. To facilitate the execution of these benchmark runs, we developed our own benchmarking tool that we fondly call memtier_benchmark.
This tool can be used to generate various traffic patterns against both Memcached and Redis instances. It provides a robust set of customization and reporting capabilities all wrapped into a convenient and easy-to-use command-line interface. We use it extensively for all our benchmarking needs, both in development as well as to help share our knowledge with the broader community (for example, as we did in this blog post). We’ve released memtier_benchmark source code under the GPLv2 licensing scheme, and you can download it from our github account. The following are some highlights about the tool.
- It supports both Redis and Memcached (binary and text) protocols so you can use it to test both and even compare their performance under different scenarios
- memtier_benchmark is capable of launching multiple worker threads (using the -t option), with each thread driving a configurable number of clients (the -c option). Multiple threads enable you to better utilize the hardware resources of your benchmarking client and generate high traffic loads even using a single server.
- When you instruct the tool to do multiple iterations of the benchmark run (the -x option), it will automatically produce reports for the best and worst runs as well as aggregated averages
- You can control the ratio between GET and SET operations with the ratio option to measure performance when dealing with different access patterns
- The tool offers control over the pattern of keys used by the GET and SET operations (key-pattern option). Each operation’s pattern can be set independently to be random or sequential.
- Keys’ expiry can be common or randomly set from a configurable range (the expiry-range option)
For more information about these and other features of the tool, below is memtier_benchmark’s usage clause (printed with the help switch). Do feel free to give it a go and let us know what you think.
Usage: memtier_benchmark [options] A memcache/redis NoSQL traffic generator and performance benchmarking tool. Connection and General Options: -s, --server=ADDR Server address (default: localhost) -p, --port=PORT Server port (default: 6379) -S, --unix-socket=SOCKET UNIX Domain socket name (default: none) -P, --protocol=PROTOCOL Protocol to use (default: redis). Other supported protocols are memcache_text, memcache_binary. -x, --run-count=NUMBER Number of full-test iterations to perform -D, --debug Print debug output --client-stats=FILE Produce per-client stats file --out-file=FILE Name of output file (default: stdout) --show-config Print detailed configuration before running Test Options: -n, --requests=NUMBER Number of total requests per client (default: 10000) -c, --clients=NUMBER Number of clients per thread (default: 50) -t, --threads=NUMBER Number of threads (default: 4) --test-time=SECS Number of seconds to run the test --ratio=RATIO Set:Get ratio (default: 1:10) --pipeline=NUMBER Number of concurrent pipelined requests (default: 1) --reconnect-interval=NUM Number of requests after which re-connection is performed --multi-key-get=NUM Enable multi-key get commands, up to NUM keys (default: 0) -a, --authenticate=PASSWORD Authenticate to redis using PASSWORD --select-db=DB DB number to select, when testing a redis server Object Options: -d --data-size=SIZE Object data size (default: 32) -R --random-data Indicate that data should be randomized --data-size-range=RANGE Use random-sized items in the specified range (min-max) --data-size-list=LIST Use sizes from weight list (size1:weight1,..sizeN:weightN) --expiry-range=RANGE Use random expiry values from the specified range Imported Data Options: --data-import=FILE Read object data from file --generate-keys Generate keys for imported objects Key Options: --key-prefix=PREFIX Prefix for keys (default: memtier-) --key-minimum=NUMBER Key ID minimum value (default: 0) --key-maximum=NUMBER Key ID maximum value (default: 10000000) --key-pattern=PATTERN Set:Get pattern (default: R:R) --help Display this help --version Display version information
EDIT: check out the description of memtier_benchmark's new version with pseudo-random data, Gaussian access pattern and range manipulation here