chris moos's blog

libhashring – high performance consistent hashing

Chris Moos — Sat, 04 Jun 2011 23:49:11 +0000

In a previous post I talked about the need to shard Redis data and how I accomplished this by adding shard/hashing support to erldis, an Erlang client for Redis. This solution proved to work well, it distributed our data very well amongst many Redis servers — but there was one problem. Performance.

In the change I made to erldis, the hash ring was stored in ETS (an erlang memory store) and anytime a key was hashed, the ring had to be retrieved from ETS. The problem with this is that Erlang copies the entire ring when it comes out of ETS. The ring we used was very large, with thousands of items. Copying this every single time became a huge performance hit.

To mitigate the performance problems, I decided to implement the hash ring in C, and write an Erlang driver to use it. That is how libhashring was born. It is very high performance and currently has bindings for Erlang and Java. It is currently deployed in our production environment and its speed is incredible. I am confident that increasing the size of the ring as we add capacity will not cause an impact to the hash ring’s performance.

libhashring supports MD5 and SHA-1. MD5 seems to be about 25% faster than SHA-1, so if you want the extra performance, MD5 is probably the best bet.

Feel free to fork libhashring and make it even better, I’d be really happy to get some feedback and contributions.

P.S. – After finishing the hash ring, I came across libketama and realized it pretty much solves the same problem that I had. Too bad I didn’t see it sooner!

C example

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

hash_ring_t *ring = hash_ring_create(8, HASH_FUNCTION_SHA1);
char *slotA = "slotA";
char *slotB = "slotB";

char *keyA = "keyA";
char *keyB = "keyBBBB";
char *keyC = "keyB_";

hash_ring_node_t *node;

assert(hash_ring_add_node(ring, (uint8_t*)slotA, strlen(slotA)) == HASH_RING_OK);
assert(hash_ring_add_node(ring, (uint8_t*)slotB, strlen(slotB)) == HASH_RING_OK);

node = hash_ring_find_node(ring, (uint8_t*)keyA, strlen(keyA));
assert(node != NULL && node->nameLen == strlen(slotA) && memcmp(node->name, slotA, strlen(slotA)) == 0);

node = hash_ring_find_node(ring, (uint8_t*)keyB, strlen(keyB));
assert(node != NULL && node->nameLen == strlen(slotA) && memcmp(node->name, slotA, strlen(slotA)) == 0);

node = hash_ring_find_node(ring, (uint8_t*)keyC, strlen(keyC));
assert(node != NULL && node->nameLen == strlen(slotB) && memcmp(node->name, slotB, strlen(slotB)) == 0);

hash_ring_free(ring);

Thrift and ZooKeeper

Chris Moos — Wed, 25 May 2011 07:50:43 +0000

Thrift provides a great framework for developing and accessing remote services. It allows developers to create services that can be consumed by any application that is written in a language that there are Thrift bindings for (which is…just about every mainstream one, and more).

This is great for systems that are heterogeneous — for example, you could write a user authentication service in Java, but call it from your Ruby web application.

Thrift manages serialization of data to and from a service, as well as the protocol that describes a method invocation, response, etc,. This is great because instead of writing all the RPC code — you can just get straight to your service logic. Thrift uses TCP (not sure if UDP is/will be supported) and so a given service is bound to a particular port.

As you start to scale an infrastructure with Thrift services, you may find, as I have, that putting all of your Thrift server IP/port combinations in a configuration file that your clients read is just not…the best. If you have an environment where Thrift servers go down for maintenance (or crash), or you add more capacity on the fly, a dynamic way to manage the state of all of the services is needed.

Enter Apache ZooKeeper, a distributed, fault tolerant, highly available system for managing configuration information, naming, and more (distributed synchronization, anyone?).

ZooKeeper appears like a filesystem to clients, it is a hierarchy of znodes, which are analogous to directories or files, both of which can contain a small amount of data.

ZooKeeper can be used to store Thrift service location information, allowing clients to dynamically discover Thrift services. ZooKeeper even provides a way to create ephemeral znodes, which means that once your Thrift service goes down, it will be removed from ZooKeeper automatically. And if that isn’t cool enough, ZooKeeper even supports watches, where clients can ask to be notified whenever a znode changes. This means that clients can begin using new Thrift service capacity instantly, and when failures happen, clients will stop attempting to contact a down Thrift service.

Laying out your Thrift services in ZooKeeper is important. Clients will need to know about the layout when performing service discovery. For example, you can do something such as:

1
2
3

/services/user/user0000000001
/services/user/user0000000002
....

Clients can get a list of all znodes at /services/user to receive the list of servers for the user service. The only problem is…getting the list of znodes is only half the battle. user0000000001 doesn’t really tell you how to access the user service on that node. This is why its important to store some kind of service location data with the znode.

ZooKeeper allows you to set the data for a znode, so when a node comes up, it just needs to also set data at the znode that describes how to locate or access the service. I’ve adopted to use a URI as the znode data — this is very flexible and easy to parse and read from most languages. For Thrift services I am using a URI that uses a thrift scheme:

1	thrift://10.1.1.10:5656

The URI can be easily adapted to your application, such as adding query string parameters with any extra custom metadata.

Thrift is great, and with ZooKeeper, its even better. I’m in the process of implementing this integration now and am looking forward to all of the benefits it has to offer. I would love to hear any feedback about this approach if anyone has personal experience, or just some good ideas. What is everyone else using to solve this type of problem?

Oh and also, if ZooKeeper isn’t your thing, you should definitely check out Doozer. It has very similar features to ZooKeeper, and although new, it is definitely on my list of projects to watch. Oh, and did I mention that it is written in golang?

Doozer is a highly-available, completely consistent store for small amounts of extremely important data. When the data changes, it can notify connected clients immediately (no polling), making it ideal for infrequently-updated data for which clients want real-time updates. Doozer is good for name service, database master elections, and configuration data shared between several machines.

Redis sharding with Erlang

Chris Moos — Mon, 11 Apr 2011 03:09:35 +0000

I’ve been using a great Erlang Redis library erldis recently and as our data set has grown much larger, we have decided we need to start sharding. Unfortunately the erldis library does not have support for sharding so I’ve forked the repository and made the changes. You can find my fork here. Pull request here.

A little background

Redis has been really great for us so far but as we get more data and more operations per second, we realized we need to start utilizing multiple Redis instances. Sharding basically splits up your keyspace across multiple servers in a consistent fashion. The typical example for finding out where a key belongs is by doing something like:

1	hash(Key) mod N

Where N is the index to a list of shards. This allows all of your clients to be consistent with knowing where a key belongs. The above algorithm is rather naive and the implementation that I’ve added to erldis is a bit different.

Hash Ring

In erldis when you create a ring each item or node is replicated on the ring multiple times, (128 by default in erldis), so that the hash ring is more evenly distributed. When a key is hashed, it is placed somewhere on the ring, and if you move clockwise the next node that is encountered is the one that will contain the key. If you are already at the end of the ring it just moves around the circle to the first node. Here is the Erlang representation of a ring with NumReplicas=2 with three nodes:

1
2
3
4
5
6
7
8
9
10
11

Eshell V5.8 (abort with ^G)
1> hash_ring:create_ring(["redis01", "redis02", "redis03"], 2).
{2,
[{"redis01",
545684121738837007922020352670259130010627269168},
{"redis03",615626879509609616059735092207848107514679530568},
{"redis02",831940194106586542005694020470691157762681039493},
{"redis02",862200733042270842852150771881884931000854854035},
{"redis01",976184092715215764651301117507107557316103934589},
{"redis03",
1324913878318290717698351186295127020927484244520}]}

The hash algorithm used above is the name of the node concatenated with the replica number. So if you look at the ring above, the first item, 545684121738837007922020352670259130010627269168 was generated by doing:

1
2
3

Eshell V5.8 (abort with ^G)
1> <<Int:160/unsigned-integer>> = crypto:sha("redis012"), Int.
545684121738837007922020352670259130010627269168

The hashed string is “redis01″ + 2 (which is the replica number). By knowing the algorithm for where a node lies on the ring, a client in any language can take advantage of the shards.

Adding/Removing nodes

The first thing I have to say about adding nodes is to not do it. Salvatore Sanfilippo wrote a great post about sharding Redis, and his recommendation is that re-hashing is difficult and instead you should create a large number of small Redis instances up front, that way you don’t have to add more nodes down the road.

For example, you might want to create 128 small Redis instances (multiple per box) and as you grow, just move those instances to their own dedicated (or larger) boxes. This lets you have a large amount of shards up front so you don’t have to worry about re-hashing for a hopefully long time.

And for removing nodes, failures etc,. try to use Redis’s replication and use some form of HA so that if a Redis instance dies, you have another one that is up to date to takeover (keepalived, Linux HA, etc,.).

Final thoughts

Redis has been fantastic and I’m really looking forward to finishing up on our Redis “cluster” and modifying our clients to support sharding. As for right now, I’m going to be looking into ways to migrate our existing Redis data into the shards in real-time and then “flipping the switch” on the clients.

MoosTrax for Blackberry — Looking for testers

Chris Moos — Wed, 11 Aug 2010 15:41:00 +0000

I am finally going to work on updating MoosTrax for BlackBerry — as there are a few outstanding bugs that need to be addressed. The most important bug I will be working on is consistency of location updates in newer, post-4.2 OS BlackBerry devices.

If you are interested in helping test the upcoming version, please e-mail moostraxsupport@gmail.com.

MoosTrax released for iPhone

Chris Moos — Fri, 30 Jul 2010 04:01:59 +0000

I am happy to say that MoosTrax has been approved by Apple and is now available for download in the App Store. Thanks to everyone who helped with the beta testing — it was a great help.

Note: MoosTrax works on devices running iOS 4+.

iPhone Support/Help
MoosTrax iTunes Link

MoosTrax for iPhone Beta Updates

Chris Moos — Tue, 13 Jul 2010 05:59:04 +0000

I’ve been working a lot on getting MoosTrax ready for the App Store and the latest beta adds a lot of great features.

View a list of all of your devices
View the current location of any device on your account.
Enhanced battery usage through the “Conserve Battery” feature.

The beta is still open, so send an e-mail to moostraxsupport@gmail.com to join.

MoosTrax for iPhone iOS4

Chris Moos — Mon, 28 Jun 2010 02:47:51 +0000

I just got a new iPhone 4 with iOS4…and it supports background location updates! I’ve decided to bring MoosTrax back to the iPhone.

I’m looking for beta testers…and then soon I’ll be pushing it to the App Store.

Applying for the beta

Please e-mail moostraxsupport@gmail.com with your iPhone’s UDID.

For information on finding the UDID, go here: http://www.geekology.co.za/blog/2009/07/how-to-check-your-iphones-udid-from-itunes/

Screenshots

AsyncRecord: Non-blocking database access for Ruby

Chris Moos — Tue, 22 Jun 2010 04:17:18 +0000

Two weeks ago I developed my first event-driven web framework for Ruby, Fastr. It helped me understand why running a web framework in an event loop is so natural.

As I continued to tackle more features in Fastr, it was time to tackle persistence — notably, database access.

AsyncRecord is/will be an ORM, similar to ActiveRecord — with one major difference — it doesn’t block. AsyncRecord currently uses em-mysql to access a MySQL database.

How it usually works

In most ORMs, when you attempt to access the database, everything in that thread will block until a response is received. This means that you waste time — just waiting. The CPU may be idle, but you cannot handle any more requests. (Typically you start multiple instances of your application to get past this, unfortunately each instance requires more resources on your server)

How AsyncRecord works

When you access something in the database with AsyncRecord, the request is sent to the database server, but control returns to the application immediately after the packet(s) are sent. When the server responds, which could be 20ms or 200ms later, a callback that you specify is invoked.

One important thing about accessing a database asynchronously, especially in web frameworks, is the ability to defer a response. Fastr has built-in support for deferred responses, a-la EventMachine/Thin.

A deferred response is when you tell the web server that you will send data to the client some time in the future, and the server is free to handle more requests until you are ready to respond.

Benchmarking

As I was implementing AsyncRecord, I knew it would be faster — but I wasn’t sure by how much. I setup a very simple Rails 2.3.5 application, as well as a Fastr application (from the latest source).

My goal was to make an application that has a single page, which shows 5 usernames from the database.

Rails controller:

1
2
3
4
5
6
7
8
9
10
11
12

class MainController < ApplicationController
def index
users = []

User.all(:limit => 5).each do |user|
users << user.username
end

headers['Content-Type'] = 'text/plain'
render(:text => users.join("\n"))
end
end

Fastr controller:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

class MainController < Fastr::Controller
def index
defer_response(200, {"Content-Type" => "text/plain"}) do |response|

User.all(:limit => 5) do |users|
users.each do |user|
response.send_data("#{user.username}\n")
end
response.succeed
end

end
end
end

The Numbers

Fastr

Average Latency: 123ms
Requests per second: 385 r/s

Rails

Average Latency: 2040ms
Requests per second: 42 r/s

The tests were performed using JMeter. 100 concurrent requests (10 requests per connection).

I also ran some tests using apache bench, here are the results:

1	ab -c 100 -n 1000 http://127.0.0.1:5000/

Fastr

Average Latency: 90ms
Requests per second: 1100 r/s

Rails

Average Latency: 2235ms
Requests per second: 44 r/s

Conclusion

I am extremely happy with what AsyncRecord can do — and I hope to make it even better. I will be moving it out of Fastr and into its own project soon.

Fastr GitHub: http://github.com/chrismoos/fastr

Fastr – A Web Framework for Ruby

Chris Moos — Wed, 09 Jun 2010 04:12:06 +0000

Foreword

Every month I go on a binge and learn something new. The most recent binge resulted in a new web framework, called fastr.

I’ve always used Rails when I needed to create a web application in Ruby, but despite how great it is to code in, the performance and concurrency is not up to par.

Don’t get me wrong — Rails can scale, but it is not inherently very good when it comes to an individual instance’s performance. While this is hopefully changing in Rails 3, I wanted to see what kind of concurrency and performance I could get out of a simple EventMachine web application. Thus, fastr was born.

Fastr’s Architecture

Fastr is nothing special. It is an extremely simple MVC style web framework, similar in configuration to Rails.

What makes it different though, is how it handles requests and responses.

The core of fastr runs on an EventMachine reactor, which is basically just a big event loop. EventMachine is very capable of maintaining thousands of connections in one instance. This is great when you want an application that supports Comet and chunked responses. There is very minimal overhead in having thousands of connections that have occasional data flowing through.

EventMachine and Fastr also support the ability to defer a response until a long running task finishes executing. This is nice because it won’t block the rest of the application. The only catch is you are limited to EventMachine’s thread pool size, which I believe defaults to 20. You can always up this number though, if you want.

Routing

The routing is very similar to Rails, an example routes.rb file looks like this:

1
2
3
4
5

router.draw do |route|
route.for '/:controller/:action'
#route.for '/home/:action', :action => '[A-Za-z]+'
#route.for '/test', :to => 'home#index'
end

Controllers

Controllers are simple, with just a list of actions that are called based on the route. Actions should return a valid Rack response. There are convenience methods in the controller that help in creating Rack responses.

The nice thing about the controllers in Fastr is that you can defer responses with ease. Here is an example of a deferred response:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

def long_running_task
defer_response(200, {"Content-Type" => "text/plain"}) do |response|
puts "in our deferred response...now we can do cool stuff!"
response.send_data("hey\n")

long_task = proc {
log.debug "Sleeping for 5 seconds...but this won't block other requests"
sleep(5)
log.debug "Finished sleeping, returning response to client."
return "finished"
}

callback = proc { |result|
log.debug "Callback result: #{result}"
response.send_data("#{result}\n")
response.succeed
}

response.task(long_task, callback)
end
end

The alternative render methods also exist, where you can render a HAML template or just text:

1 2	render(:text, "Hello, world!") render(:haml, :template => "index") # this searches for index.haml in your app/views/ folder

Future

For now, fastr is just a toy framework for me, that I hope to share with anyone interested in learning about building web frameworks.

I intend to start writing some tests as well as making fastr more robust. I don’t mean to displace any other Ruby web framework, but as I learn with fastr I hope to contribute what I learn back to projects like Rails and Sinatra.

Also, I’d like to get em-mysql or equivalent integrated so that you can really enjoy a event-driven web framework — event with database access.

Check out the GitHub page for more examples on getting fastr up and running.

GitHub: http://github.com/chrismoos/fastr

Unlocking the full potential of the iPhone

Chris Moos — Sun, 18 Apr 2010 04:43:48 +0000

Since Apple launched the iPhone in 2007 there has been an unprecedented growth in the mobile phone market. From new advances in hardware, such as touch screens, to operating systems mimicking the full power of a desktop computer. Never has a smartphone been able to penetrate the demographics of 10 year olds, to teenagers, or mom and dad — until the iPhone.

There are now hundreds of thousands of applications built for mobile phones, with every use case you can imagine. But even with the growth and advances of the past few years, the industry is still in its youth.

Apple, with its enormous success, is still constantly on the front page because of its questionable practice — sole control of the iPhone platform. Unlike desktop operating systems, such as Windows or Mac OS X, where the user is in complete control of the operating system, the iPhone is a jail, with Apple deciding what can come in and go out.

The App Store, the only way for a developer to target an iPhone, or a user to install an application on the iPhone, is in constant censorship mode. Certain API’s are not allowed and questionable content is put to rest.

It is time that Apple realizes that it is not only the hardware in the iPhone, or the software that powers it, that makes it a powerful force in the mobile phone market. The power is in the apps, the platform, and in the imaginations of developers.

Even though there are many applications for the iPhone, there could be so much more. The iPhone is a powerful device that can be used to provide a platform for mobile applications in every industry possible. Medical, industrial, retail — the list goes on. But as long as Apple is in control of what happens on the iPhone, developers will never be able to harness the full potential of such a beautiful device.

Apple now has the ability to turn the iPhone, and the new iPad, into a commodity device. The PC of the mobile world. My intuition tells me that Apple won’t let this happen. They prefer to maintain complete control of their platform, and while making it a premium and luxury item, they stop it from attaining the growth that it most definitely can achieve.

The future is in Apple’s hands — and they have the ability to make it an awesome place.