Houston, we got attacked

Houston, you there ?? We’ve had a problem here.

One of our EC2 instance which has Redis server on it, got hacked out of nowhere.
Just before we knew it is hacked, we were screwing up with Redis configuration and thinking what could go wrong with it.

Houston – Roger that, give us more details.

Well, we’ve recently shifted our Redis server to new EC2 instance. The reason we had to do that was because, our Sidekiq processing got much bigger and we couldn’t afford it alongside the Nginx + Passenger. So we took a call to separate it out.

But while configuring Redis, we think we made some mistake :(.

Houston – What is that ?

Basically, we wanted the Redis to listen to all of our Passenger instances . Though by default, it listens on localhost because it’s  bind directive is set to  127.0.0.1 , it’s possible to listen on multiple interfaces by providing multiple IP addresses like,

bind 192.168.1.100  10.0.0.1

According to that, we set IP addresses of our Passenger instance(s) to listen from. But Redis started complaining about it by saying,

bind: Cannot assign requested address

Why is that Houston ??

Houston – Hmm, I pretend to not read that. What next ?

Seriously Houston ?? 😐 (Actually, I didn’t find any convincing answer / way to overcome this. If you know it, please spread your knowledge through comments !)

Another way to listen on multiple connections is by commenting out bind configuration or setting it to 0.0.0.0. So, we did it and everything worked out nicely after that.

Since few days, Redis has started to act weirdly. I mean, our data has started getting lost at any certain point of time. All of our scheduled or enqueued Sidekiq jobs are getting cleared abruptly !!

Houston – What is Redis working directory’s location ?

We’ve set it to /var/lib/redis/default/ in redis.conf using dir directive.

But, if we run $ redis-cli config get dir in terminal, it’s giving /tmp. WTH !

Houston – Roger, it seems to be a big trouble (which might get double 😮).

(Houston continued …)  It appears to us that, by setting bind directive to 0.0.0.0, you allowed the whole world talk to your Redis server.

It’s obviously not secured, as anyone can issue a CONFIG SET dir directive to it (or any other command for that matter). If setting bind directive to some specific IP address isn’t working, then at least update your firewall to allow only those IP addresses or set password to Redis.

You can set password to Redis using requirepass directive.

Affirmative Houston, that explains it.

So if we block whole world talk to Redis except given IP, how can we test it (or hack it 😏) ?

Houston – Behold (but at your own risk).

You can login to Redis console by,

$ redis-cli -h IP_ADDRESS -a PASSWORD

Or run any Lua script like,

$ redis-cli -h IP_ADDRESS -a PASSWORD --eval "some Lua script"

If you know EC2’s IP address ranges, think of the possibilities you got (you know what I mean 🙂 ).

That’s it. Roger, out. Happy hacking !!

Advertisements

One comment

  1. Gautam Rege · August 8, 2016

    Reblogged this on Josh Software – Where Programming is an Art! and commented:
    Yogesh talks about configuring and securing your Redis server to avoid attacks. Voice of experience rules!

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s