Houston, you there ?? We’ve had a problem here.
One of our EC2 instance which has Redis server on it, got hacked out of nowhere.
Just before we knew it is hacked, we were screwing up with Redis configuration and thinking what could go wrong with it.
Houston – Roger that, give us more details.
Well, we’ve recently shifted our Redis server to new EC2 instance. The reason we had to do that was because, our Sidekiq processing got much bigger and we couldn’t afford it alongside the Nginx + Passenger. So we took a call to separate it out.
But while configuring Redis, we think we made some mistake :(.
Houston – What is that ?
Basically, we wanted the Redis to listen to all of our Passenger instances . Though by default, it listens on localhost because it’s
bind directive is set to
127.0.0.1 , it’s possible to listen on multiple interfaces by providing multiple IP addresses like,
bind 192.168.1.100 10.0.0.1
According to that, we set IP addresses of our Passenger instance(s) to listen from. But Redis started complaining about it by saying,
bind: Cannot assign requested address
Why is that Houston ??
Houston – Hmm, I pretend to not read that. What next ?
Seriously Houston ?? 😐 (Actually, I didn’t find any convincing answer / way to overcome this. If you know it, please spread your knowledge through comments !)
Another way to listen on multiple connections is by commenting out
bind configuration or setting it to
0.0.0.0. So, we did it and everything worked out nicely after that.
Since few days, Redis has started to act weirdly. I mean, our data has started getting lost at any certain point of time. All of our scheduled or enqueued Sidekiq jobs are getting cleared abruptly !!
Houston – What is Redis working directory’s location ?
We’ve set it to
But, if we run
$ redis-cli config get dir in terminal, it’s giving
/tmp. WTH !
Houston – Roger, it seems to be a big trouble (which might get double 😮).
(Houston continued …) It appears to us that, by setting
bind directive to
0.0.0.0, you allowed the whole world talk to your Redis server.
It’s obviously not secured, as anyone can issue a
CONFIG SET dir directive to it (or any other command for that matter). If setting
bind directive to some specific IP address isn’t working, then at least update your firewall to allow only those IP addresses or set password to Redis.
You can set password to Redis using
Affirmative Houston, that explains it.
So if we block whole world talk to Redis except given IP, how can we test it (or hack it 😏) ?
Houston – Behold (but at your own risk).
You can login to Redis console by,
$ redis-cli -h IP_ADDRESS -a PASSWORD
Or run any Lua script like,
$ redis-cli -h IP_ADDRESS -a PASSWORD --eval "some Lua script"
If you know EC2’s IP address ranges, think of the possibilities you got (you know what I mean 🙂 ).