StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POHow to debug/fix random occurring Redis::TimeoutError?
text
Body
copied!<p>I have a rails app running which is using redis quite a lot - however - I'm seeing quite a few <code>Redis::TimeoutError</code> occurring here and there, from time to time. There is no pattern in the circumstances. It occurs both in the web app and in the background jobs (which is being processed using <a href="https://github.com/mperham/sidekiq">sidekiq</a>) - not often but from time to time.</p> <p>Now I have no idea how to track down the root cause of this and hence no idea how to fix it. </p> <p><strong>Here is a little background on my setup:</strong></p> <p>The redis instance is running on a separate physical server which is connected to both my web server and background server in a private local 1Gbit network. All servers are running ubuntu 12.04. The redis version is 2.6.10. I'm connecting from my rails app (which is 3.2) using an initializer like so:</p> <pre><code>require 'redis' require 'redis/objects' REDIS = Redis.new(:url => APP_CONFIG['REDIS_URL']) Redis.current = REDIS </code></pre> <p>This is the output of <code>redis-cli INFO</code>:</p> <pre><code># Server redis_version:2.6.10 redis_git_sha1:00000000 redis_git_dirty:0 redis_mode:standalone os:Linux 3.2.0-38-generic x86_64 arch_bits:64 multiplexing_api:epoll gcc_version:4.6.3 process_id:28475 run_id:d89bbb1b81d3169c4228cf23c0988ae437d496a1 tcp_port:6379 uptime_in_seconds:14913365 uptime_in_days:172 lru_clock:1507056 # Clients connected_clients:233 client_longest_output_list:0 client_biggest_input_buf:0 blocked_clients:19 # Memory used_memory:801637360 used_memory_human:764.50M used_memory_rss:594706432 used_memory_peak:4295394784 used_memory_peak_human:4.00G used_memory_lua:31744 mem_fragmentation_ratio:0.74 mem_allocator:jemalloc-3.3.0 # Persistence loading:0 rdb_changes_since_last_save:23166 rdb_bgsave_in_progress:0 rdb_last_save_time:1378219310 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:4 rdb_current_bgsave_time_sec:-1 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok # Stats total_connections_received:932395 total_commands_processed:3088408103 instantaneous_ops_per_sec:837 rejected_connections:0 expired_keys:31428 evicted_keys:3007 keyspace_hits:124093049 keyspace_misses:53060192 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:17651 # Replication role:master connected_slaves:1 slave0:192.168.0.2,6379,online # CPU used_cpu_sys:54000.21 used_cpu_user:73692.52 used_cpu_sys_children:36229.79 used_cpu_user_children:420655.84 # Keyspace db0:keys=1498962,expires=1310 </code></pre> <p>In my redis config I have the following set:</p> <pre><code>\ﬁdaemonize yes pidfile /var/run/redis/redis-server.pid timeout 0 loglevel notice databases 1 save 900 1 save 300 10 save 60 10000 stop-writes-on-bgsave-error yes rdbcompression yes rdbchecksum yes dbfilename dump.rdb dir /var/lib/redis slave-serve-stale-data yes slave-read-only yes slave-priority 100 maxclients 1000 maxmemory 4GB maxmemory-policy volatile-lru appendonly no appendfsync everysec no-appendfsync-on-rewrite no auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 64mb lua-time-limit 5000 slowlog-log-slower-than 10000 slowlog-max-len 128 hash-max-ziplist-entries 512 hash-max-ziplist-value 64 list-max-ziplist-entries 512 list-max-ziplist-value 64 set-max-intset-entries 512 zset-max-ziplist-entries 128 zset-max-ziplist-value 64 activerehashing yes client-output-buffer-limit normal 0 0 0 client-output-buffer-limit slave 256mb 64mb 60 client-output-buffer-limit pubsub 32mb 8mb 60 </code></pre>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload