RedisConf 2019

Redis Labs presented another well-organized, strong technical conference this year. Once again it was held at Pier 27 in SF.

The conference theme this year was, “Redis, Database for the Instant Experience.”




Executive Summary:

  1. Redis Enterprise active-active (based on CRDTs) is being used in production already, which is ideal for HA distributed session caches, and more. RedisSearch active-active is now in preview. 2019 is the first time you can easily add multi-region support to your MySQL with Redis applications.
  2. Box, Inc. has developed a zero-downtime deploy and key rotation proxy for Open Source Redis. It was developed for IT compliance, and is being Open Sourced in April 2020. See diagram.



The Holy Grail: a geo-distributed application that uses active-active Redis


Redis Database Feature Groups

Conference Day One (Tuesday)

Keynotes

Lectures

Extreme Performance with Redis, Andi Gutmans and Kevin McGehee, AWS

– C5n 100 Gbps
– I3 16 Gbps SSD at 3.3 M iops
– R5d nitro coprocessor board
– Rdbtools tools for redis
– Main use cases: Strings and hashes with TTL
– Socket io contention with command parsing
– AWS added dedicated thread
– 50 to 83% faster
– setReadFrom nearest
– ConnectionPool
– Pipeline/execute
– Scan (paginated) not keys
– Async delete (unlink vs delete)
– Sz of collections
– Slowlog
– Client reply off/skip
– Upgrade
– Cluster mode allows scale-in and out with no downtime
– gutmans@
– McGehee@

Redis Security, Box

– Asked by Secops to solve the problem of securing passwords, but found password rotation to be a big problem
– Old and new password in Vault
– Old and new Redis proxy processes
– iptables to switch proxy process using SO_REUSE
– Drain old proxy
– RPX intelligence for different command groups: Harmless, auth, internal Redis, and other commands
– Zookeeper has list of authorized nodes especially for replication

– 20k new conn per host
– 15k concurrent conn per host
– 5 GBps per redis host
– 0.5 ms per req
– TLS being tested.

Patrick King, New Relic

– used mcrouter for about 5 years
– 7 Tbps bandwidth from monitoring agents
– 232 million agents, some last only 90 seconds
– First rails with Redis
– Mesos -> K8s
– Megabase is their custom persistence service
– 12M agents
– 20M wpm
– 12 M rpm
– Had to shard Sentinel
– consistent hashing can be brittle so use 2 layers to isolate config from app
– Demos of online auto redis-cli create and reshard

Work-stealing, Jim Nelson, Internet Archive

– Instead of monolithic cron jobs, use distributed workers
– Watch/exec with hash and array for field-level ttl
– Srandmember/srem
– http:s//github.com/internetarchive/work-stealing
– Ok for auditing, sampling, etc
– Not ok for sequential important results

How to do 1 million OPS with Redis, Jane Paek, Redis Labs jane@redislabs.com,

– typical warnings about slow operators like keys and delete
– memtier-benchmark
– redis-benchmark
– redis-benchmark -q script-load “redis.call(‘set’,’foo’,’bar’)”

Game with Redis, Python and Websockets

– IXWebSocket
– async
– no Boost needed
– auto-reconnect

– Python 3.5+ async IO
– MagicStack lib uvloop
– Nice web sockets module
Mypy static type checker
Tracemalloc module new in version 3.4

– Neo
zlib on web sockets saves 75% bandwidth

Conference Day Two (Wed.)

Keynotes

Lectures

CRDT’s by Roshan Kumar, Redis Labs

Good use cases:

– counters
– inventory
– sessions
– most else

Not good use cases:

– financial
– order processing
– only one client should pop data

Points to remember:

1. Your app will not receive notification when conflict occurs
2. You cannot override the default conflict resolution semantics
3. Lua script synchronizes commands

Best Practices:

1. Make your apps stateless (let Redis manage the state of your data)
2. Design your solution assuming conflicts may occur

Synchronize your clocks for best results. Note that some operations are time-independent, like adding to an array.

Github RedisCRDTDemo repo for network partitioning (split and restore) test scripts



Successful Redis active-active-active demo of updating a key-value using CRDTs across 3 regions: Atlanta, Boston and Chicago 🙂

RedisTimeSeries Module, Danni Moiseyev, Redis Labs

– introduces time series TS.* operators
– can insert timestamps with multiple labels
– can define aggregation labels
– compatible with Prometheus and Grafana
– benchmarks show RedisTimeSeries much faster than Prometheus and TimeScaleDB
– very high performance
– compression coming soon

Writing Redis Modules with Rust, Redis Labs

– memory-safe compared to C
– use macros
– see https://github.com/RedisLabsModules/redismodule-rs for sample code
– Rust compiles to a .so for loading like a normal library

RedisSearch Benchmarks and CRDTs, Redis Labs

– 50% to 400% faster than ElasticSearch because C/RESP vs. Java/HTTP
– able to do tens of thousands of indexes for multi-tenant applications, unlike ES which crashed at 912 indexes
– CRDT support in Redis Enterprise Preview
– Best Practice is not to use RedisSearch clusters with session caches, etc. to avoid key conflicts

Cache, Zohaib Hassan, Doordash

– random ahead of time eviction (time jitter) in multiple nodes (clusters)
– one way to avoid thundering herds: timestamp + ttl + (rand() * gap) > now()
– another way: p886-vattani.pdf
– L1 -> L2 -> DB
– See Java Caffeine cache for similar idea, but it might lock thread
– compress large values. lzbench said LZ4 was faster than Snappy for menu content (64 KB – 700 KB)
– prolly more reads than writes when caching
– 2x faster GETs with compression
– no spikes after pre-eviction + compression applied during busy periods now
– 15% extra RAM after compression
– Redis is great, but pay attention to the details
– Cloudflare uses Facebook’s zstd for message queues and is happy.

RedisConf 2019 Exhibit Area




Small exhibit area with AWS, GCP, Azure, Redis, Heimdall Data (proxy), Western Digital, RDBTools (bought by Redis Labs 2 days ago), Redis U.

Got a comprehensive demo of RDBTools. It’s an amazing web UI for Redis database. I particularly liked the slowlog report and online resharding. Redis Labs bought them 2 days ago, so it is now offered as part of their product lineup now for a small monthly fee.



Impressive Figure-8 RC car racetrack!

Several food trucks served lunch.

No walls between the 6 lecture tracks, so Silent Events headphones were provided to amplify regular speaker voices.

Note to Conference Organizers

If possible, please serve drinks on the 2nd floor next year.

Simple Developer Tutorial

$ brew install redis
$ redis-cli
> SETEX pages:about 5 "about us"
OK
> GET pages:about
"about us"
> KEYS *
1) "pages:about"
# wait 5 seconds
> GET pages:about
(nil)
> KEYS *
(nil)
> shutdown
> quit

To start Redis later:

brew services start redis

infoworld.com: When to use a CRDT-based database
Redis Labs Videos
Using the command line to check redis health

twitter.com: Improving Key Expiration in Redis

This entry was posted in API Programming, Cloud, Linux, Tech. Bookmark the permalink.

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.