Las Vegas 2019 Trip

While on vacation in Las Vegas last week, I saw the record rainfall. I had just entered the hotel, and noticed blackening cumulonimbus clouds over the hills – within 30 minutes torrential rains lashed the hotel windows.

Ultimately a record 0.25 to 1 inch of rain and hail fell.

Torrential rain and hail arriving in Las Vegas. The sky view from downtown Las Vegas was almost all black.

I saw the Blue Man Group, Hershey’s Chocolate World and went on a once-in-a-lifetime tour of the Hoover Dam.

I booked the tour through Grayline Tours, and saw the following:

  1. Ethel M/Mars Chocolate Factory and Botanical Cactus Gardens
  2. Lake Mead “Desert Princess” boat tour
  3. Hoover Dam tour inside (conduits and generators)
  4. Hoover Dam tour from top

The Hoover Dam took 5,000 workers and 5 years to build in the middle of the Depression.

It was the largest construction in history at the time, and still looks like it was built yesterday.

The dam was built primarily for water resource management, with the electrical generation a bonus. The electrical production paid off the dam project by 1986.

Hoover Dam – view from “Desert Princess” boat on Lake Mead

Hoover Dam – 7×130 MW Generators (~1 GW per side totaling 2 GW, supplying half of region’s electrical needs)

Hoover Dam – view from top of dam towards Colorado River showing concrete bridge, which took 9 years to build

W: Hoover Dam

Posted in Tech | Leave a comment

Buckethead Shrine

Buckethead is the greatest electric guitarist and composer of all time.

I became a Buckethead in 2019 – it’s never too late! 🙂

Buckethead Shreds:

Buckethead Ambients:

Worthy “Soothsayer” Covers:

Buckethead Disciple on YT:

“He was born and raised amongst chicken. His outfit is his way of protesting against poultry slaughter and he declared that the sound of his guitar was really just the screams of his dying brothers (again, the chicken).”

W: Buckethead
Photos of Buckethood and Hendrix’s Pinky Fingers

Posted in Tech | Leave a comment

Getting Started in Computer Programming

From time to time, people ask me how to get started in the career of computer programming.

If you like solving puzzles and have the ability to focus for hours at a time, then you’re a good candidate to enjoy a programming career. Many programmers are introverts, so if that describes you, great! 🙂

First ask your friends, co-workers or hosting provider which computer language they use. If they don’t have an opinion, read on.

For writing scripts (small programs), the 3 main languages are:

Language Tutorial Book
Perl Tutorial “Learning Perl”, Randal Schwartz, O’Reilly
Python Tutorial Books
PHP (mostly for web programming) Tutorial Books

Perl has some advantages in that older sample scripts were usually written in Perl, and it’s an extremely powerful and compact language. Perl is popular with people who want the biggest hammer available for solving problems, or who want to specialize in security engineering scripting. 🙂

On job sites, Python is currently the most popular scripting language. It also has good Machine Learning support. Python is popular with people who have OCD or like to “follow the rules” with its indentation rules. Google uses python or Java for most programming.

PHP is popular with people who mostly want to do web programming, and has the widest hosting provider (ISP) support. Besides WordPress, Facebook also uses PHP.

After learning simple programming, build a web site for a CD collection inventory to learn basic HTML, CSS and database programming with MySQL.

Posted in Tech | Leave a comment

International Committee To Review Boeing 737 MAX

Southwest Airlines Boeing 737 MAX’s stranded like seals

Wow, you don’t often see two organizations destroy their brand globally at the same time – way to go, Boeing and FAA!

The 737 was the best-selling airliner in history, but not a single one has been ordered since the Ethiopian Airlines crash.

My prediction is that Boeing CEO, Dennis Muilenburg, will be fired over this, and then everybody will be asking, “What took so long?”

As the European Aviation Safety Agency (EASA) has reminded Boeing and the FAA, it will end its grounding “only once there is complete reassurance that it is safe” since their safety certifications are now worthless.

“On Wednesday, Canadian Transport Minister Marc Garneau [a former astronaut] suggested that while no final decision had been reached, simulator training could be required for MAX pilots in Canada regardless of the FAA’s approach. “Simulators are the very best way from a training point of view to go over exactly what could happen in a real way and to react properly to it,” said Garneau. “It’s not going to be a question of pulling out an iPad and spending an hour on it.”

Stay tuned for the ongoing DOJ criminal investigation … International Committee To Review MAX
‘Passengers are afraid of this airplane’: How Boeing is handling its 737 Max problem FAA Convenes Advisory Board To Vet 737 MAX Updates (without Boeing)
Boeing Crash Fight Will Set Price on Victims’ Minutes of Terror
Europe’s Pilots Censure ‘Delegated Certification’ of 737 Max
Maintenance Ops Face Unsung Effect From Max Grounding

Posted in Tech | Leave a comment

RedisConf 2019

Redis Labs presented another well-organized, strong technical conference this year. Once again it was held at Pier 27 in SF.

The conference theme this year was, “Redis, Database for the Instant Experience.”

Executive Summary:

  1. Redis Enterprise active-active (based on CRDTs) is being used in production already, which is ideal for HA distributed session caches, and more. RedisSearch active-active is now in preview. 2019 is the first time you can easily add multi-region support to your MySQL with Redis applications.
  2. Box, Inc. has developed a zero-downtime deploy and key rotation proxy for Open Source Redis. It was developed for IT compliance, and is being Open Sourced in April 2020. See diagram.

The Holy Grail: a geo-distributed application that uses active-active-active Redis using a counter

Redis Database Feature Groups

Conference Day One (Tuesday)



Extreme Performance with Redis, Andi Gutmans and Kevin McGehee, AWS

– C5n 100 Gbps
– I3 16 Gbps SSD at 3.3 M iops
– R5d nitro coprocessor board
– Rdbtools tools for redis
– Main use cases: Strings and hashes with TTL
– Socket io contention with command parsing
– AWS added dedicated thread
– 50 to 83% faster
– setReadFrom nearest
– ConnectionPool
– Pipeline/execute
– Scan (paginated) not keys
– Async delete (unlink vs delete)
– Sz of collections
– Slowlog
– Client reply off/skip
– Upgrade
– Cluster mode allows scale-in and out with no downtime
– gutmans@
– McGehee@

Redis Security, Box

– Asked by Secops to solve the problem of securing passwords, but found password rotation to be a big problem
– Old and new password in Vault
– Old and new Redis proxy processes
– iptables to switch proxy process using SO_REUSE
– Drain old proxy
– RPX intelligence for different command groups: Harmless, auth, internal Redis, and other commands
– Zookeeper has list of authorized nodes especially for replication

– 20k new conn per host
– 15k concurrent conn per host
– 5 GBps per redis host
– 0.5 ms per req
– TLS being tested.

Patrick King, New Relic

– used Facebook’s memcached mcrouter for about 5 years
– 7 Tbps bandwidth from monitoring agents
– 232 million agents, some live only 90 seconds
– First rails with Redis
– Mesos -> K8s
– Megabase is their custom persistence service
– 12M agents
– 20M wpm
– 12 M rpm
– Had to shard Sentinel
– consistent hashing can be brittle so use 2 layers to isolate config from app
– Demos of online auto redis-cli create and reshard

Work-stealing, Jim Nelson, Internet Archive

– Instead of monolithic cron jobs, use distributed workers
– Watch/exec with hash and array for field-level ttl
– Srandmember/srem
– http:s//
– Ok for auditing, sampling, etc
– Not ok for sequential important results

How to do 1 million OPS with Redis, Jane Paek, Redis Labs,

– typical warnings about slow operators like keys and delete
– memtier-benchmark
– redis-benchmark
– redis-benchmark -q script-load “‘set’,’foo’,’bar’)”

Game with Redis, Python and Websockets

– IXWebSocket
– async
– no Boost needed
– auto-reconnect

– Python 3.5+ async IO
– MagicStack lib uvloop
– Nice web sockets module
Mypy static type checker
Tracemalloc module new in version 3.4

– Neo
zlib on web sockets saves 75% bandwidth

Conference Day Two (Wed.)



CRDT’s by Roshan Kumar, Redis Labs

Good use cases:

– counters
– inventory
– sessions
– most else

Not good use cases:

– financial
– order processing
– only one client should pop data

Points to remember:

1. Your app will not receive notification when conflict occurs
2. You cannot override the default conflict resolution semantics
3. Lua script synchronizes commands

Best Practices:

1. Make your apps stateless (let Redis manage the state of your data)
2. Design your solution assuming conflicts may occur

Synchronize your clocks for best results. Note that some operations are time-independent, like adding to an array.

Github RedisCRDTDemo repo for network partitioning (split and restore) test scripts

Successful Redis active-active-active live demo updating a key-value using CRDTs across 3 regions: Atlanta, Boston and Chicago 🙂

RedisTimeSeries Module, Danni Moiseyev, Redis Labs

– introduces time series TS.* operators
– can insert timestamps with multiple labels
– can define aggregation labels
– compatible with Prometheus and Grafana
– benchmarks show RedisTimeSeries much faster than Prometheus and TimeScaleDB
– very high performance
– compression coming soon

Writing Redis Modules with Rust, Redis Labs

– memory-safe compared to C
– use macros
– see for sample code
– Rust compiles to a .so for loading like a normal library

RedisSearch Benchmarks and CRDTs, Redis Labs

– 50% to 400% faster than ElasticSearch because C/RESP vs. Java/HTTP
– able to do tens of thousands of indexes for multi-tenant applications, unlike ES which crashed at 912 indexes
– CRDT support in Redis Enterprise Preview
– Best Practice is not to use RedisSearch clusters with session caches, etc. to avoid key conflicts

Cache, Zohaib Hassan, Doordash

– random ahead of time eviction (time jitter) in multiple nodes (clusters)
– one way to avoid thundering herds: timestamp + ttl + (rand() * gap) > now()
– another way: p886-vattani.pdf
– L1 -> L2 -> DB
– See Java Caffeine cache for similar idea, but it might lock thread
– compress large values. lzbench said LZ4 was faster than Google’s Snappy (previously known as Zippy) for menu content (64 KB – 700 KB)
– prolly more reads than writes when caching
– 2x faster GETs with compression
– no spikes after pre-eviction + compression applied during busy periods now
– 15% extra RAM after compression
– Redis is great, but pay attention to the details
– Cloudflare uses Facebook’s zstd for message queues and is happy.

RedisConf 2019 Exhibit Area

Small exhibit area with AWS, GCP, Azure, Redis, Heimdall Data (proxy), Western Digital, RDBTools (bought by Redis Labs 2 days ago), Redis U.

Got a comprehensive demo of RDBTools. It’s an amazing web UI for Redis database. I particularly liked the slowlog report and online resharding. Redis Labs bought them 2 days ago, so it is now offered as part of their product lineup now for a small monthly fee.

Impressive Figure-8 RC car racetrack!

Several food trucks served lunch.

No walls between the 6 lecture tracks, so Silent Events headphones were provided to amplify regular speaker voices.

Note to Conference Organizers

If possible, please serve drinks on the 2nd floor next year.

Simple Developer Tutorial

$ brew install redis
$ redis-cli
> SETEX pages:about 5 "about us"
> GET pages:about
"about us"
> KEYS *
1) "pages:about"
# wait 5 seconds
> GET pages:about
> KEYS *
> shutdown
> quit

To start Redis later:

brew services start redis When to use a CRDT-based database
Redis Labs Videos
Using the command line to check redis health Improving Key Expiration in Redis Beating round-trip latency with Redis pipelining redis-cli, the Redis command line interface

Posted in API Programming, Cloud, Linux, Tech | Leave a comment