My new github repo, mysql-trigger-logger, demonstrates how to use MySQL triggers to log the timestamp, user, SQL and a note for unexpected database changes (“heisenbugs”) from UPDATE or INSERT statements to a logging table according to custom logic.
This is professional-grade code with documentation and tested notification scripts in Perl, Python and bash, and goes beyond what is available in StackOverflow answers.
github: mysql-trigger-logger
Nice article on what’s new in DNS in the past 2 years – not much apparently, according to Cricket Liu and ThousandEyes.
Also check out the new DNS book from Mark Jeftovic, the founder and CEO of EasyDNS.com.
When deploying a production change, usually you have a rollback procedure – documented even! 🙂
But sometimes after a deploy, things don’t work exactly as expected. At that time, you need to decide which is better: rollback, or fail forward?
If it’s a small, isolated code change and you can operate with the old version, often it’s an easy decision to rollback.
But in more complex situations, sometimes it’s better to accept that the change wasn’t perfect, but:
and keep the change and fail forward instead of doing a rollback.
Generally to know if failing forward is an option, you need:
Some actual examples of when I have failed forward successfully:
Some actual examples of when fail forward was not acceptable, and rollback was required:
Especially with databases, the arrow of time cannot be reversed. So database restores results in the loss of data on busy systems, making fail forward the default policy at many SaaS companies. Additionally, failing forward helps with development velocity.
I recently created a new minimal build system, called pico-build, that is hosted on github.
pico-build is the world’s smallest, yet featureful, three-environment (dev, stage and prod) build and deploy system.
It uses make, but in a different way than normal – pico-build manages entire environments, not individual files.
$ make
usage: make [help|check|dev|stage|prod|dist|all]
It is intended for individual programmers and small teams who want to test and deploy to three environments, but don’t want to spend time setting up (and patching) Jenkins or building a CI/CD pipeline.
$ make dev
$ make stage
$ make prod
$ make all
After using Github for several years, I recently used Gitlab for a solo developer project.
Some of the differences I noticed over the past two weeks were:
1. After initial signup, I got a 422 error HTTP response for about 5 minutes. I assume my new login was propagating to their caching layer(s). See screenshot below.
2. By default, the master branch on repos is “protected” and a maintainer must remove that for regular users to push to.
3. Gitlab markdown is different than Github’s.
4. No tagging of repos with the probable computer language. (seems to work for me as of Nov. 13, 2018.)
5. When sharing a private project, the “Guest” role cannot view source code.
I hope this helps programmers transitioning from Github to Gitlab. Please leave a comment below if you encounter other differences.
PSA: If you host domains with Godaddy, and you’re a day or two late in renewing a domain, it will be “parked” – in other words, assigned a different IP address than the previous one.
This is as bad as it sounds, since your site availability and SEO rank will be hurt as various crawlers will cache the random parking pages for varying periods of time, which can last for weeks.
To remove domain parking so that Internet users can see your site again:
Otherwise, contact Godaddy support ASAP.
If you need quick temporary access for intranet/personal use, just add a hosts entry pointing to the domains and sudbdomains you need and restart your browser. Your site will work perfectly, but just for you.
StackOverflow has a solution to the selinux problems encountered by some CentOS 7.4 and 7.5 users related to D-Bus errors.
SELinux corrupted? Now unable to boot CentOS 7 with SELinux enabled
Honda also worked on the main complaint of the first version, cabin noise, which was about 3 db higher (double) than desired for passengers.
$4.9 million is an interesting price point. You have to be a billionaire to afford a Gulfstream, but many successful business owners can handle a new HondaJet or a used Hawker lease, if they want full-size cabin space.
HondaJet Elite is Honda’s Newest Light Jet
HondaJet HA-420 : Buyer’s and Investor’s Guide (2013, but interesting)
HondaJet Flight Demo
[FullHD] Private Honda HA-420 HondaJet takeoff at Geneva/GVA/LSGG
I attended the Facebook Production Engineering Open House at their Menlo Park HQ.
“Production Engineering is Facebook’s secret sauce – it draws on multiple disciplines (Software Engineering, Systems Administration, Distributed Systems and Networking) to plan, build and maintain the massive Facebook infrastructure.”
It’s always interesting to see how large operators solve problems at scale. Even if you’re small, usually one can borrow one or two nuggets.
6:00 – 7:00: Registration, apps and drinks
7:00 – 7:30: “Welcome and PE overview” by Fernanda Weiden (OG Ads Team Member) Wikipedia
Production Engineering (PE) is Facebook’s term for Devops. PE’s must know how to program, but don’t necessarily end up doing so. Newer services may require more programming from PE’s, and older services less.
Tech Talks:
7:30 – 7:50: “Scaling Instagram On-call” by Nick Shortway
– took a long time to refine on-call schedule
– 1-day on-calls were too much administrative effort to schedule
– now newbies are Level 1 and experienced people are Level 2
– “loop” is a period of time
– 3-day loops where the #1 priority Is being on-call
– Level 1 escalates to Level 2 within 2-3 minutes if no apparent solution
– Level 1 might not get sleep for 3 days
– 3 day loops are a lot of effort to administer, but manageable.
7:50 – 8:10: “How we monitor and scale FB” by Patrick Taylor
– lsof, strace, nm, /proc/pid/exe and trace.py
– cubism (folded perf graphs) click through to “Deep Dive” graphs
– region, data center, cluster, rack, server
– examples with retransmit, cluster cache failure
Patrick/Facebook has developed a “Maszlow Hierarchy of Needs for PE”, similar to this one.
Facebook uses a style of time series visualization called a cubism or horizon graph, first implemented in D3 by Mike Bostock at Square in 2012.
8:10 – 8:30: “Supporting Global Events in FB Live” by Peter Knowles (10-year employee)
– 3 methods of load testing:
8:30 – 9:00 Q&A Speaker Panel and Mingle
Q: If you run out of capacity, do you prioritize ads or user platform up?
A: Ad systems are lower priority than user platform.
Q: How does Facebook support multiple development languages?
A: Thrift. Longer answer is developers should use “officially supported” languages, but they can use anything they want as long as they write their own Thrift client library.
Q: When is Facebook moving to the Public Cloud?
A: Unnecessary. (Editor: FB is the Cloud.)
Q: What was your longest full outage?
A: 36 minutes or so about 18 months ago. (Speaker said he wrote the memo, so he won’t forget the number of minutes.)
Q: Are you using AI/NLP in monitoring?
A: Not yet, but something we follow.
There were some exhibits, with one item apparently being an OpenCompute server.
Food was chicken sliders, tomato brusciatta, triangle-turnovers and smaller appetizers with an open bar.
Thanks to Facebook for hosting the event.
code.facebook.com: How production engineers support global events on Facebook, PE Blog
Cubism.js Time Series Visualization Slides
r-bloggers.com: Cubism Horizon Charts in R
acm.org: Sizing the Horizon: The Effects of Chart Size and Layering on the Graphical Perception of Time Series Visualizations
tableau.com: Horizon Chart Workarounds in Tableau
PSA: I haven’t seen a solution to this issue online, so posting it here for search engines to index.
PostgreSQL is an ACID-compliant database, but filesystem corruption can still prevent it from starting.
In my case, a linux test instance running in a Virtualbox VM was not cleanly stopped after a power failure.
The result was that data/pg_stat_tmp/ was corrupted, and an invalid replication checkpoint value was written, resulting in PostgreSQL refusing to start.
If you see the following error in data/log/:
2018-05-19 06:18:38.437 UTC [2647] PANIC: replication checkpoint has wrong magic 1767992667 instead of 307747550
2018-05-19 06:18:38.446 UTC [2595] LOG: startup process (PID 2647) was terminated by signal 6: Aborted
2018-05-19 06:18:38.446 UTC [2595] LOG: aborting startup due to startup process failure
2018-05-19 06:18:38.448 UTC [2595] LOG: database system is shut down
The solution after you fix any filesystem problems (if you don’t have a slave):
Although you can fix (“monkey-patch”) missing or corrupted/indexes like the following with REINDEX INDEX idx_name, you may just want to re-install PostgreSQL from backup at this point:
2018-05-19 17:30:18.052 UTC [11795] ERROR: index "idx_16573_primary" contains unexpected zero page at block 1733
2018-05-19 17:30:18.052 UTC [11795] HINT: Please REINDEX it.
If you have a slave, you will likely have to:
PostgreSQL Manual: 19.6. Replication
Kernel: disabling IRQ #19 (network card)
When is a load balancer not really a load balancer?
When it’s an AWS Elastic Load Balancer (Classic ELB or ALB.)
Although it’s been well-documented for at least 6 years, it’s still not well-known that one side of a Classic ELB (or ALB) does load balancing, but the other side doesn’t have a VIP, or static IP address, which most users expect.
The problems with not having a static VIP are:
Additional problems with ELBs are:
By now, you’re likely horrified as you see your 5×9’s rapidly disappear in the rear-view mirror. 🙂
For most users, the lack of that static IP disqualifies Classic ELBs and ALBs from any HA architecture design.
Solutions if your client program is expecting a static VIP are:
Lame workarounds:
Non-solutions:
AWS High Availability Patterns : DNS Load Balancing Tier
AWS ELB FAQs
GCE: Network Load Balancing (uses a static IP as expected)
I attended RedisConf 2018 on Thursday at Pier 27 (The James R. Herman Cruise Terminal) in SF.
Just like 2017, the talks were extremely high-quality and imaginative – I wished I could see all of the tracks.
The Herman Center is a beautiful new (2012) venue with breathtaking views of Golden Gate bridge, Alcatraz and the downtown skyline.
Keynotes
Some of the talks I attended:
Thursday Talks
Techniques for Synchronizing In-Memory Caches with Redis
Ben Malec, Paylocity
“For some highly-accessed value, a network roundtrip incurs too much latency. An obvious solution would be adding an in-memory caching layer, but that brings many challenges around keeping data in sync across multiple clients. This presentation will detail the approach Paylocity implemented, which leverages Redis Pub/Sub, bucketing keys to minimize synchronization message length, and carefully exploiting order-of-operation to eliminate the need for a master synchronization clock.”
– co-worker suggested multilevel cache design. In past times, that was overly complex. now, time to reconsider.
– data Source of Truth (SoT) is MS SQL Server
– multilevel cache with Redis as the cache SoT (still has TTLs) and pub/sub to 50 local clients
– .NET (Windows) using default MS memory cache on clients
#1 Possible Multilevel Cache Design
– broadcast keys and values
– will blow up network
#2 Possible Multilevel Cache Design
– broadcast just keys
– can still blow up network
#3 Possible Multilevel Cache Design
– broadcast 16-bit custom hash slot generated on client nodes
– store last updated array in RAM on client nodes and use lazily
– we will explore this option (see diagram)
– various race conditions to think about though when requiring local cache to always be correct despite various latencies and TTLs
– Redis lets a database application exploit Redis’ O(1) data structures, which RDBMS’ cannot match. Note that Publish is O(N_subscribers) and does not guarantee delivery (James)
– the slot hash starts to look like a zero-knowledge proof – interesting area to research (James)
– can decrement timestamp by 1 to know the newly arriving Redis event is later. (Some people postfix with -1, -2, etc.)
– can ignore self pub/sub messages:
RedisMultilevelCache/MultilevelCacheProvider.cs:122
if (dataSyncMessage.senderInstanceId == _instanceId)
{
return;
}
– can write Lua script on Redis side to send the pub/sub event and save sending 1 event over network
– possibility of cache thrashing in this design since hash slot is a range of keys with 16-bit ID’s, not a concern in practice
– but 18-byte (instance guid and slot int) messages (very small)
Future
– publish hit/miss metrics into ELK or Redis time-series module maybe
– possibly use Redis XFETCH to optimize cache reload
– add support for more Redis types
– track hot keys
– StackExchange also doing similar multilevel cache design.
– 45x faster with local cache than accessing Redis over network, approaching zero bandwidth
– also client could tell Redis a TTL and not send a pub/sub message!
– look at Redis’ key notifications (Salvatore commented on that)
– Salvatore: “We could spend a month talking about how to incorporate this into Redis.” 🙂
Bandwidth Optimizations
– delete notifications from clients to Redis are slot hash values (that represent a range)
– update notifications to clients are slot hash values that are cached in lastupdate array. This array is consulted before using a value in case it needs to be re-fetched (lazily.)
Latency Optimizations
– slot hash values, not key:values (Redis keys can be 512 MB)
– local lastupdate array, no network check
Code Optimizations
– using MS Cache and Redis pub/sub
– only need 1 page of code to implement multilevel cache, easier to verify correctness and/or test
– short code can be customized per use case
Reception
– idea for pub/sub came in shower
– implemented within one data center
– but could be useful for reducing latency in geo-distributed databases as Redis multi-master goes GA (James)
– IoT by former embedded engineer intended for avocado farmers
– 22 sprinkler wires plus PLC for $300
– tell field-hand where to find sprinkler damage (3 Gallons/Minute)
– gopher can chew on sprinkler, cracks can be non-visible (0.5 Gallons/Minute)
– started with typical embedded programming/web solution with Mongoose web server
– but needed to share data between 2 processes …
– Raspberry Pi with Redis is job queue controller, python apps talk to it
– need to schedule watering, collect info for flow and pump problems, log
– “I have the smallest database at the conference: 40 MB.” (“Power of small data!”)
– then 3 processes, graph database module needed
– Reference by Chinese Electric Utility to IEC1970 standard for SCADA, studied it
– SNMP on steroids
– not using Grafana now, but if he did, would do with monitoring as code
– some graph nodes are weather stations
– 2 types of data structures: system graph/logs and irrigation schedules
– code generates Redis keys to ensure keys are managed properly
– evaporative loss and moisture loss calculations
– Deep Learning and ML to interpret graphs, trends
– “You’re not going to run Tensor Flow on a Raspberry Pi.”
– Method of Synchronization between Cloud and Edge with AMQP
– adoption limited by less sophisticated neighbors and commercial SCADA interests
– cameras are for security: avocado theft
– github
Integrating Redis with ElasticSearch to Get the Best Out of Both Technologies
Dmitry Polyakovsky, Zumobi
ft.aggregate API
– can we count this? faceted search in Redis
– insert item and index in 1 ms
– aggregate vs. search
Search
– top N
– no processing involved
Aggregate
– top surname and count
– count StackOverflow questions by database by month
– filter -> group -> apply -> sort -> add more (all Redis types)
– country -> age -> profession -> languages
– numeric functions and expressions
– distributed: naive uses window function, still too much bandwidth and time
– distributed: better uses coordinator and does aggregates only (1000x less data)
– hyperloglog maintains 4% precision even after coordinator merge
– reservoir sample -> median
– query plan translator has 2 parts: remote query and local query
Limitations
– single GROUP BY advisable for good performance
– high number of groups – still slow
– exact COUNT DISTINCT and quantiles slow
– for huge parallel workloads, use Spark, etc.
Demo
– live sub-second searches of multi-million row StackOverflow data with AWS one-server 15-shard demo server
– APPLY is used last, so efficient on TOP N queries
– redash-client, plus custom changes
Future
– port all existing Redis simple search functionality
– streaming time-window searches
Conference Closing Session
– Open bar with bar snacks
– prizes giveaway
– talked to some of the presenters.
The James R. Herman Cruise Terminal
Though beautiful, it has some limitations as a conference venue, being split on 2 floors with occasional wind rattling. Parking is also limited.
Booth staff wore parkas and were still unbearably cold. Coffee was only available on the first floor, but talks were on the second floor – rather inconvenient, but easy to fix for next time.
The “F” streetcar stops outside.
Scaling a High-traffic Rate Limiting Stack With Redis Cluster
Multilevel cache system for Java 8
I attended the Percona Live 2018 database conference in Santa Clara again.
As always, the conference was very well-organized and had great talks.
For 2018, Percona attempted to make it more affordable, with prices in the $600 range. Much appreciated!
Executive Summary
I mainly attended the PostgreSQL track this year. The presenters were experts – either PostgreSQL server developers, professional trainers or AWS RDS/Aurora staff.
Tuesday Keynotes
Linux Performance 2018
Brendan Gregg, Netflix
Important to DBA
– BPF support added to perf
– BPF acts as a “sandbox” for monitoring agents
– Google’s BBR TCP algorithm 3x performance with 1% packet loss (hired Van Jacobsen)
– Facebook’s Kyber block reduce 99% latency by 300x
– Slides
Lectures
Real-time with Redis
Jon Hyman, Braze (formerly AppBoy)
– add jitter to timestamp to fix thundering herd
– 15 job queues
– SADD 7 GB arrays overwhelm replication slaves
– CPU utilization can stop replication
– API aggregation of keys
– fine-grained for user flexibility and triggered event use
– 120-140 redis servers at 40,000 ops/seconds
– expensive for replication
MySQL HA
– ProxySQL can help with 1 minute RDS failover and nearly-immediate Aurora failover
– ProxySQL does not forward the client IP yet (bug filed)
– ProxySQL is a good solution for filtering reads to slaves
Tuning Postgresql for High Write Workloads
Grant McAlister – Sr. Principal Engineer – Amazon RDS
– WAL compression will help reduce full page writes
– testing with random data may not provide useful results with WAL
– max_wal_size = 16GB, but increases recovery time
– checkpoint each minute
– async with random data maybe slower (code path not optimized for this workload)
– extra indexes in pg are expensive – hundreds or thousands of %
– prefix uuid with date to update btree on right (right lean)
– run vacuum to maintain perf (HOT and block cache) and avoid TID wraparound
– vacuum in memory:
– increase checkpoint_timeout
– alter table X set
– in memory before checkpoint 3.5 seconds
– In memory after checkpoint 84.5 seconds
– Vacuum not in memory 165.8 seconds
– Aurora: no checkpoints, no FWP, no log buffer
– not san, so block level. Also 4/6 quorum
– hash indexes are crash-safe in aurora 9.6, with be same with GA in 10
– managed service AWS staff gets paged on TID wrap-around
Securing Your Data on Postgresql
Payal Singh – OmniTI Computer Consulting Inc. payal@omniti.com
– Policies
– Row-level security (FORCE for table owner)
– Public is “public”
– before 10, policies were “OR”. 10 can do “AND”
– SSL (Certs needs to be 600 or restart will fail)
– event triggerimg like ddl event for auditing, can detect ownership changes
– table_rewrite notification to reject ddl and send message (possible solution for write amp)
– pg_audit
– at-rest encryption: pgcrypto. There is performance impact, so choose columns
– backups
– monitors to monitor queries for compliance (pg_stats extension)
– cryptdb can do encrypted queries (mysql-based)
– read-only replica can be a compliance problem because no logging possible
– desired features: redaction, oracle the, show grants
– pw strength is only on plaintext pa’s, not md5 or scram
– no rush for AWS KMS support
Saving Bandwidth When Using MySQL
Georgi Kodinov, Oracle MySQL SrvGen Team Lead (includes Security Team)
– Wireshark (smallest item is data response!) – disable SSL, will show sub requests
– SELECT and CALL generate metadata is large, but needed for MySQL client program, not for PHP apps. 443 vs. 106 bytes
– SET @@session.resultset_metadata=NONE;
– MySQL protocol has “capability flag” to help with backward compatibility
– Text protocol vs. binary protocol
– MySQL Protocol Doxygen in MySQL 8.0
Amazon Aurora MySQL and RDS MySQL: Lessons Learned
Mariella Di Giacomo
– did benchmarks with t2.medium and a large of RDS MySQL vs. Aurora MySQL. Aurora won, but I’m not sure of the difference in cost
Exhibit Hall
– got a demo of Vivid Cortex SAML/Okta support. Very nice UI.
VividCortex Dinner at Levi’s Stadium
– nice chance to meet other Vivid Cortex users and conference attendees
– Brendan Gregg is starting a cricket team at Netflix.
Wednesday Keynotes
Wednesday Lectures
PostgreSQL Replication
Christophe Pettus, PostgreSQL Experts thebuild.com
WAL-Based Replication
– first core replication was WAL shipping. All or nothing, monitor your disk. Same major version.
– archive_command can copy to secondary, manage it yourself
– for stream replication, use recovery.conf to point secondary at primary
– optional sync available (sometimes used to avoid lag)
– replicas can cascade
– max_standby_*_delay to control replication lag
. Use 0 for DR
– hot_standby_feedback
Trigger-Based Replication
– triggers cascade and run in alphabetical order
– on tables (Slony 1). Bucardo does multi-master replication (ping-pong protection)
– tedious, fiddly, performance impact, no DDL
– Slony requires C-language extensions, so cannot use on RDS
– Bucardo can be used on RDS
– RDS version 10 has WAL files access
Logical Decoding
– introduced in 9.4, improved in 9.6
– create replication slot to capture WAL stream
– tracks WAL position of consumer
– can run out of disk space
Replication Plugins
– can do whatever you want
– eg. export to Kafka
Logical Replication
– PG 10 has built-in logical replication, 9.4 has pg_logical
– pg_dump to dump schema
– primary key or unique index is a good idea, often required
– sequence values are not replicated, use disjoint ranges per server or UUIDs
– truncate not replicated, nor cascade
– replicate real table to real table, not view, FK
– pg 10 partitioning cannot be replicated because root table is not a real table
– no temp tables or unlogged tables
– COPY is individual sql inserts, could be millions
Pgpool2
– statement-based replication splits commands and sends to 2 servers – don’t use
Amazon DMS
– pg logical decoding
– timestamptz not supported at this time
2nd Qiuadrant BDR
– closed source, bi-directional
Streaming does DDL – reliable since 9.3, monitor secondary in case of quiet disconnect
PostgreSQL Replication
Simon Riggs, 2ndQuadrant
– Physical Streaming replication is sending WAL. Avail. 8 years. In-core
– Logical is non-WAL, like MySQL
– hot standby is same as read replica (AWS)
– repmgr and barman help mng
– postgres_fdw (foreign data wrapper)
– file_fdw access data like COPY
– PG pub/sub replication (push)
Multi-Node Advanced Features
– Push (Logical) or Pull Data Access (FDW)
– Multi-server hetero SQL
– Sharding (native in PG 11)
– Multi-node Query
– Multi-master database
– PostgreSQL-XL MPP similar to Teradata, Greenplum, Redshift
– Postgres-BDR Geo Cluster
– https://www.2ndquadrant.com/en/resources/postgres-bdr-2ndquadrant/
– BDR = Bidirectional Replication
Deep Dive into the RDS PostgreSQL Universe
AWS
– use cross-region read replica for migration
– create a paramater grep with rds.force_ssl=1
– RDS uses pg_upgrade or DMS
– OS Level Enhanced Monitoring
– Amazon RDS Performance Insights (Database Level) – Postgres Aurora today
Conference Closing Session
Lightning Talks
See this Percona Blog.
Prizes Giveaway
– Nice prizes for filling out the exhibitor pass.
As a commercially-rated airplane pilot who’s read over 2,000 accident reports, I followed the Southwest Flight 1380 fatal uncontained engine explosion closely.
Executive Summary:
The FAA is likely investigating the following concerns:
There are FAA regulatory and political expectations that uncontained engine failures don’t happen. However the physics of large machines with rotating components says otherwise. Likely this will result in armoring the engine shroud and hull with kevlar or metal, and further engine design changes based on energy of rotating components.
There are FAA expectations that the aviation industry mitigates risk so that the uninformed flying public is not injured. However, a passenger died en route.
After the depressurization, oxygen masks were expected to descend and be used by passengers over their nose. Instead they were put over their mouth. Likely passenger briefings will become more detailed, and oxygen masks will be changed from a round to pear-shaped.
Fortunately the pilots descended relatively quickly, at 3,000 fpm, to 10,000′, as:
However, was the emergency descent actually fast enough? With an improperly-worn mask, generally-speaking the body needs sufficient cabin oxygen pressure within 4 minutes to prevent brain-damage or death.
avweb.com: Southwest Accident Brings Passenger Safety Briefings To The Forefront
philg: Southwest 1380: think about the flight attendants
avweb.com: Southwest 1380: “Flew Like a Rock”
ainonline.com: Southwest Airlines 1380 Engine Failure 4/17/2018 ATC Audio
1 killed as Southwest jet makes emergency landing after apparently blowing an engine in flight
Qantas Oxygen Mishaps (2007 and 2008)
These Qantas incidents illustrate the importance of vigilance when maintaining passenger oxygen canisters:
1) Careless filling with nitrogen instead of oxygen:
theage.com.au: Probe after Qantas pumps wrong gas into jets PPRuNe
2) Oxygen canister valve failure due to unknown cause:
nytimes.com: Officials Ask Qantas to Inspect Oxygen Canisters
telegraph.co.uk: Exploding oxygen bottle caused hole in Qantas jet
smh.com.au: Valve in oxygen cylinder the culprit in 747 explosion
I spent the afternoon at the amazing Hiller Aviation Museum at San Carlos Airport (SQL).
The museum is a real gem, with compact (3 hours to see everything) but nice exhibits in the areas of:
1) Hiller helicopters
2) flying platforms
3) engines from 30 HP (Wright B) to 3,500 HP (Wasp), including a Merlin
4) flight simulators
5) 727 and 747 “steam gauge” cockpits
6) a truly beautiful Grumman Albatross
7) models of most other aircraft
(The Boeing SST that was on loan has been returned to Boeing.)
Flight Simulators at Hiller
Hiller has done a great job of integrating modern computer displays and flight simulators into their exhibits:
Mechanical Flight Simulators
– Curtiss P-2 with map table
– portable flight director
– IFR simulator
– Stearman simulator
Computer Flight Simulators
– 16x PC on 2nd floor (FS9 with Saitek yoke, engine quadrant, and pedals with 3 monitors)
– 1x Redbird FMX full-motion AATD
– Google Earth “wall”
– Wright Flyer (MS FS9 with wood levers and 3 screens)
– a real helicopter with cyclic control and 1 screen
They are moving to X-Plane 11 as PC hardware compatibility leaves FS9 behind.
1) The 747-100 fuselage static display behind the museum lets you climb up the 1st class cabin and sit in the cockpit. It has 5 seats (pilot, copilot, navigator, radio and FAA.) It is closed at 4:30 pm.
2) Behind the 747-100 is a small platform where you can observe takeoffs and landings. It is closed at 4:30 pm.
3) The Burger King next door has a helicopter on static display outside, and historical aviation fotos and a “briefing area.”
avweb.com: Personal Flight Simulators: one-G Simulation
There are few people who have used both MySQL and Postgresql in production at scale. Both are great Open Source databases, so I thought I’d add some comments based on my experience.
| Feature | MySQL/InnoDB | Postgresql |
|---|---|---|
| Replication | Many options, simple to admin | Many options, many warts |
| Index Write Amplification | No | Yes |
| Error handling on disk full | Sketchy | Yes |
| Grants complexity | Simple to understand | Complex |
| Sync and Semi-sync options | Several | Yes |
| Easy-to-use CREATE PARTITION | Yes | PG 10+, root table is not real |
| CHECK CONSTRAINT | Ignored | Yes |
| CTE (WITH) | MySQL 8+ | Yes |
| ON DUPLICATE KEY | Yes | PG 9.5+, but odd |
| Partial Indexes | No | Yes |
| Functional Indexes | No | Yes |
| DELETE Purging | Automatic Purge thread | VACUUM |
| Object Names | Finely-scoped | Globally-scoped |
| Invisible Indexes | 8.0+ | hypopg extension |
GuardDuty is free for 30 days and will report on what future use will cost, so when you’re ready, just enable it. Since it monitors other AWS logs, there’s no impact on your other services or instances.
One of the filter methods is a lambda, which can call another lambda (chained lambdas.)
If you create a separate “forensics account” in the same AZ, you can automatically do some sophisticated things:
The lecture “A Case Study on Insider Threat Detection” was mostly on GuardDuty. In my experience, Loft lecturers are excellent, and this was no exception.
Afterwards was a detailed 2-part lab where you create two hosts, have them interact, and view the events in GuardDuty. Bring a mouse to AWS labs, because you’ll be doing a lot of clicking around. 🙂
I noticed that most of the attendees and even the “Ask the Experts” were not familiar with newer AWS services and features, like GuardDuty and PrivateLink. Such is the rapid progress that AWS is making.
Gripes: The pasta salad was slightly crunchy (pasta was under-cooked by about 3 minutes) and there was no half-and-half for the coffee. Also, the previous Loft configuration with lectures upstairs and Ask the Experts downstairs made more sense, since putting them on the same floor causes noise interference problems.
AWS Pop-up Loft
1446 Market Street, San Francisco
Feb 20 – Feb 23 10:00AM – 4:00PM
(You should register the week in advance for AWS Loft events, but you can also register on-site with a photo id.)
AWS Forensics Marketplace Vendors
AWS Loft London: Incident Response and Forensics Slides
/r/aws: Is AWS GuardDuty an IDS/IPS?
Congrats to SpaceX on the successful launch of Falcon Heavy, with a Tesla for ballast.
And yes, recovering two boosters simultaneously on adjacent pads is showing off, but of the best kind!
One of the main goals of this launch was to not blow up the launch pad, since that would:
Although it’s always uplifting to see a successful launch, it’s actually valuable to have failures early-on. Having a successful first launch means the rocket was likely over-engineered (ie. heavy.) Hopefully review of the sensor data can find something worth investigating and fixing before launching satellites or people.
One launch doesn’t mean the mechanical and acoustical vibrations between the 27 engines is dampened enough, or even understood at this point.
“In America, if you have lots of money, you can build a space program and launch your own car into heliocentric orbit.” 🙂
SpaceX launches Falcon Heavy, the world’s most powerful rocket
WATCH LIVE: SpaceX to Launch Falcon Heavy Rocket #MarsRocket @3:45pm EST delayed
While watching Superbowl 52, I finished my github project Maintenance Banner, aka mx_banner.php.
It lets an application administrator schedule maintenance event notifications for a web application in a database, and the web application can then automatically display those on the post-login page.
mx_banner.php should be installed on a web server that supports SSL and uses authentication, but I suppose for internal applications you could get by without those security layers.
Your web application can be written in any programming language that can query the maintenance events database and insert the event information into a div on your application’s post-login page.
If you host multiple applications that can experience unique maintenance windows, then just install an extra copy of mx_banner.php and the database schema for each application maintenance “zone.”
mx_banner.php is a minimal but non-trivial web application that is also good for learning PHP application programming. Also see the WordPress plugin I wrote in PHP on github.
I watched Superbowl 52 on restaurant cable. Great seat for a great game with the the most ground and air yardage in history.
PHI won 41 – 33 NE in Minneapolis with some nice trick plays in the end-zone.
Brady started the game with an injured hand and had to bench for a while.
Sadly Brandon Cooks didn’t see a legal but huge hit coming after a reception and ran into a player with helmet contact, resulting in a head injury.
The kickers said they “had problems with the in-turf logos.” There were a few missed field goals, including one that hit the post.
Justin Timberlake did the half-time show, but with only 15 minutes to perform his appearance was too brief.
Nick Foles, High-Powered Eagles Stun Tom Brady, Patriots to Win Super Bowl 52
Jack in the Box (Jack and Martha Stewart) and Amazon (Alexa “turks”) had some funny commercials.
We’re lucky to have two great Open Source databases, MySQL and Postgresql.
One of the killer features in both MySQL and Postgresql is table partitions – for example, most Silicon Valley adtech companies are powered by MySQL partitions.
They let enterprises, and growing startups, easily manage large volumes of data.
Pluses
Minuses
Simpler syntax for Postgresql 10, known as declarative partitioning, has been announced that is similar to MySQL’s syntax.
Here’s a chart comparing partitions across database products:
| Feature | MySQL 5.0+ | Postgresql 8.0+ | Postgresql 10.0 | Postgresql 11.0 |
|---|---|---|---|---|
| Table Partition Syntax | 5.7 | 9.6 | 10 | TBD |
The MySQL bug database lists many serious partition bugs in 5.0 and 5.1, so it’s important to use the latest version possible. MySQL also supports merge tables for MyISAM tables.
Chapter 15 of “PostgreSQL 9 High Performance Book Review” covers Postgresql 9 partitions.
PostgreSQL Partition Manager Extension (pg_partman), Blog, github
Yahoo MySQL Partition Manager, Blog
PalominoDB pdb-parted
PostgreSQL 10: Partitions of… partitions! HN Discussion
wiki.postgresql.org: Table partitioning
Creating partitions automatically in PostgreSQL
rhaas.blogspot.com: Plans for Partitioning in v11
reddit: What are some bad things about PostgreSQL?
PostgreSQL 10 – table partitioning – how to check partitions and manipulate with them
Uniqueness in PostgreSQL: Constraints versus Indexes
Avoid naming a constraint directly when using ON CONFLICT DO UPDATE
PostgreSQL Upsert Using INSERT ON CONFLICT statement
How to install PostgreSQL 9.5 on CentOS 7
Alias for table name in SQL insert statement
Postgres 9.5 feature highlight – Upsert
psql: FATAL: database “
psql: FATAL: Ident authentication failed for user “username” Error and Solution
Postgresql Repos: 9.5, 9.6
Postgresql Sample Databases
Postgres.app
brew install postgresql@9.6
The last 747 has been retired from passenger service in the USA.
I’ll miss the 747 because it was the only widely-used airliner built for trans-oceanic flights:
Delta Retires Last USA 747 to Boneyard
Malaysia hunts owners of Three Boeing 747s abandoned at airport
This is a multi-part series starting with Part 1.
After perfecting the schema and data migration with pgloader rules and getting the application sessions and login code working, it was time to convert the non-working SQL statements to Postgres syntax.
Here’s the SQL syntax I had to change (note: I use placeholders (?) whenever possible):
| MySQL | Postgresql 9.1 | Notes/PG 9.5 |
|---|---|---|
| UNIX_TIMESTAMP(?) | FLOOR(EXTRACT(EPOCH FROM ?)) | |
| FROM_UNIXTIME(?) | TO_TIMESTAMP(?) | |
| DATE_ADD(?, INTERVAL ? unit) | ? + INTERVAL ‘$x timeunit’ | Perl DBI can do placeholder with ?::interval and “$x timeunit” |
| DATE_SUB(?, INTERVAL ? unit) | ? – INTERVAL ‘$x timeunit’ | see above for interval placeholder info |
| ? = 0 or ‘0000-00-00 00:00:00’ | ? IS NULL | |
| RAND() | RANDOM() | |
| INSERT IGNORE | INSERT | in the case when the action is not important. PG 9.5: INSERT … ON DUPLICATE KEY IGNORE |
| REPLACE INTO | custom code | in the case when rows are immutable (never updated). PG 9.5: INSERT … ON DUPLICATE KEY UPDATE |
| IF() | CASE WHEN … THEN … ELSE … END | |
| LIMIT ?, ? | LIMIT ? OFFSET ? | can use the pg syntax for both databases 🙂 |
| GROUP BY | GROUP BY | Postgresql is stricter about the columns list here |
| ORDER BY NULL | MySQL optimization only | |
| SQL_CALC_ … FOUND_ROWS | SELECT COUNT(*) | MySQL optimization only |
| DATE(?) | TO_CHAR(?, ‘YYYY-MM-DD’) | |
| HOUR(?) | EXTRACT(HOUR FROM ?) | |
| table `user` | table users | Postgresql has public.user, so treat the user table name as a reserved symbol |
| UPDATE table1, table2 … | UPDATE table1 … WHERE id IN (SELECT id FROM table2 WHERE …) | |
| last insert id (various) | RETURNING |
Don’t forget to:
Perl CGI::Session Notes
pgloader migrates the MySQL CGI::Session table using a Postgresql text column. This works better:
alter table sessions alter a_session type bytea using a_session::bytea;
innodb_buffer_pool_size, to use 90% of RAM you should consider 80% or 75% to avoid OOMs.
Note: innodb_buffer_pool_size can be set dynamically in MySQL 5.7 with some caveats:
SET GLOBAL innodb_buffer_pool_size=4G;
The above applies doubly to server consolidation and microservices in VMs.
Of course, if you’re an experienced production DBA, then you never trusted VMs anyway. 🙂
Some numbers from Redhat (paywalled):
> Measureable: 8-12% – Highly cached random memory, with buffered I/O, OLTP database workloads, and benchmarks with high kernel-to-user space transitions are impacted between 8-12%. Examples include Oracle OLTP (tpm), MariaBD (sysbench), Postgres(pgbench), netperf (< 256 byte), fio (random IO to NvME).
>Modest: 3-7% – Database analytics, Decision Support System (DSS), and Java VMs are impacted less than the “Measureable” category. These applications may have significant sequential disk or network traffic, but kernel/device drivers are able to aggregate requests to moderate level of kernel-to-user transitions. Examples include SPECjbb2005 w/ucode and SQLserver, and MongoDB.
Redis: Meltdown fix impact on Redis performances in virtualized environments
Cassandra: Meltdown/Spectre Linux patch – Performance impact on Cassandra?
I’ll leave it to others to pontificate on what it means when you can’t trust any desktop, server or mobile computer in an Internet-connected world. Or what HIPAA compliance means in the cloud where your server is a party-line telephone.
forums.aws.amazon.com: Degraded performance after forced reboot due to AWS instance maintenance , HN
ARM: Vulnerability of Speculative Processors to Cache Timing Side-Channel Mechanism
Escaping Docker container using waitid() – CVE-2017-5123
theregister.co.uk: Azure VMs borked following Meltdown patch, er, meltdown
CPU hardware vulnerable to side-channel attacks (Replace CPU hardware), HN (I called this in advance, but there needs to be two steps: re-design CPUs in 2018 if there’s no possible microcode update, then replace them in 2019)
blog.appoptics.com: Visualizing Meltdown on AWS
Intel alerted computer makers to chip flaws on Nov 29 – new claim – Total coincidence: That’s the same day Chipzilla’s CEO sold off his shares
zdnet.com: Researchers discover seven new Meltdown and Spectre attacks HN discussion
phoronix.com: Bisected: The Unfortunate Reason Linux 4.20 Is Running Slower HN
aws.amazon.com: Processor Speculative Execution Research Disclosure
forums.aws.amazon.com: Spectre/Meltdown Vulnerabilities – AWS please clarify
Recently I migrated a small but non-trivial (25 tables, about 500 columns, no triggers or SPs) MySQL schema to Postgresql using the Open Source pgloader utility.
pgloader supports migration from several databases/formats (MySQL, Sqlite, MS SQL, dBase, CSV) to Postgresql.
pgloader is fast: it took about 20 seconds to migrate/load both the schema and data into Postgresql 9.2 on my Mac notebook running in Virtualbox. The total number of rows was about 3 million narrow rows.
The impressive speed of pgloader lets you iterate quickly when writing pgloader rules, and lets you embed the loading process into automated scripts without any concern about performance.
Executive Summary
pgloader is a free scriptable ETL tool that lets you quickly migrate database schemas and data to Postgresql. It is helpful for setting up a PoC to estimate the work required for a full migration analysis. (It does not do SQL syntax or application code migration.) Although pgloader is excellent at what it does, any database migration is a major project.
Installation on CentOS 7
yum install freetds postgresql createdb mydb # unpack and run the build script from github or use the Docker image make pgloader pgloader mysql://root@localhost/mydb postgresql:///mydb
Installation on Debian or Ubuntu
apt-get install pgloader postgresql createdb mydb pgloader mysql://root@localhost/mydb postgresql:///mydb
pgloader pluses:
– free
– fast
– 100% scriptable and customizable with pgloader rules (called a “load file”)
– reasonable result for a first pass even without custom pgloader rules
– “pro DBA” feeling – accepts configuration file, and emits log files, similar to the behavior of Oracle’s BCP utility.
pgloader minuses:
– not included with Postgres
– unusual in that it is written in Lisp, but that is not user-visible
– doesn’t create Postgresql users/roles (to be expected as the SQL standard doesn’t specify these, thus they vary greatly across databases)
– doesn’t convert MySQL partitions to Postgresql inheritance-syntax partitions in 8.x or 9.x, but should work with declarative-syntax partitions in 10.x+
– detects and double-quotes reserved object names, but doesn’t notify of schema name conflicts with the Postgresql public namespace ie. table name ‘user’
– will require time-consuming schema cleanup for most use cases. See below.
My pgloader project files are available for download from my github.
Schema/Data Cleanup Notes
My Migration Results
After renaming the table ‘user’ to ‘users’ and altering the sessions table (see above), around half of the SELECT queries worked as-is and I could login to the application and click around. 🙂
However, virtually all of the INSERT and UPDATE statements had to be rewritten, taking 2 man days to get to an alpha version and 2 weeks for something worth doing formal QA.
Database Migration Alternatives
Amazon AWS provides 3 powerful data migration tools under the AWS Database Migration Service banner that are either free to use, or have a 6 month trial.
pgloader: Homepage, github, Manual, Licence
postgresql.org: pg_dump
http://rhaas.blogspot.com: The State of VACUUM
severalnines.com: Upgrading Your Database to PostgreSQL Version 10 – What You Should Know
Macbook Pro 2009/2010 notebooks came pre-installed with Snow Leopard (10.6) Mac OS X.
As of Dec. 2017, very little popular software will work or update on anything older than Mavericks (10.9):
Before you update Mac OS X, note that many users have complained of post-update issues including:
The best way to update Mac OSX is to use the Apple softwareupdate command line interface (CLI) tool:
sudo softwareupdate -l (copy and paste the updates list to a file and save it for later reference)
sudo softwareupdate -ia
$ sudo softwareupdate -l Password: Software Update Tool Finding available software Software Update found the following new or updated software: * macOS High Sierra 10.13.2 Supplemental Update- macOS High Sierra 10.13.2 Supplemental Update ( ), 138293K [recommended] [restart]
Otherwise you need to register with the App Store using a credit card to use Software Update. Expect 10 or more reboots/logins and about 4 hours total time if everything goes smoothly. The final reboot will actually install the new software – it will take about 41 minutes on a hard drive, or 20 minutes on an SSD.
Tips
Troubleshooting
sudo softwareupdate -l
apple.com: High Sierra macOS freezing and stops, HN
W: Macintosh operating systems
I opened WordPress Bug 42866: “WordPress Community Scheduler Interval Bug.”
I found that when writing the RackPing Monitoring widget, scheduled tasks frequently run up to 45 seconds earlier than the 15 minute interval specified. This may have wide and serious impact to the widget community, especially widgets that do scheduled posts and other visible blog changes.
[ WordPress has commented on my bug report with a detailed explanation. ]
theregister.co.uk: WordPress captcha plugin on 300,000 sites had a sneaky backdoor
wordfence.com: Display Widgets Plugin Includes Malicious Code to Publish Spam on WP Sites
I don’t see a lot of links on autoscaling AWS RDS read slaves.
Is the reason:
Below are some insightful comments and some projects that have implemented this.
“It should be fairly easy to create a data-driven auto scaling database cluster using Read Replicas and CloudWatch metrics.
At the low end, this cluster would consist of a Small DB Instance running in one Availability Zone with 5 GB of storage.At the high end it would consist of a High Memory Quadruple Extra Large Multi-AZ deployment (primary/secondary pair) DB Instance with 1 TB of storage and 5 associated Read Replicas. That’s quite a range!”
harish11g.com: Load Balancing Amazon RDS Read Replica’s using HAProxy
github.com: ReDS – ReActive Database System – Terraform – Auto-Scale for RDS Instances
Other Resources
docs.aws.amazon.com: Replication with a MySQL or MariaDB Instance Running External to Amazon RDS
I propose a new normal form for databases, Blob Normal Form (Blob NF.)
Definition: A database is in Blob Normal Form (Blob NF) when all fields containing binary data or long arbitrary text are moved to tables consisting of a primary key and a blob column.
Good candidates for Blob NF are images and documents where database constraints are not applicable.
Motivation: Blob Normal Form solves physical and operational problems involving storage, performance, retention and character set assignment as blobs have wildly different characteristics from typical database values. A blob generally cannot act as a real primary key, so it can almost always be moved to a different table by itself (possible exceptions would be unique constraints, rarely used for blobs.)
Comment: Since Blob NF is not strictly a relational algebra database form, it should not have a numeric name as 1NF to 6NF have.
Relationary: The Art of Database Normalization
W: Sixth normal form
There’s 2 common misconceptions in engineering West Coast – East Coast data centers:
Most engineers halt with those rules-of-thumb in mind, but what happens when we look at the actual numbers with ecommerce/advertising applications in mind?
Cross-USA Internet Packet Latency (One-Way)
| Transmission Method | End-End Speed | Time SF-NY | Note |
|---|---|---|---|
| Light in vacuum | 299,792 kps | 16 ms | similar speed in air |
| Microwave in air | 235,000 kps | 20 ms | Repeaters every 48 km (actually built in 1950s in both USA and Canada! Currently HFT applications use 15+ microwave routes from Chicago to New York.) |
| “Electricity” in wire | 235,000 kps | 20 – 30 ms | Depends on how wire is constructed and what electromagnetic property is used to transmit information. As a rule of thumb, electrons flow in a typical electronics wire at c/2 while an electromagnetic wave propagated in an air gap is up to .99c. |
| Light in silica fiber (theoretical) | 204,081 kps | 22 ms | Index of refraction is 1.45 |
| Oceanic cable for comparison | 156,666 kps | 30 ms | Including amplification and switching |
| Google Routing in silica fiber | 150,000 kps | 31 ms | Extrapolated from The Dalles to Ashburn (4,350 km) at 29 ms |
| AT&T Routing in silica fiber | 150,000 kps | 31 ms | |
| Public Internet Packets in silica fiber | 137,000 kps | 35 – 36 ms | Public Internet already using MPLS |
| Transmission Method | End-End Speed | Time SF-NY | Note |
From Figure 1 above, we see that light can travel from SF to NY in 16 ms, yet the public Internet averages 35 ms. That’s 2.2x longer than expected if packets are supposed to travel at the speed of light in a vacuum.
Ever watched the expression on somebody’s face when you tell them, “Packets don’t travel at the speed of light.”? It’s priceless.
Now that we’ve described numerically the latency limits, there’s some very interesting things to investigate:
So how does that help us with multi-master database latency?
What Does a Reliable Network Mean?
Reliable for wide-area multi-master database transactions means:
At this time, that requires a private network, either yours or a cloud provider.
What is the Low-Hanging Fruit?
From lowest-cost to highest-cost for making database transactions WAN-safe:
In my experience, most organizations never even get to step #1 above: 🙂
Fortunately, there is a half-measure: multi-AZ with AWS uses different data centers in the same region with only 1-2 ms inter-DC latencies. James Hamilton from AWS calls using small data centers in the same region “limiting the blast radius.”
The Speed of Light – Depends on the Medium
The speed of light in a vacuum is 299,792,458 meters per second, or 186,282 miles per second. In any other medium, though, it’s generally a lot slower. In normal optical fibers (silica glass), light travels a full 31% slower.
Exercises for the Reader
Please leave a comment!
Please leave a comment (no registration required) if you have any experience implementing similar topologies, or have suggestions or corrections.
Resources
How Google Does It
cloudplatform.googleblog.com: With Multi-Region support in Cloud Spanner, have your cake and eat it too
Google Public NTP
Microwave WAN Transmission
The secret world of microwave networks
The Abandoned Microwave Towers That Once Linked the US
Trans Canada Microwave
mckay-brothers.com: Microwave Bandwidth at Extreme Low Latency
109 Microwave Towers Bring the Internet to Remote Alaska Villages
Fiber Optic
Calculating Optical Fiber Latency
$1.5 billion: The cost of cutting London-Tokyo latency by 60ms
Researchers create fiber network that operates at 99.7% speed of light, smashes speed and latency records (fiber optic waveguide)
Public Internet Latency Measurements
SO: How much network latency is “typical” for east – west coast USA?
AWS Inter-Region Latency
Research and Other News
netflix: Active-Active for Multi-Regional Resiliency
Network latency – how low can you go?
W: Multiprotocol Label Switching (MPLS)
Latency: The New Web Performance Bottleneck
developer.apple: Networking Concepts
hpbn.co: Primer on Latency and Bandwidth
Network performance: Links between latency, throughput and packet loss
Turning the Optical Fiber Network into a Giant Earthquake Sensor
fgiesen.wordpress.com: Network latencies and speed of light
Einstein, Poincaré & Modernity: a Conversation
Cloudify and Gigaspaces wrote a really useful slide deck titled, “Orchestration Tool Roundup – Docker Swarm vs. Kubernetes, TerraForm vs. TOSCA/Cloudify vs. Heat” dated June 11, 2015.
There’s 3 things that make the slides awesome:
So it’s really an evaluation framework for you to add new tools, or customize for your environment. 🙂
Adding Cloudformation Provisioning
To add AWS Cloudformation (CF) after Terraform, basically they’re both so simple that you can just repeat the TF slides, change “Terraform” to “Cloudformation”, except CF is AWS-specific.
Cloudformation scripts are written as JSON or YAML templates and can be executed using the AWS CLI or API or the Management Console UI, and the resulting cluster definition is called a “stack.”
Cloudformation itself is free to use, thus you only pay for the underlying AWS resources consumed. Conceptually for Chef and Puppet users, a CF template just saves you from calling the AWS API multiple times.
This is a short linux bash script I wrote that’s adequate for zero’ing a filesystem for on-premise server storage.
Note that the only “secure erase” is to destroy the physical drive because of various internal disk-level caches. 😐
To do multiple passes (in this case 10 passes):
# seq 1 10 | xargs ./zero.sh
#!/bin/bash # Program: zero.sh # Purpose: write on pass of zeros in free space on current volume. # Date: 2017 10 28 # Author: James Briggs, USA # Usage: cd "filesystem"; screen ./zero.sh # Env: CentOS 6,7 bash # Notes: # This program is write-intensive, so not recommended for SSD or Flash drives unless disk wear is ok # This is not a secure erase, but is ok for on-premise storage. outfile=zero sz=1G delay=0 n=1 err="" # cleanup after previous runs /bin/rm -f *.zero.bin "$0.out" /bin/sync # while [ "$n" -le "4" ]; do while true; do /bin/echo "$0: writing chunk #$n of $sz ..." /bin/dd if=/dev/zero of="$outfile-$n.zero.bin" oflag=nocache bs=$sz count=1 conv=fsync || let err="yes" /bin/echo "$0: syncing chunk #$n of $sz ..." /bin/sync [ -n "$err" ] && break sleep $delay let n++ done /bin/echo "$0: deleting $n chunks of $sz ..." /bin/rm *.zero.bin /bin/echo "$0: syncing filesystem after chunk deletions ..." /bin/sync msg="done zeroing current volume with $n chunks of $sz." /bin/echo "$0: $msg" /bin/echo "$0: `/bin/date` $msg" > "$0.out" exit 0
A one-line log file is written:
# cat zero.sh.out ./zero.sh: Sun Oct 29 04:16:30 UTC 2017 done zeroing current volume with 12 chunks of 1G.
A more complicated utility:
github: redeemer
Redis is a feature-packed cache that’s easy to install and work with. Here’s some notes on installing and using Redis with CentOS 7.
Install available redis package from your available yum repos:
# yum install redis # systemctl enable redis # systemctl start redis
By default, Redis is configured at installation to bind to localhost with no password in /etc/redis.conf.
See status of redis and redis services:
# systemctl list-unit-files 'redis*'
| UNIT FILE | STATE |
| redis-sentinel.service | disabled |
| redis.service | enabled |
| 2 unit files listed. | |
Setup for programming with some Perl modules:
# cpan Redis Redis::Fast Redis::List
Perl code snippet for a web page performance footer:
use strict;
use diagnostics;
use Redis::Fast;
my $r_cache = cache_info();
my $cache_info = '';
$cache_info = "Redis $r_cache->{'redis_version'} \
up $r_cache->{'uptime_in_seconds'} s \
$r_cache->{'keyspace_hits'}h/$r_cache->{'keyspace_misses'}m \
/$r_cache->{'expired_keys'}e last write: \
$r_cache->{'aof_last_write_status'}" if defined $r_cache;
print $cache_info;
sub cache_info {
my $h = Redis::Fast->new;
if ($h) {
my $r_hash = $h->info();
$h->quit();
return $r_hash;
}
return undef;
}
will output a useful message like:
Redis 3.2.10 up 414908 s 1146h/301m/141e last write: ok
Interesting news that Amazon is selling a waterproof Kindle, and that previously Jeff Bezos used Kindles in a one-gallon Ziploc bag in the bathtub.
I worked on the Cloud backend for the discontinued Ricoh eQuill waterproof business tablet. There are myriad uses for a waterproof work tablet:
Ricoh Waterproof Ewriter Tablet (2011)
W: International Protection Marking Code
Disclaimer: I have worked on tablet Cloud software for Amazon, Apple and Ricoh.
Ricoh Introduces the eWriter Solution ─ New Business-class Tablet and Back-end Services Improve Business Efficiencies by Moving Paper Processes Online
The Ricoh eQuill is an e-ink tablet for businesses
Ricoh Targets Channel With Tablet, Digital Workflow Services
LWN has a nice article on “Safety-critical realtime with Linux.”
The intro has a good overview of real-time concepts, and the rest of the article has a summary of Linux implementation techniques.
(RTLinux is not mentioned. The company was bought by Wind River Systems in 2007, which has dropped commercial support.)
W: RTLinux, RTAI
linuxfoundation.org: Preempt RT
The recent CentOS 6 Apache httpd 2.2 server update changes the envariable HTTP_HOST from ‘domain’ to ‘domain:port’, causing cookie domain match and redirect issues with web apps, especially in a reverse proxy setup.
/var/log/yum.log: Sep 23 23:55:25 Updated: httpd-2.2.15-60.el6.centos.5.x86_64
The fix for WordPress in my setup with a httpd front-end and a httpd_php backend (reverse proxy) is:
wp-includes/canonical.php:
- if ( ! $requested_url && isset( $_SERVER['HTTP_HOST'] ) ) {
+ if ( ! $requested_url && isset( $_SERVER['SERVER_NAME'] ) ) {
// build the URL in the address bar
$requested_url = is_ssl() ? 'https://' : 'http://';
- $requested_url .= $_SERVER['HTTP_HOST'];
+ $requested_url .= $_SERVER['SERVER_NAME'];
$requested_url .= $_SERVER['REQUEST_URI'];
}
To fix similar redirect problems in other Open Source web products, just:
s/(:\d+)$//;
PSA: If you update to CentOS 7.3 and see odd console or log errors like this resulting in a hung boot:
Failed to start Import network configuration from initramfs
and/or
work still pending FAILED Failed to start Login Service. See 'systemctl status systemd-logind.service' for details. FAILED Failed to start Authorization Manager. See 'systemctl status polkit.service' for details. DEPEND Dependency failed for Dynamic System Tuning Daemon.
then you have a problem with CentOS 7.3 and SE Linux 2.5.
The workarounds are surprisingly simple:
ask.fedoraproject.org: Startup fails with multiple errors
http://danwalsh.livejournal.com/38157.html
SSH: Unable to get valid context
Keywords: Virtualbox, linux, selinux, enforcing.
Cloudera hosted another Solr Meetup at their office in Palo Alto. About 30 people attended.
Two software engineers from Cloudera did presentations tonite:
1) Michael Sun talked about his nitely Solr microbenchmark and cluster benchmarks.
Solr Nightly Benchmarks (SOLR-10317)
2) Mano Kovacs from Cloudera talked about Solr LIR trouble-shooting techniques
Nice t-shirt: “Data is the New Bacon.” 🙂
Leader incorrectly publishes state for replica when it puts replica into LIR (SOLR-9555)
He also talked about limitations and issues of of autoAddReplicas.
Recommends PlantUML for documentation.
Audience Questions
– piercing – checking 100 document versions may not be enough if there’s 1000 total versions
After party at Antonio’s Nuthouse on California Ave.
Thanks to Cloudera for hosting this event and the Mediterranean food!
Cloudera 1001 Page Mill Road Palo Alto, CA 94304
Apache Solr Memory Tuning for Production
techcrunch.com: Algolia raises $53 million for its search engine API
Avery’s Law of Distributed Systems Reliability: “Distributed systems are more reliable when you can get a service from one node OR another. They get less reliable when a service depends on one node AND another. And the numbers combine multiplicatively, so the more nodes you have, the faster it drops off.”
Lamport’s Observation: “A distributed system is one in which the failure of a computer you didn’t even know existed can render your own computer unusable.”
Both sound simple enough, even obvious for systems serving HTTP(S) or caching.
But apply those to clustered and distributed databases, and you can do powerful analysis on expected availability.
| Topo | Nodes | Database | Configuration | Reads during single node failure | Writes without a failure | Writes during single node failure |
|---|---|---|---|---|---|---|
| WAN | 2 | MySQL | Master-Slave Async Replication | OR | Single Master | NA |
| WAN | 2 | MySQL | Master-Master (standby) Async Replication | OR | OR (Single Active) | OR (Single Active) |
| LAN | 2 | MySQL | Master-Slave Semi-sync Replication | OR | Single Master | NA |
| LAN | 2/4 | MySQL CGE NDB Cluster | Sync Replication | OR | OR | NA |
| LAN | 3 (Min.) | MySQL Galera | Sync Replication | OR | OR | Requires quorum for OR |
| WAN | 3 (Min.) | Cassandra | Appropriate RF and CL | OR | OR | OR |
acm.org: Turing Award to Leslie Lamport
MySQL’s modern UTF-8 encoding is named utf8mb4 (4 bytes), not utf8 (3 bytes.)
For new applications, especially web, you should start with utf8mb4. For existing applications, you need to decide if an upgrade is worthwhile, and test extensively before a production upgrade.
Caveats:
Testing utf8mb4
Production Migration to utf8mb4
If you have applications written in Perl, you need to first upgrade DBD::mysql to a version greater than 4.041. (Even CentOS 7 comes with only 4.023.)
Before (CentOS 7):
$ perl -e 'use DBD::mysql; print $DBD::mysql::VERSION' 4.023 # mysql_config -bash: mysql_config: command not found
After:
# yum install mysql-devel # cpan DBD::mysql $ perl -e 'use DBD::mysql; print $DBD::mysql::VERSION' 4.043
blogs.perl.org: DBD::mysql – all your UTF-8 bugs are belong to us!!
SO: Trying to install Perl-Mysql DBD, mysql_config can’t be found
MySQL utf8 vs utf8mb4 – What’s the difference between utf8 and utf8mb4? HN
mysqlserverteam.com: MySQL 8.0: When to use utf8mb3 over utf8mb4?
Using Databases with DBI: What Not To Do
I gave a talk at the Silicon Valley Perl Meetup on “REST API Server Programming With Perl.”
Here are the slides:
github: Perl Petstore Enhanced REST API Framework, Sample REST API Clients
There was an extended audience discussion afterwards with some interesting observations:
Swagger 2.0: How to specify an input parameter of type ‘object’?
Reading the Swagger 2.0 spec:
File Structure
“The Swagger representation of the API is made of a single file. However, parts of the definitions can be split into separate files, at the discretion of the user. This is applicable for $ref fields in the specification as follows from the JSON Schema definitions.
By convention, the Swagger specification file is named swagger.json.”
Thanks to Nvidia for hosting the meetup again.
OpenAPI Specification Version 2.0
URLs are UI
API-Security-Checklist
JWT Comments
Jeremy Zawodny: From mod_perl to Mojolicious at craigslist Slides
amihaiemil.com: What is HATEOAS?
Ask HN: What’s your biggest struggle with Microservices?
Ask HN: What are the not-so obvious things to consider while API integration?
Ask HN: In 2018, What makes a good API?
PSA for Mac OS X users on older machines like Lion and Mountain Lion: if you’re stuck on Skype 6.15, and can’t do video chat after the Microsoft’s July 3, 2017 breaking update …
Apple Facetime is a free video chat application that comes with Mac OS X.
Some tips for using Facetime:
Configuring IPv6 on your linux server is this easy if your ISP is IPv6-ready (if not, see Tunnelbroker links below) on CentOS 5 and 7:
vi /etc/sysconfig/network-scripts/ifcfg-my_interface file, note the settings that start with “IPV6” and update them:
You don’t need new IPv6 switches, since switching is done at Layer 2. I’m using the old bargain web-managed HP Procurve J9028A. 🙂
(When people say IPv6-capable, that means the management features can be assigned IPv6 addresses, or that it can do Layer 3 routing functions with IPv6 addresses.)
To test:
ping6 ipv6.google.com
# ping6 ipv6.google.com PING ipv6.google.com(sfo07s17-in-x0e.1e100.net (2607:f8b0:4005:80a::200e)) 56 data bytes 64 bytes from sfo07s17-in-x0e.1e100.net (2607:f8b0:4005:80a::200e): icmp_seq=1 ttl=56 time=1.39 ms
traceroute -6 ipv6.google.com
# traceroute -6 ipv6.google.com traceroute to ipv6.google.com (2607:f8b0:4005:80a::200e), 30 hops max, 80 byte packets 1 gateway (xxx:xx:x:xxx::1) 3.959 ms 3.978 ms 4.005 ms 2 10ge7-3.core3.fmt2.he.net (2001:470:0:274::1) 7.859 ms 7.933 ms 11.998 ms 3 10ge10-5.core1.pao1.he.net (2001:470:0:263::2) 11.296 ms 0.785 ms 11.311 ms 4 google-as15169.10gigabitethernet8-2.core1.pao1.he.net (2001:470:0:244::2) 0.944 ms 0.946 ms 0.980 ms 5 2001:4860:0:1004::1 (2001:4860:0:1004::1) 1.471 ms 2001:4860:0:1006::1 (2001:4860:0:1006::1) 1.478 ms 2001:4860:0:1004::1 (2001:4860:0:1004::1) 1.577 ms 6 2001:4860:0:1::1f71 (2001:4860:0:1::1f71) 1.328 ms 1.266 ms 1.221 ms 7 sfo07s17-in-x0e.1e100.net (2607:f8b0:4005:80a::200e) 1.150 ms 1.197 ms 1.210 ms
Tips:
rootusers.com: Configure IPv6 Addresses And Basic Troubleshooting In Linux
google.com: Google Public DNS IP addresses
centos.org: Are you using Network-Manager in no-GUI CentOS 7 Server?
HE Tunnelbroker Links
If your ISP does not support IPv6 yet, you can tunnel traffic in and out of your machine using the HE Tunnelbroker. This is also simple to setup, taking about 5 minutes if your server has IPv6 enabled. (Note that IRC and email traffic to port 25 are filtered to reduce abuse.)
Create Hurricane Tunnel Broker on Raspberry Pi
Hurricane Electric Free IPv6 Tunnel Broker
AWS IPv6 Update – Global Support Spanning 15 Regions & Multiple AWS Services
In the 90’s and 00’s, programmers used to scorn front-end designers who called writing HTML “programming.” That changed when CSS arrived and pixel-perfect results raised the bar.
Here’s some tips for programmers struggling with fixing CSS rendering problems:
Before Getting Started
Getting Started
Note: if you’re paying for HTML or CSS, always validate it before payment. Even better, ask your designer to validate it weekly and before delivery.
Debugging Strategies
Emergency Fallbacks
If you’re on a deadline or don’t have story points for a CSS project, there are workarounds.
Getting Done
Little CSS Stuff Newcomers Get Confused About
Related Annoyances
Tools
I went to RedisConf 2017 Wed. and Thu. at the Marriott Marquis in SF hosted by RedisLabs.
About 2,000 people showed up – initially surprising to me, but then Redis is more than a key-value store.
Kudos to RedisLabs for a great conference: excellent venue, very well-organized, good food but intermittent Wifi.
Special credit for the combination of excellent, very technical talks with “lateral” keynotes that were thought-provoking even when not Redis-specific.
Conference slides are here.
Executive Summary
Wednesday Keynotes
Kelsey Hightower, Staff Developer Advocate, Google
– Kelsey built redis kubernetes cluster in live demo and did failures
– used “Siri” voice commands to do some of the deployment
Sam Ramji, VP Product Management, Google
– very interesting high-level comments
– Dr. Eric Brewer is a VP Eng.
– “Spanner defeats CAP” because:
1) Google’s network is reliable (no partitions expected)
2) atomic clocks in each DC, less than 200 microsecond drift
– Spanner has SQL UI now, so anybody can access data.
Case Study: Redis Cluster at Flickr (Yahoo!)
Sean Perkins, Sr. Operations Engineer (18-year veteran)
– traditional large-scale IT environment (still pets not cattle with long-lived masters and names/IP addresses)
– had to do a cleanup and doc phase, dedicated vs. general cache instances
– recommends rcm esp. since hard to tell if redis cluster nodes up-to-date
– managed with yinst, chef, jenkins/screwdriver
– not full-on HA, but good enough for easy upgrades
– no redis cluster talks in 2016, so this is one of the first. very well-received.
Operationalizing Redis At Scale
Brian Ip, Software Engineer, Square
– SSL everywhere with ghostunnel
– read-only using min clients 1000
– redis is ro-cache, but backups made for operational reasons like cache warming
– LXC environment
– looking forward to slides!
Building Large, High Performance Databases with Redis using Flash Memory and Emerging Hardware
Cihan Biyikoglu/Frank Ober
(Redis Labs/Intel)
– RAM is expensive, SSD is cheaper
– so use SSD in tiered storage where redis is aware
– Intel has Optane SSD series for this (gave one away to audience member)
How Roblox keeps millions of users up to date with Redis Pub/Sub
Peter Phillips, Senior Software Engineer
– crazy numbers – 1 million persistent chat connections using web tier, 4 redis nodes
– Windows environment
– try to keep connection count down to 1 per device but not easy
– pub/sub used for cache invalidation
– some attendees said “best technical talk”
Using Redis at scale at Twitter
Rashmi Ramesh, Senior Software Engineer
– LB proxy layer in front of redis
– online rebalancing for qps avg. 600k qps, not size
– nighthawk is internal name
– still use memcached for flat values
– patches or designs for identifying hot keys, recommends caching in app.
Thursday Keynotes
Doing More with Redis
Ofer Bengal and Yiftach Shoolman, Co-Founders, Redis Labs
Real-time intelligence with Spark and Redis
Reynold Xin, CTO, Databricks
– great talk using the Google NIPS 2015 machine learning diagram: LHS can be implemented Spark, RHS in Redis.
Microservices and Redis
Chris Richardson, Founder, Eventuate.io
– “microservices reflect business units”
– for distributed data modelling, look up Saga and CQRS
Chris is a world-class deep thinker. Check out his slide deck.
Trends in DevOps
Charity Majors, CEO, Honeycomb.io (ex-Parse, ex-Facebook)
Listening to Charity is definitely a unique experience. Very brassy, but also dead-on accurate.
– Redis is “Memory as a Service”
– monitoring vs. observability
– what’s changed?
– infrastructure and storage complexity is growing
– do you know who the DBA is? If not, it’s you! 🙂
– distributed system: your system is never entirely up
– your green dashboard is a lie
– there are no more easy problems. there are only hard problems. (combinations, and you fixed the easy ones)
– context is everything. aggregates throw away data
– dashboards must be people-first and consumer-friendly
– don’t make everyone be an expert
– don’t promote swdevs who don’t know operations: #OwnYourProblems
Thomas Middleditch
(Star of the hit HBO show “Silicon Valley”)
Thomas is a professional comedian/actor who portrays an alpha nerd on TV. IRL, he is a gamer and did some QBasic programming as a kid. He fielded questions from the audience and was generally witty and entertaining. His TV persona is a composite of nerds, not inspired by one particular person. One of the funniest things he did was to read the conference poster buzzwords and exclaim, “I want to get me some more pam-auth.” 🙂
Thursday Talks
Utilizing Redis in a High-tech Ad Traffic Stack
Rahul Babbar, Timesinternet (Advertising in Colombia, South America)
– 99% percentile is 2 ms
Best practices (also see slides):
– do BGSAVE on masters sequentially to reduce IO hotspot
– ensure TTL is set
– renamed destructive commands so that rogue client programs don’t have accidents
– set idle timeout
– hz patameter
– have an app strategy in case of redis slowdown or failure
Machine Learning in Real-time with Redis-ML
Shay Nativ, Redis Labs
– Redis’ plugins are a great way to add features
– one plugin is ML, with several models available
– one adtech user was able to use the tree model to reduce from 1200 Java app servers to 40 Redis servers
– slides
A Collaborative Canvas Using Bitfields in Redis
Daniel Ellis, Reddit
– about Reddit’s April Fools collaborative painting tool
– initial data model for Cassandra, not really a good fit
– changed to Redis using one key with 15 MB string representing one nybble for each pixel
Multi-Master Redis : A Deep Dive
Elad Ash, Redis Labs
– current Redis is just LWW (last write wins)
– for multi-master, it’s desirable to have some kind of write conflict strategy
– each data type is getting “vector clock-ized”
– implemented as a Redis plug-in
– available now for beta, to be released in 2018
Geofencing Using Redis Geospatial Queries
Matthew Hicks, Appboy
– Appboy library is installed on up to a billion mobile devices/apps, bundled with various apps
– use Redis built-in commands to calculate distance from an advertiser location to the current user location for ad targeting (circle diameter)
– For example “is a passenger approaching a branch location? if so, push an ad.”
– not difficult to implement using Redis, but need detailed user requirements to do so in a relevant way.
Geospatial Indexing At Scale: the 10 Million QPS Redis Architecture Powering Lyft
Daniel Hochman, Lyft
– good talk.
Getting There
From peninsula, take Caltrain to 4th and King then Muni to Powell Station, or BART from Millbrae Station to Powell Station.
This is a follow-up to my 2016 post.
Not much new in mid-2017 for online shopping sites, but Honestbee is doing the first grocery home delivery now in the Philippines. The dominant grocery chain is Robinson’s, but they don’t offer home delivery. Honestbee will mainly shop for clients at Robinson’s and other specialty food providers.
The Manila area is interesting for home delivery for 5 reasons:
Also some recent payment processing changes to checks, credit cards and Philippines Check Image Clearing System (CICS):
Family Ties Bind Philippine Banks
Miscellaneous
Note: NAIA Airport cancelled the P750 (USD $17.00) airport departure fee a while ago, and still hasn’t replaced it. They seem to have working AC throughout the airport now, as this was the most pleasant departure that I can remember from Manila.
When looking at geographic distribution you can see the Philippines makes up the 5th spot by [Les Paul] search interest (this place was held by Norway for Gibson), highlighting the premium vs. standard marketing (and pricing). We think it’s a great thing that Epiphone guitars are accessible to lower income markets.”
Apple Pay and the rise of the five-party network
For Western Union, Refugees and Immigrants Are the Ultimate Market
W: Honestbee
bloomberg.com: Last year, $1.2 trillion in mobile payments were made through WeChat, with each user averaging about $85 a month in peer-to-peer transfers
Visa USA Interchange Reimbursement Fees
jhead
Xend Business Shipping
Online shopping in Africa doesn’t work because of this web form
Landmark “corporate” technical blog post from Willy Tarreau titled “Truly Seamless Reloads with HAProxy – No More Hacks!”
Summary is that there is progress concerning HAProxy zero-downtime reloads on linux, even under heavy connection load.
Interestingly, microservice owners doing frequent updates and reloads have been driving the interest behind this.
Thanks to Willy for both the improvements, and the very detailed (and lengthy) blog post explaining the history and status of the fixes.
haproxy.com: DNS for Service Discovery in HAProxy
github.com: AirBnB Synapse
Percona Live 2017 was again a well-organized and well-attended Open Source database conference this year in Santa Clara.
VividCortex actually had the primary expo real estate for the first time, as well as delivering multiple talks. Congrats!
Keynote Videos
Conference Videos
Some other interesting topics were:
These were the BoF Lightning Talks:
The MyRocks BoF was very popular and lively. RocksDB is used internally at Facebook, and is in MariaD 10.2 as MyRocks. Mark Callaghan answered several questions and talked about Linux IO accounting as being inaccurate for SSD according to his customized version of fio, and generally lacking in performance-tracking features. Some audience members acted keen to adopt MyRocks, but it sounds like early days for public use. There was a session on MyRocks.
Thanks to Continuent for hosting their “customer appreciation dinner.” It was interesting talking to the CEO/owner, Eero Teerikorpi, about the history of Continuent over the years, MySQL replication and other major features used by enterprises.
Eero sold Continuent to VMware as a component in their cloud hosting plans, but when those plans were cancelled bought it back in 2016. So Continuent is independent again. (Robert Hodges and Giuseppe Maxima stayed at VMware.) They plan to invest more in their replication product. (I’ve used it previously for MySQL to Vertica replication.)
TheRegister has an article about a Cisco recall on the ASR 920 Series Aggregation Services Routers PSUs
I enjoy reading TheRegister, but this article is a little light on research.
Looking around online, it turns out that PSUs are often designed for certain altitudes when using air as a dielectric and for cooling, and 2,000 meters is common. However, the current ASR 920 datasheet specifies 4,000 meters, hence the recall.
The real story is:
I attended 2 days of the AWS Loft SF Database Architecture Conference.
The software scalability work that Amazon has done on databases and caches, especially the Aurora distributed MySQL and Postgres databases, is very impressive. (DBA note: do careful acceptance testing of any distributed database.)
Executive Summary:
Some of the talks I went to:
What’s New in Amazon Aurora for MySQL and PostgreSQL
by Kevin Jernigan, AWS Manager of Tech Product Management, DBS
– very impressive engineering work by AWS engineers – complete internals modernization of MySQL and PostgreSQL
– split Open Source MySQL and PG code into two components (SQL engine and SAN storage modules)
– rewrote algorithms (btree => Z-index), log replay (max. 1.5 seconds) and locking code pushed down for MySQL
– no checkpoints, so 3x less jitter (query latency variation) since data is written to network, so no disk stalls
– Aurora MySQL 5x faster or more
– Aurora PG 2x faster (already well-written internals)
– 6 nodes, 4 required for quorum.
– my opinion as a DBA is that SANs are always a problem, so carefully evaluate this.
– story on edge cases are not well understood yet, critical for operating large distributed database
– you still choose and instance size for IO/CPU since that’s how their billing works.
The speaker was well received as being authoritatively technical by my co-attendee. I found the MySQL comparisons to Aurora a little contrived as being the worst-case configuration of MySQL. ie. I can fail over in 2-5 seconds with master-master and a load balancer, as compared to his discussion of 30 seconds to a minute or more.
What’s New in Amazon RDS for Open-Source and Commercial Databases
by KD Singh, AWS Partner Solutions Architect:
– MariaDB, Oracle 12c now supported
– Read slaves use regular async replication from product, so could lag. Your app needs to handle it.
– HIPAA, ITAR, USgov, UKgov, SGgov, PCI Level 1 seller approval
– during RDS failover, your app must be programmed to reconnect automatically. expected downtime is about a minute for the CNAME => IP address to update
– SQL Server limit is 4 TB, supports AD, .bak files
– 1-second monitoring now included in Cloudwatch
– “pick the smallest you think will work, and migrate when you need a bigger instance size”
– local timezone now supported everywhere
Migrating to Amazon RDS with Database Migration Service
by Dhanraj Pondicherry, Senior Manager of Solutions Architecture, AWS
– SCT (Schema Conversion Tool) then DMS (Data Migration Service)
– Successfully used for Oracle => PG by marquis clients like Shaadi.com
– may need careful VPC setup for source or target
– SCT lists count of tables, SPs so easy to eyeball for QC
– inbound bandwidth is free, so migration is very cheap within AZ
– DMS requires a CDC method to be enabled on the source, like Oracle CDC or MySQL binlogs.
– very impressive effort on these migration tools, with almost any combination of source and target possible now, including OnPrem, EC2 Classic, RDS, Aurora, and Redshift.
– “for your migration tool, pick the smallest you think will work, and migrate when you need a bigger instance size”
Amazon Aurora and Amazon Database Migration Service
by Joyjeet Banerjee, Solutions Architect, AWS Migration Lab
– download and try SCT (OLAP option is for Redshift)
– DMS Online Lab: qwiklabs.com/focuses/2965
Introduction to Amazon DynamoDB
by Sean Shriver, NoSQL Solutions Architect, AWS
– is a key-value store, key up to 2KB, is a string
– not currently related to original Dynamo paper. Most of the authors highly promoted.
– 1 partition, 5 GSIs, 5 LSIs (per partition)
– GSI and LSI separate tables
– GSI need to provision IOPs
– charged 1k writes 4k reads. reading from GSI reduces io cost
– partition key uses consistent hashing
Amazon Elasticache
by Darin Briskman, Developer Evangelist, AWS
– used to be based on memcached, now redis
– average operation 480 microseconds for 4 KB object, 240 microseconds for 1 KB object
– max. 3.5 TB per server, in clusters of 15 servers. 20 million reads per second, 4.5 million writes
– 300,000 TPS
– HA is 1,000 little details. 999 doesn’t count
– “Fast Data” sub-millisecond requests for IoT, mobile real-time info
– Alexa 1,500 ms budget, but 1,000 ms is network trip. So 500 ms for calculation.
– Alexa is DynamoDB+ElastiCache
memcached challenges:
1. no persistence
2. no HA
3. race conditions on threads
Thus Redis (“Remote Dictionary Service”)
Oracle, SQL Server, Mysql then Redis
– AWS-redis persistence via snapshot to S3
– snapshots to 90% of RAM (even 95%) network copied to alternate node. Allowed 20 snapshots per day.
– replication for HA. 30 seconds to failover usually
– primary and replica (don;t like master-slave sounds)
– 1 ms in same AZ, 2-3 ms different AZ
– 55 DCs in an AZ in US-EAST
– new Intel chips 34% faster
– no cross-AZ data transfer costs, so similar cost to Classic EC2
– don’t use T2 for prod. Use R or M.
– key CRC16
– promotion is to last-written replica, timeout of 15 seconds in case of network problem
– string key up to 512 MB, really just binary
– hash, set, list, geo, hyperloglog
– could use lambda to notify of OnPrem database update and invalidate Elasticache row
– IGA Works/Adpopcorn is Korean mobile business platform. Moment scoring on mobile users, including when to show ads
– Expedia’s real-time analytics with Dynamodb was 35000 writes, down to 3500 with elasticahe, 6x savings. 200 million messages daily.
– only a few airlines overseas, but lots of hotels at the destination. mom and pop agencies refreshing expedia also as their backend. 100 most popular routes are 50% of queries. TTL 24 hour, updates 10 time per day
– https://www.youtube.com/watch?v=ie4dWGT76LM
– one day of work and 5 days of testing
– beyond time of year caching, you may know the most popular teams/items
– or cache the whole database if small enough
– cannot add another node for say 5 shards to 6 shards because could lost data now. maybe later.
– “you don’t have to do anything. when you woke up later, it’ll be there.”
ElastiCache Best Practices
– set reserved-memory to 90% so writes can fit in without eviction
– swap usage should be zero
– position a read replica in another AZ for HA
– primary with 2 replicas is 5×9’s
– avoid KEYS and other long-running commands
– not needed for like 1 MB of data
– 50% – 90% reduction in cost
– former Solution Architect at IBM for 20 years. At AWS, allowed to recommend ways to save money.
Everything You Need for a Viral Game, Except the Game
by Darin Briskman, Developer Evangelist, AWS
– use Dynamodb and redis
– wechat runs on redis
– publish and subscribe redis commands: subscribe to a channel then publish to it
– twitch offers hosted chat
– CloudTrail tracks every action including DBA-level access to RDS in JSON
– talk to your Solution Architect, available to every AWS account holder
Elasticsearch
by Darin Briskman, Developer Evangelist, AWS
– most downloaded Open Source app after linux kernel
– nice REST interace
– same code as Open Source, but manageable in AWS
– AWS is green-blue instead of red-black
– can resize
– AWS answers
– Centralized Logging
– CloudSearch is Solr
Hands-on Labs: Amazon ElastiCache
by Darin Briskman, Developer Evangelist, AWS
– https://s3-us-west-2.amazonaws.com/fastdata/ElastiCacheLab.zip
AWS Talks link
https://s3-us-west-1.amazonaws.com/architectureweeks/Database/SF+Februray+21-23%2C+2017/Database+S3.pdf
Kudos to Kevin Jernigan and Darin Briskman for their excellent Aurora and ElastiCache talks – the best database talks I’ve ever seen.
This version of the AWS Loft is nice as far as “pop-up” conferences go – everything is hosted on the 2nd Floor, so no sprinting up and down stairs every 30 minutes.
Amazon Aurora Under the Hood: Fast DDL
Amazon DynamoDB Accelerator (DAX)
Tips
– Windows 10 has openssh support via Ubuntu. No Putty needed.
Getting there: 1446 Market St. SF. Take Muni K or T line to Van Ness station. or take Castro bus on Market St.
The benefits of using Swagger/OpenAPI are to maintain consistent API specifications, validation and documentation.
And the Swagger UI documentation tool initially looks very attractive cosmetically. However there’s no publishing or privacy restriction features. The reason is that the Swagger API spec itself doesn’t support publishing controls.
That’s fine for Open source authors and most internal-only corporate users, but commercial sites will be unhappy without more control.
So it’s important to decide well before ship date if the Swagger UI will work for you, or if you need to find another solution (ie. you might find it to be “more hole than donut.”)
| # | Swagger UI Issue | Solution |
|---|---|---|
| 1. | Swagger UI makes your Swagger API spec file downloadable by end-users | A half-step is to use Basic Auth for viewing it, but authenticated end-users can still save it to disk. A plan would be to run it server-side. |
| 2. | Swagger UI by default sends your Swagger API spec file to an external validation service | Fixable. validatorUrl: null |
| 3. | the “Try it out” buttons” are active, letting anybody too easily send GET, DELETE, PUT, POST, PATCH commands to your server, whether the request makes sense or not | Fixable. See supportedSubmitMethods parameter. |
| 4. | the default branding is for the Swagger project | Fixable. Read docs or just use Firefox Inspector on Swagger UI header and change the CSS/JS. |
| 5. | the UI tool is nice, but even nicer is professionally written text with detailed examples | Not fixable. Outside the scope of Swagger UI. |
| 6. | the UI tool does not publish to a static file. Most commercial publishers want to provide a PDF file. | Not easily fixable, though you can load your Swagger spec file into the Swagger Editor, drag the left frame further left, “print as PDF” and edit the text with Libre Office. |
| 7. | No ability to restrict on displaying paths or request methods | Not easily fixable. You could export a minimal Swagger API spec file just for use with Swagger UI. |
| 8. | Add username and password auth | Easy. Just add a securitydefinitions block in your Swagger spec file and define the key name in Swagger UI’s index.html. Note that multiple auth methods require showing and clicking multiple “Authorize” buttons (see screenshot above) since the Swagger specification considers multiple auth methods to logically OR and your app has to sort them out. |
| 9. | Add API key auth | Easy. Supported by current versions. |
| 10. | URL bar shows address of Swagger spec file | Fixable, but end-users can still download your spec file: document. |
| 11. | Themes | Some are available at swagger-ui-themes |
| 12. | There can be one “body” parameter at most. | The Swagger spec only allows one body element (formData or JSON) while you may want to allow both. Some parsers enforce that, and some don’t. |
| 13. | Swagger UI markdown is broken/incomplete. | As of Feb. 2017, markdown support is missing numbered lists, italics, tables and more. Please post a comment when numbered item lists and line continuations work. See Issue #825 for more context. |
* by “not fixable” and “not easily fixable” I mean “total Swagger UI re-design and rewrite required.” 🙂
Getting Started with Swagger UI
Customizing the Swagger UI
Test Parameter
Security Tokens
How to break swagger 2.0 JSON file into multiple modules
Tom Johnson’s Tutorial
Tuan’s Tip to Add Username and Password (Check spelling carefully)
A Visual Guide to What’s New in Swagger 3.0
Note: if you google or stackoverflow for help, ignore any bug reports about Swagger UI before 2016.
Swagger Editor Custom UI
Spectacle
blog.novatec-gmbh.de: The problems with Swagger, HN discussion
Top 7 Myths about HTTPS and Browser Caching
Ask HN: What’s the best way to write an API spec?
robwin.eu: Documentation of a REST API with Swagger and AsciiDoc
Document your Already Existing APIs with Swagger
When you need to edit a PDF, you really need to edit a PDF.
The free Preview app that comes with Mac OS X can annotate and do some simple operations on PDF files, but does not have a feature to edit the actual text.
I tried the following free or trial PDF editors on Mac OS X 8.5 for editing text on a 30-page Firefox “print to PDF.”
| Product | Recommendation | Notes |
|---|---|---|
| Libre Office for Mac (Open Source) | Recommended | Does a great job of editing text |
| Inkscape (Open Source) | Not Recommended | Can only edit first page “due to SVG spec” unless you install plugin |
| Mac OSX Preview (Manual) | Not Recommended | Can remove PDF pages, copy text and add text. |
| iSkysoft PDF Editor Trial | Not Recommended | Can edit text, but inserts large yellow watermark on save |
Best Super Bowl that I can remember, with first Super Bowl Overtime.
The New England Patriots (QB Tom Brady) came from behind to win 34-28 over the Atlanta Falcons (QB Matt Ryan).
Atlanta scored 3 TD’s in Q2:
Patriots got a 41-yard field kick for the 3 points.
Lady Gaga’s half-time performance was good, with surprising acrobatics.
Then the Patriots scored 31 unanswered points to win.
Commercials ($5 million for 30 seconds) weren’t too good, although John Malkovich trying to get his vanity domain from a domain squatter was pretty good (squarespace.com.)
Seemed like mostly cell phone, car and VR-related ads.
| Organization | Link | Purpose |
|---|---|---|
| Apple | http://captive.apple.com/ | non-SSL ProbeURL for WiFi hand-shaking. Replaces success.html. |
| http://www.google.com/ncr | force plain English site (“no country redirect”) | |
| IANA | http://www.example.com | official domain for examples in documents |
| IANA | http://www.example.org | official domain for examples in documents |
How to fix “SuccessSuccess” Wi-Fi issue on MacOS X Mavericks temporally
The big news from the SpaceX launch of 10 Aireon Iridium 2G “Next” satellites is that ADS-B was also included on each satellite.
ADS-B is used for tracking and sending ATC digital information to airplanes. The FAA has mandated that almost all aircraft will install ADS-B transceivers before 2020, at a cost of $5,000 to $1 million per airplane, plus downtime.
Since there’s 150,000 registered US aircraft and thousands of foreign airliners, that just isn’t going to happen with the existing number of Mx shops and remaining 1,080 days. 🙂
These are fairly large satellites at 860 kg each:
Perl on Linux supports the POSIX C clock_gettime() function to get the monotonic time (always increasing system time, except for variable overflow) values:
Comparing monotonic time values:
Disadvantages of clock_gettime() over time/gmtime:
use strict; use diagnostics; use Time::HiRes qw(clock_gettime CLOCK_REALTIME CLOCK_MONOTONIC); my $realtime = clock_gettime(CLOCK_REALTIME); my $mono = clock_gettime(CLOCK_MONOTONIC); print "realtime = $realtime, monotonic = $mono\n";
$ perl /tmp/clock.pl realtime = 1483451061.64625, monotonic = 4536159.37919642
Perl – Time::HiRes
clock_gettime(3) – Linux man page
Erlang – Postscript: Time Goes On
lwn.net: The leap second of doom
SO: CLOCK_MONOTONIC Max value
SO: How do I get monotonic time durations in python?
SO: Linux clock_gettime(CLOCK_MONOTONIC) strange non-monotonic behavior
SO: Is CLOCK_MONOTONIC process (or thread) specific?
How the NYE leap second clocked Cloudflare – and how a single character fixed it
Time::Local
W: Swatch Internet Time (Beats)
PSA:
“Changes to eBay Bucks Rewards Program starting January 1, 2017
Effective January 1, 2017 the Base earnings are changing from 2% to 1%.”
Guess I’ll be advising sellers to wait for 8% or 10% eBay Bucks days.
Basic Economy Fares Don’t Lower Ticket Prices, They Increase Ticket Prices
The Champions of the 401(k) Lament the Revolution They Started
From the MicroProfile FAQ:
“The MicroProfile is a baseline platform definition that optimizes Enterprise Java for a microservices architecture and delivers application portability across multiple MicroProfile runtimes.
The initially planned baseline is JAX-RS + CDI + JSON-P, with the intent of community having an active role in the MicroProfile definition and roadmap.”
Looks more like a a REST bundle to me that avoids fixing Java’s inherent flaws:
microprofile.io: Home, FAQ
theregister.co.uk: MicroServices-friendly Java lands on Eclipse
eclipse.org: Eclipse MicroProfile
Avweb has an interesting article on ‘Extreme Maneuvering’ about practical applications of the FAA commercial aerobatic maneuvers (chandelles, lazy 8’s, steep spirals) that mentioned “rolling G’s.”
A rolling G occurs when you maneuver an aircraft in more than one axis at a time, causing the airframe or wing to twist. The rolling G design limit is considered to be 2/3 of the normal G limit, according to FAR 23.
Although I performed the commercial maneuvers during my Commercial Airplane practical test and Citabria checkout, I wasn’t really aware of two things:
The asymmetric lift, resulting in a torque, caused by the ailerons travelling up and down simultaneously with yawing and pitching maneuvers is believed to have caused several airshow accidents in older airplanes, shearing the wing spar. Contributing factors are the acceleration rate of the control movement, airspeed above maneuvering speed, Va, and wing and fuselage harmonics. Sideslip also affects G limits.
It would be difficult to calculate actual rolling G limits without destroying several aircraft to build a mathematical model. There are a number of reasons for that, but primarily the problem is that dynamic torque must be calculated for multiple types of members, including spars, fuselage skin, and especially attach points. The latter is tricky because attach point hardware may be very strong in one axis, and very weak when loaded off-axis (or corroded.)
pprune.org: Why is Rolling G dangerous?, Normal G limits vs Rolling G limits?
W: Chandelle,
Ice and Tail Stalls
Current consumer ecommerce weather report for Manila in Dec., 2016 …
US ecommerce sites – Just Say No
Manila sellers are wary of Facebook Pages commissions on retail listings, and “meh” on ebay for the same reason. Craigslist is free but doesn’t have any mindshare in Manila.
Southeast-Asian ecommerce sites – Just Say Yes
So they’re going with Lazada, probably #1 in Manila, or Shopee, which is ad-supported, and Carousell.
Shoppee is owned by the Garena Group of Singapore. They have registered country-specific top-level domains (TLDs) for each Asian country supported.
How Shopee works:
I got a tour of the Shopee office. It’s similar to Silicon Valley start-up offices, but has a staffed reception area. 🙂
Car Hire
The most popular car apps are Uber and Grab. Riders use car apps because buses and the MRT (train) are inadequate for longer commutes, and unsafe due to petty criminal gangs. Drivers see car apps as a way to pay off their car loan, and to kill time, due to rampant underemployment.
Uber is rumored to have increased Manila traffic by the equivalent of 19,000 cars. Rush hour used to be 7 am to 9 am and 5 pm to 8 pm. Now it’s 6 am to midnite. (In the USA, there have been mixed reports of increased traffic. SF is reported to have problems, while Phoenix less.
Uber passengers used to cancel arriving sub-compact hatchbacks like the Toyota Wigo (MSRP USD$10,000) in favor of sedans, but the Wigo is getting more respect 2 years after market introduction.
Garena’s Shopee could be on its way to beating Carousell in Asia
Percona supports Percona Xtradb Cluster 5.7 on CentOS 6 and CentOS 7, but not CentOS 5.
You can install the RPMs or tarball binary, but on start will see various package dependencies that can’t be resolved on openssl.0.10 and others.
So your options are:
yum install Percona-XtraDB-Cluster-server-56
Here’s the build instructions that compiled for me with gcc 4.1.2:
1. So download and install cmake 2.8.2 or higher from source first:
yum remove cmake wget --no-check-certificate https://cmake.org/files/v3.7/cmake-3.7.1.tar.gz tar zxvf - < cmake-3.7.1.tar.gz cd cmake-3.7.1 ./bootstrap && make && make install cd ..
2. Download and install boost 1.62 from source:
yum remove boost wget --no-check-certificate https://sourceforge.net/projects/boost/files/latest/download?source=files yum install p7zip 7za x boost_1_62_0.7z cd boost_1_62_0 ./bootstrap.sh --prefix=/usr/local ./b2 install cd ..
3. Build Percona-XtraDB-Cluster-5.7 source like this:
wget --no-check-certificate https://www.percona.com/downloads/Percona-Server-5.7/LATEST/source/
cd Percona-XtraDB-Cluster-5.7.16-27.19
# remove new gcc 4.4 flag -Wvla:
# -Wvla
# Warn if variable length array is used in the code. -Wno-vla will prevent the -pedantic warning of the variable length array.
perl -i.orig -p -e 's/-Wvla//g' `find . -name maintainer.cmake`
cmake . -DMYSQL_DATADIR=/var/lib/mysql
mv boost_1_59_0 /tmp
# fix the boost and gcc version errors. Just replace with your versions.
vi cmake/os/Linux.cmake +27
vi cmake/boost.cmake +265
# insert 2 "out-of-scope" macros os_compare_and_swap_thread_id and os_compare_and_swap from storage/innobase/include/os0atomic.h into these 2 source files:
vi storage/innobase/lock/lock0lock.cc +1904
vi storage/innobase/trx/trx0trx.cc +204
# define os_compare_and_swap(ptr, old_val, new_val) \
__sync_bool_compare_and_swap(ptr, old_val, new_val)
# define os_compare_and_swap_thread_id(ptr, old_val, new_val) \
os_compare_and_swap(ptr, old_val, new_val)
make -j 8
make test
# the new server is located at sql/mysqld
make install
# note that 5.7 has a new grants schema, so your old database won't work until upgraded
# in /etc/init.d/mysql, bindir=/usr/local
I don’t often go to the movies, but saw ‘Rogue One’ with a date.
The first half seemed kind of slow and disconnected, dealing with various rebel assassination plots (!) on Jedah and Eadu. Good visuals but weak story-telling.
The protagonist, Jyn Eros, portrayed by Felicity Jones, must be a pretty bad actor to have her mother killed in front of her, yet convey being unsympathetic and uninvolved throughout most of the film – I’d rather watch paint dry.
However, the second half dealing with the invasion and destruction of the Imperial base at Scarif and testing of the Death Star was riveting.
Darth Vader’s brutal but ultimately futile light-saber fight scene at the end will please action fans.
And seeing a youthful Princess Leia (CGI) at the end receiving the Deathstar plans was a nice surprise.
The rebels’ companion robot, a re-programmed Imperial model named K-2SO, was intelligent and funny enough to be unsettling. Admiral “Fishlips” Raddus a Mon Calamari, also provided comedic distraction.
W: Rogue One
IMDB: Rogue One
Can we talk about that final Darth Vader scene in Rogue One?
bbc.com: Rogue One is Star Wars for Better and for Worse
Rogue One: Meet Admiral Raddus, thecharacter inspired by Winston Churchill
Behind the scenes of ‘Rogue One’ with director Gareth Edwards
The story behind Princess Leia’s hairstyle
Keywords: General Fishlips
The sorry tale of the Cessna 162 Light Sport Aircraft (LSA) has finally concluded. The remaining 80 airplanes have been crushed with a backhoe outside the the Chinese factory, including the installed engines and avionics.
It’s believed that liability insurance and parts support didn’t pencil out for the accountants. Crushing solved the liability problem, and also any agreements with suppliers like Continental and Garmin prohibiting resale.
Kind of a shame they couldn’t have sold them to the Chinese government for $1 for use in flight training in exchange for indemnity from lawsuits.
Of the original 1,000 projected orders, 200 were actually delivered and 80 crushed.
There were many problems with the 162:
Cessna Scraps Unsold Skycatchers
Crushing More Than Airplanes
Skycatcher’s Demise: Barely a Ripple [2013]
W: ADHARS
Wow. Who knew the little TextEdit application supports .doc, .docx and PDF formats?
This actually worked for me today:
So you can do light but professional documentation, invoicing, etc. without installing additional software.
For multi-column and chart formatting, just use the the Format => Table… menu option, similar to old-school HTML layout. You can set the table cell borders to 0 pixels to make them disappear.
Bonus tip: Preview, which is also included with Mac OS X, can be used to professionally annotate images. If you’re a manager or engineer, you will love the results. Just click on Tools … Annotate.
osxdaily.com: Opening DOCX Files on a Mac, Without Microsoft Office
There’s basically 3 ways to store online data, where “storage” includes block and/or network locations:
CleverSafe struggled along with VC funding until Oct. 2015, when it was bought by IBM for $1.3 billion.
“At a glance, this seems like a clear explanation of using standard SIMD instructions to solve the problem, but I think the landscape has changed since this was written such that there are now better approaches.
In 2010, Intel released processors with a dedicated instruction for “packed carry-less multiplication.”
Unfortunately, the early implementations (through Sandy Bridge) were slow, and could be beaten by combining other SIMD operations as shown in this paper.
With the Haswell generation released in 2013, though, PCLMULQDQ got much faster. Instead of being able to complete one instruction every 8 cycles, it became possible to finish one every 2 cycles (inverse throughput went from 8 to 2). This 2015 paper “Faster 64-bit universal hashing using carry-less multiplications [PDF]” shows the difference this makes:
If you are looking for an explanation of how the problem could be solved with the basic building blocks of SIMD, the 2013 Plank, Greenan, Miller paper might be a good resource. But if you are hoping to implement high performance solution for modern processors, the 2015 Lemire and Kaser paper is probably a better starting point.
(This is with the caveat that I don’t actually understand the theory or terminology of Galois fields, and maybe there is something about applying it to Erasure Coding that makes the faster PCLMULQDQ approach inapplicable.)”
FAST-2013: Screaming Fast Galois Field Arithmetic Using Intel SIMD Instructions (2013) [PDF]
OpenStack Swift
register.co.uk: IBM Buys CleverSafe
Patent Troll Kills Open Source Project On Speeding Up The Computation Of Erasure Codes
Keywords: Reed, Solomon, Galois, GFC.
apis.json is a site discovery format like sitemap.xml, but for your APIs. It is an open project to create a new standard by some ambitious API evangelists .
Steps to create your own apis.json file:
apis.json: validator, Google Group, github, Proposed Intranet Properties
API Evangelist
ProgrammableWeb.com
JSONLint
github: OpenAPI Specification
I had a chance to see the GeoCities Exodus 1 cage in 2000 when I was at eBay Payments.
It used LaCie JBOD stacked to the ceiling as storage devices, and a 3′ diameter floor fan to move the hot air to other customers’ cages. 🙂
Below is some photos of one of their cages with Sun and Netapp gear:
The GeoCities Cage at Exodus Communications [1999] HN Comments
Note their use of Veritas Volume Manager.
Until around 1998, linux did not have a journalled filesystem. I started evaluating Reiserfs 3 on Suse Linux at that time on my personal machine. A Suse salesrep a decade later refused to believe that anybody in the USA could have been using Suse back then. 🙂
The other cage from 2000 that left an impression on me was About.com’s, which had a Sun E10k server ($2 million each fully populated). I don’t think they ever launched a product, yet they had the same equipment as eBay’s main cage.
I was talking to some other sysadmins with gear at Exodus 3 and 4, and they mentioned a lot of customers also built out their colo but never launched.
Dedicated Internet Access & Hosting Agreement between Exodus and GeoCities
W: Exodus Communications
After all, when you yoke a bunch of water buffalo together, your team is only as fast as the slowest buffalo.
But I find it fascinating and convenient that they’re doing all that distributed file system testing for me. Thanks, guys! 🙂
On the plus side, supporting a distributed file system is almost possible on homogeneous hardware …
Here’s some free consulting from somebody who works on x,000 to xx,000-server data centers:
“There is a threshold of performance on the cloud and if you need more, you will have to pay a lot more, be punished with latencies, or leave the cloud.”
GitLab.com: How We Knew It Was Time to Leave the Cloud
HN discussion (with Cloud Apologists)
Proposed server purchase for GitLab.com HN
It’s a short-range single-engine 4-seat passenger jet for $1.5 million with a parachute that can be operated for $660/hour.
I’ve been following the news on this during the last decade of development. General Aviation (GA) moves slowly, but the SF50 and HondaJet show that eventually small aircraft do get certified.
The numbers:
Obviously it’s intended for existing Cirrus owners who want to step up to a jet.
It uses a Williams FJ33-5A jet engine, similar to the original $800,000 Eclipse 500 jet plans. Williams is a cruise-missile manufacturer, so has lots of manufacturing experience with small jet engines, but not much experience with passenger airplane maintenance.
China has bought most of the American GA manufacturing capacity out of bankruptcy in the past decade to position itself for growth in the emerging Chinese civilian market, including Continental Engines (2010), Superior Air Parts (2008), Mooney (2013) and Diamond Canada (2016.)
So far, that has resulted in much-needed investment, though it’s unclear what the long-term implications are for the USA.
avweb.com: Cirrus SF50 Vision Jet: Learning From the Past
aopa.org: Hourly operating costs of 45 jets compared
Mooney’s Fortunes Tied to China
Luxury VIP jets: How the super-rich fly
avweb.com: Checking The China Acquisition Score Card
Cirrus Delivers First Vision Jet
httperf is an easy-to-use but powerful GPL2 command line (CLI) stress and load testing tool for linux.
Installing httperf
CentOS 6:
yum install httperf
CentOS 7:
wget http://ftp.tu-chemnitz.de/pub/linux/dag/redhat/el7/en/x86_64/rpmforge/RPMS/rpmforge-release-0.5.3-1.el7.rf.x86_64.rpm rpm -Uvh rpmforge-release-0.5.3-1.el7.rf.x86_64.rpm yum install httperf
Running httperf
$ httperf --server www.example.com --uri /index.php --print-request --print-reply -d10
If you see non-200 HTTP responses, like this 301 example response below, then you need to ensure you have the correct –uri parameter:
httperf: warning: open file limit > FD_SETSIZE; limiting max. # of open files to FD_SETSIZE httperf: maximum number of open descriptors = 1024 SH0:GET /index.php HTTP/1.1 SH0:User-Agent: httperf/0.9.0 SH0:Host: www.example.com SH0: SS0: header 83 content 0 RH0:HTTP/1.1 301 Moved Permanently
You can ignore the open files warning – it’s a bug in httperf. Just keep the load under 200 connections, or compile your own version from source.
Now we’re ready to do concurrent testing:
$ httperf --server www.example.com --uri /index.php --num-conn 20 --num-cal 10 --rate 2 --timeout 5
httperf --timeout=5 --client=0/1 --server=www.example.com --port=80 --uri=/blog --rate=2 --send-buffer=4096 --recv-buffer=16384 --num-conns=20 --num-calls=10 httperf: warning: open file limit > FD_SETSIZE; limiting max. # of open files to FD_SETSIZE Maximum connect burst length: 1 Total: connections 20 requests 200 replies 200 test-duration 10.675 s Connection rate: 1.9 conn/s (533.8 ms/conn, <=4 concurrent connections) Connection time [ms]: min 1175.2 avg 1266.2 max 1728.3 median 1179.5 stddev 179.3 Connection time [ms]: connect 63.4 Connection length [replies/conn]: 10.000 Request rate: 18.7 req/s (53.4 ms/req) Request size [B]: 73.0 Reply rate [replies/s]: min 18.2 avg 19.1 max 20.0 stddev 1.3 (2 samples) Reply time [ms]: response 120.3 transfer 0.0 Reply size [B]: header 238.0 content 0.0 footer 0.0 (total 238.0) Reply status: 1xx=0 2xx=0 3xx=200 4xx=0 5xx=0 CPU time [s]: user 2.36 system 8.30 (user 22.1% system 77.7% total 99.9%) Net I/O: 5.7 KB/s (0.0*10^6 bps) Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0 Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
Always check for non-zero error counts.
Going Pro
After you're comfortable using httperf, here's how to take it to the next level:
git clone https://github.com/httperf/httperf.git cd httperf # read Readme.md sudo yum install automake openssl-devel libtool libtoolize --force autoreconf -i automake ./configure make sudo make installread the links below and configure open files, port range and TCP timeout
strace httperf ..., we see that httperf does polling with the select() system call. Hmm ...
[..]
select(4, [3], [], NULL, {0, 0}) = 0 (Timeout)
select(4, [3], [], NULL, {0, 0}) = 0 (Timeout)
select(4, [3], [], NULL, {0, 0}) = 0 (Timeout)
[..]
akamaras.com: stress test your web server with httperf
SO: Changing the file descriptor size in httperf
easyengine.io: Increase "Open Files Limit"
brendangregg.com: The USE Method
This is an intro to load testing with Apache JMeter, an Open Source load testing tool.
As a developer, QA or Operations engineer, it’s important to be familiar with what load testing tools can do, and to know how to configure a few actual tools.
I usually reach for a load testing tool in the following scenarios, which appear similar, but really are very different. You can divide stress and QA testing into 4 categories:
Note that I don’t consider a simulated load to be meaningful for predicting human loads.
For example, on one intranet project, 70,000 users were happy with a phone book web app that only load tested to 20 simultaneous users. The test was useful in indicating that nothing was misconfigured, but not useful in predicting how many people could actually use it.
Load tests just tell me:
Why JMeter?
JMeter is:
The disadvantages of JMeter are that:
JMeter Installation
java -version
Automating
Make 4 copies of your jmx file with:
cp -p Mysite.com.jmx Mysite1.jmx cp -p Mysite.com.jmx Mysite2.jmx cp -p Mysite.com.jmx Mysite2.jmx cp -p Mysite.com.jmx Mysite3.jmx
Edit each jmx file to customize the properties according to the 4 strategies I listed above:
LoopController.loops ThreadGroup.num_threads
(If you want to invest some time, you can parameterize those as documented in the JMeter FAQ.)
Create the following bash script make_mysite1.sh so that you can run your test from the command line:
#!/bin/bash # Program: test_mysite1.sh rm -f mysite1.log ./jmeter -n -t Mysite1.jmx -l mysite1.samples.log -j mysite1.log grep "Thread Group" mysite1.samples.log | grep -v [O]K
Running Tests
bash test_mysite1.sh
When to Run
Every time a change is made to your environment, you should re-run the load tests. So include it as part of your release process checklist.
Distributed Testing
After you’re familiar with load testing using a single client with JMeter, you can learn about using multiple load test clients.
Bonus – “Soak Testing”
In the telco industry, historically new systems have undergone “soak testing.” This is operating test systems under a realistic load for one month or more to “provide a measure of a system’s stability over an extended period of time.”
JMeter Too Difficult?
There’s a couple options for people who want results without fussing with JMeter:
JMeter: FAQ, Best Practices
SO: Load test with varying number of threads in JMeter
Installing the rkt container manager on CentOS 6.x with yum will give you this error:
# yum -y install go rkt # rkt run rkt: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by rkt)
glibc is not something you can easily upgrade yourself, but you can build rkt from source. On CentOS 6.6 and 6.7, this works:
#!/bin/bash sudo yum install -y go squashfs-tools libacl-devel glibc-static trousers-devel wget https://github.com/coreos/rkt/archive/v1.15.0.tar.gz && tar zxvf - < v1.15.0.tar.gz && cd rkt-1.15.0 && ./autogen.sh echo "insert 'echo' at line 5667 to workaround old autogen bug" vi configure ./configure --disable-sdjournal --with-stage1-flavors=fly --disable-tpm && echo 'readlink -f "$@"' > realpath && chmod +x realpath && export PATH=$PATH:. && make # now test by downloading a Docker Ubuntu image and running it (requires about 256 MB RAM) sudo ./build-rkt-1.15.0/target/bin/rkt run --interactive docker://ubuntu --insecure-options=image
CoreOS Issue #1063: build with old glibc so rkt runs on CentOS 6?
The rkt (pronounced “rocket”) container manager is just too easy to run on CentOS7!
Here’s me running a Docker Ubuntu 16.04.1 LTS image on CentOS7 (Dell 1950 III with 8 GB RAM on 100 Mbps Internet connection) for the first time in under a minute. The Ubuntu Docker image actually starts in 3 seconds once downloaded.
Download the rkt RPM then …
# rpm -Uvh rkt-1.18.0-1.x86_64.rpm # cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) # uptime 06:16:04 up 141 days, 51 min, 1 user,load average: 0.18, 0.26, 0.22 # rkt run --interactive docker://ubuntu --insecure-options=image Downloading sha256:6bbedd9b76a [================] 49.9 MB / 49.9 MB Downloading sha256:fc19d60a83f [================] 824 B / 824 B Downloading sha256:668604fde02 [================] 160 B / 160 B Downloading sha256:de413bb911f [================] 444 B / 444 B Downloading sha256:2879a7ad314 [================] 678 B / 678 B root@rkt-2ee79be0-a70b-44be-90fd-1a1a54c17216:/# cat /etc/os-release NAME="Ubuntu" VERSION="16.04.1 LTS (Xenial Xerus)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 16.04.1 LTS" VERSION_ID="16.04" HOME_URL="http://www.ubuntu.com/" SUPPORT_URL="http://help.ubuntu.com/" BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" UBUNTU_CODENAME=xenial root@rkt-2ee79be0-a70b-44be-90fd-1a1a54c17216:/# uptime 06:16:27 up 141 days, 52 min, 0 users,load average: 0.25, 0.27, 0.22 root@rkt-2ee79be0-a70b-44be-90fd-1a1a54c17216:/# ps -ef UID PID PPID C STIME TTY TIME CMD root 1 0 0 06:16 ? 00:00:00 /usr/lib/systemd/systemd --default-standard-output=tty --log-target=null --show-status=0 root 3 1 0 06:16 ? 00:00:00 /usr/lib/systemd/systemd-journald root 5 1 0 06:16 console 00:00:00 /bin/bash root 13 5 0 06:16 console 00:00:00 ps -ef root@rkt-2ee79be0-a70b-44be-90fd-1a1a54c17216:/# exit # cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) # uptime 06:16:58 up 141 days, 52 min, 1 user,load average: 0.15, 0.24, 0.22
Is 0.0% memory usage light-weight enough? 🙂
# ps aux | egrep -e "[U]SER|[r]kt" USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 6711 0.4 0.0 41772 2244 pts/0 S+ 07:22 0:01 stage1/rootfs/usr/lib/ld-linux-x86-64.so.2 stage1/rootfs/usr/bin/systemd-nspawn --boot --notify-ready=yes --register=true --link-journal=try-guest --quiet --uuid=40660e8b-09c0-46c2-893e-53de6d4068ff --machine=rkt-40660e8b-09c0-46c2-893e-53de6d4068ff --directory=stage1/rootfs --capability=CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FSETID,CAP_FOWNER,CAP_KILL,CAP_MKNOD,CAP_NET_RAW,CAP_NET_BIND_SERVICE,CAP_SETUID,CAP_SETGID,CAP_SETPCAP,CAP_SETFCAP,CAP_SYS_CHROOT -- --default-standard-output=tty --log-target=null --show-status=0
https://coreos.com/blog/getting-started-with-rkt-1.0.html
rkt prepare
Get Started with rkt Containers in Three Minutes
build with old glibc so rkt runs on CentOS 6?
https://github.com/JCMais/node-libcurl/issues/45: ./build-rkt-1.15.0/target/bin/rkt run –interactive docker://ubuntu –insecure-options=image
When doing server upgrades with multiple servers, the ideal way is to:
1. take one instance out of the pool 2. drain connections on it 3. upgrade it 4. put it back into the pool 5. back to #1.
The various techniques can be categorized as:
1. application-level 2. load-balancer-level 3. OS-level
The most graceful method is to use an application-level feature, since the application knows what its worker status is.
For example, with httpd on CentOS or Redhat, either use the apachectl command, or add the graceful-stop option to /etc/init.d/httpd:
#!/bin/bash set -e echo "info: Draining connections ..." apachectl graceful-stop echo "info: You have 5 minutes to start and finish your upgrade." sleep 300 apachectl start echo "info: httpd restarted!" exit 0
If we didn’t have an application-specific way to do that, we could use iptables:
#!/bin/bash set -e iptables -I INPUT -j DROP -p tcp --syn --destination-port 80 echo "info: Draining connections ..." sleep 60 echo "info: You have 5 minutes to start and finish your upgrade" sleep 300 iptables -D INPUT -j DROP -p tcp --syn --destination-port 80 echo "info: iptables allowing new incoming connections!" exit 0
With HAProxy we can do this on the HAProxy host (do yum -y install socat first):
#!/bin/bash set -e echo "set server application-backend/www0 state drain" | socat unix-connect:/var/run/haproxy.sock stdio echo "info: Draining connections ..." sleep 60 echo "set server application-backend/www0 state maint" | socat unix-connect:/var/run/haproxy.sock stdio echo "info: You have 5 minutes to start and finish your upgrade" sleep 300 echo "set server application-backend/www0 state ready" | socat unix-connect:/var/run/haproxy.sock stdio echo "info: haproxy allowing new incoming connections!" exit 0
Sample HAProxy “show stat” output while www0 is draining (notice the “DRAIN” status):
[root@gw ~]# echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio Name: HAProxy Version: 1.5.10 [..] [root@gw ~]# echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio # pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime, http-in,FRONTEND,,,0,1,2000,6,998,19219,0,0,0,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,1,,,,0,5,1,0,0,0,,0,1,6,,,0,0,0,0,,,,,,,, https-in,FRONTEND,,,0,20,2000,12,294,2556,0,0,11,,,,,OPEN,,,,,,,,,1,3,0,,,,0,0,0,2,,,,0,0,1,11,0,0,,0,2,12,,,0,0,0,0,,,,,,,, application-backend,www0,0,0,0,1,5000,3,583,1078,,0,,0,0,0,0,DRAIN,1,1,0,0,0,385,0,,1,4,1,,3,,2,0,,1,L7OK,301,0,0,2,1,0,0,0,0,,,,0,0,,,,,442,Moved Permanently,,0,0,0,1, application-backend,www1,0,0,0,1,5000,4,709,18640,,0,,0,0,0,0,UP,1,1,0,0,0,755,0,,1,4,2,,4,,2,0,,1,L7OK,301,0,0,3,1,0,0,0,0,,,,0,0,,,,,141,Moved Permanently,,0,1,8,8, application-backend,BACKEND,0,0,0,1,400,7,1292,19718,0,0,,0,0,0,0,UP,1,1,0,,0,755,0,,1,4,0,,7,,1,0,,1,,,,0,5,2,0,0,0,,,,,0,0,0,0,0,0,141,,,0,1,8,8,
For nginx:
#!/bin/bash set -e echo "info: Draining connections ..." nginx -s quit echo "info: You have 5 minutes to start and finish your upgrade." sleep 300 nginx -s start echo "info: nginx restarted!" exit 0
If you’re using a configuration management system, like puppet or Chef, you can remove the service from your load balancer pool. This works well in practice with only 2 or 3 servers, though draining is usually not considered.
Note that when using the popular “reverse HAProxy” setup with application servers running HAProxy on localhost, and HAProxy forwarding localhost requests to the real servers (like httpd), then you want to stop or block the httpd services on the real server end. Otherwise you would have to make changes on multiple application servers.
In a future post, I’ll discuss zero-downtime deploys.
Drain connections on restart of NGINX process? (with iptables)
Tomcat’s Graceful Shutdown with Daemons and Shutdown Hooks
Get haproxy stats/informations via socat
Go net/http: add built-in graceful shutdown support to Server #4674
haproxy.tech-notes.net: HAProxy Socket Commands
Just thinking about how to configure HAProxy with two backend Java servers to be HA, despite GC pauses.
Java programs pause periodically to recycle temporary variables, known as garbage collection (GC). This is called a “GC Pause.”
The description “Stop the World” (STW) illustrates their true severity – GC pauses are a slow-motion train wreck for incoming requests. They can last from hundreds of milliseconds to minutes, and require intense CPU activity.
Executive Summary:
option redispatch to load balance across multiple Java servers to maintain availability during GC pauses. You can either use the HAProxy drain feature for rolling deployments, or in more complex setups, iptables.
If you’re new to this topic, please read:
Willy: “I work with people who use a lot of Java applications, and I’ve seen them spend as much time on tuning the JVM as they spend writing the code, and the result is really worth it.” Anybody have some extra time? 😐
My operational requirements for Java in production are:
1. Understand GC pause activity for my application servers
Detailed GC logging and heap dump on OOM can be enabled with:
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError
and you can specify a separate GC log with:
-verbose:gc -Xloggc:/tmp/gc.log
See “Understanding Garbage Collection Logs.”
2. Control GC pause activity to a reasonable and known extent
One of the biggest challenges is to control the frequency, duration and intensity of GC pauses …
Some Java configuration approaches:
An example of some of the tuning options:
java -Xms512m -Xmx1152m -XX:MaxPermSize=256m -XX:MaxNewSize=256m MyClass.java
JRockit JVM: Tuning For a Small Memory Footprint
Tuning Java Virtual Machines (JVMs)
Weblogic Tuning JVM Garbage Collection for Production Deployments
Programming best practices to reduce GC pauses:
3. Configure HAProxy load balancer requests to not be sent to servers undergoing GC pause events
The first thing to do is to read up on HAProxy’s option redispatch feature. Continue reading for more in-depth considerations.
This is tricky for several reasons:
a) Some things to think about:
This method would be risky with two Java servers, since during maintenance on one server, the other could GC pause. (facepalm)
b) Another possible approach would be to handle MemoryPoolMXBean MEMORY_THRESHOLD_EXCEEDED events. Maybe that can be used to update the health check on the server side and send a drain socket request to HAProxy if you reliably had advance notice and could force a GC now, trying the Java Tool Interface ForceGarbageCollection()?
c) And another idea is to write a sentinel file every 250 ms, and if it reaches 750 ms, assume a GC is happening and drain HAProxy. Unfortunately the TI events GarbageCollectionStart() and GarbageCollectionEnd() are sent after the VM is stopped, so you’re limited in what you can do when you need the most flexibility.
Some Java 8 Classes related to GC notifications:
Also, investigate mod_jk and AJP. tomcat uses the same heap as your application, so tuning is very important here too.
4. Use an affordable amount of RAM to accomplish the above, preferably 8 or 16 GB in a shared VM environment
If you work in a VM consolidation environment, it’s important to minimize the footprint of your applications. See above for rewriting applications to minimize heap and GCs.
Garbage Collection JMX Notifications Example Code
Blade: A Data Center Garbage Collector
How to Tame Java GC Pauses? Surviving 16GiB Heap and Greater
SO: Garbage Collection Notifications
Letting the Garbage Collector Do Callbacks
HAProxyController.java
How to force garbage collection in Java?
SSL Termination, Load Balancers & Java
Github: Measuring Java Memory Consumption – sample code
Java is not “angry” with you.
Set State to DRAIN vs set weight 0
Scalable web applications [with Java]
Examples of forcing freeing of native memory direct ByteBuffer has allocated, using sun.misc.Unsafe?
Lucene ByteBuffer sample code
Improve availability in Java enterprise applications
The Four Month Bug: JVM statistics cause garbage collection pauses
Memory management when failure is not an option
Making Garbage Collection faster
The Complete Guide to Instrumentation: How to Measure Everything You Need Inside Your Application
Java heap terminology: young, old and permanent generations?
5 Coding Hacks to Reduce GC Overhead
Java Debugger Changes Lifetime of Variables
Objects Should Be Immutable
Thread Safety and Immutability
Azul Blog: So, G1 may become the default collector for Java 9?
Java and Scala Type Systems are Unsound
Go-related
Golang: sub-millisecond GC pause on production 18gb heap HN
Getting Past C
Go GC: Prioritizing low latency and simplicity
Sub-millisecond GC pauses in Go 1.8 Graphs
Cassandra-related
CASSANDRA-5345: Potential problem with GarbageCollectorMXBean
Java GC pauses, reality check
So, let me get this straight …
Doesn’t leave much left! 🙂
Thanks to Greg Lindahl, founder of Blekko, for making my day. You’re The Man when it comes to performance!
Another good one that made *me* pause:
Links
Apache jMeter
Distributed Testing with JMeter on EC2
I would have gladly just linked to somebody else’s sample clients, but I couldn’t find any remotely complete or professional-grade code (complete working program with error-handling, Basic auth and timeout.)
The closest to useful REST clients that I saw were the Java tutorials by mkyong.com, RESTful Java client with Apache HttpClient
Here’s my notes:
Java
javac -Xlint:allGo
Python
pindent.py -r -s 4 -eRuby
httparty HTTP frameworkPHP
curl
wget
Perl
RESTful Java client with Apache HttpClient
Why Pylint is both useful and unusable, and how you can actually use it
Notes on Managing Java in the Cloud
Static typing will not save us from broken software
OpenFeign Java HTTP Client Library
Java: How To Read a Complete Text File into a String
Golang’s Real-time GC in Theory and Practice
I went to the complimentary PagerDuty Summit Sept. 13 on Market Street in SF.
The well-organized conference format was 2 tracks downstairs, with breaks and a small expo area upstairs.
Andrew Fong of Dropbox had a very good talk on their struggle to go from four 9’s (“can use tactics”) to five 9’s (“has to be strategic”.) Their solution was to have a working group composed of anybody who wanted to contribute, across departments. (Not dedicated HA staff.)
Andre Kelly of Google talked about having well-defined post-mortem processes in place now to capture outages in an organized manner and data mine the results over time later.
Apparently there’s some popular Open Source post-mortem systems for that. Please leave a comment if you have any experience with those.
Sean Reilley of IBM discussed people issues in communicating agile across a large company with pockets of staff who were used to waiting for permission (ie. not inherently agile.)
Upstairs, the mini-expo seemed to have a couple booths for security-related start-up Cloud products, Datadog, plus a booth for PagerDuty itself to do customer demos and get beta feedback.
The money shot was seeing their new beta graphical incident timeline, to be released in November, which made the trip worthwhile. Until then, you can enable HTML emails for a slightly richer experience.
The “Village” historic venue, [pic], was not my favorite: climbing up and down steep stairs with a backpack got old fast.
CNN linked to a video of John Collins, “The Paper Airplane Guy.”
John holds the world-record for paper airplane distance throwing.
I had a chance to see John live recently when he gave a lecture and demo at my office in Silicon Valley.
It was a unique experience:
I hold a commercial airplane licence and can say that he really knows his stuff. Highly recommended.
Back in the day, Joel wrote an infamous post asserting that “software doesn’t rot” over time.
I believe Joel was addressing the tendency of new programmers on a project to avoid learning the old codebase and write a new one instead, at great cost in terms of time and money.
But let’s discuss the more interesting topic of whether software can actually rot.
I would say that he was correct that it doesn’t rot in a very narrow sense, namely a program written for a single version of Windows.
But in the big picture, he was completely wrong. Even Windows software requires re-writes for “Certified for Windows” assurance for new shrink-wrapped versions to be shelved in US chain stores. (Stores were trying to reduce the rate of returns and customer support.)
And how’s Silverlight, discontinued in 2012, working out for developers? 🙂
When it comes to web software, total re-writes have been required for:
Additionally, Y2K often required software changes, as will Y2038.
In Joel’s article Fire And Motion he makes an interesting observation that changes can also be used to keep developers off balance. It’s a very effective technique in the software world where buyers chase versio numbers.
Apple could kill almost 200,000 apps with iOS 11
joelonsoftware.com: Fire And Motion
I’ve been doing a lot of work with REST APIs and microservices, so I decided to write a complete REST API framework in Perl based on the Mojolicious and Swagger2 Petstore sample.
You can git clone the repo and add a new API endpoint in about 5 minutes with automatic parameter validation and documentation:
git clone git@github.com:jamesbriggs/perl-petstore-enhanced.git
cd perl-petstore-enhanced/pets
less ../README.md
vi api.spec set.sh cgi-bin/pets.cgi ./lib/Pets/Controller/Pet.pm
# add an Alias for cgi-bin/pets.cgi to httpd or nginx
# point your browser at http://www.example.com/api/v1.0/pets/1
# Good job. Have a Modelo! :)
or you can spend an hour to rename the files for your project and tweak it to requirements.
This project serves as a convenient bridge for those who:
Also of note is the samples/ folder, which has non-trivial client programs in several languages (bash, Java, Perl, PHP and Ruby.)
I learned the importance of Swagger2 and auto-generated API documentation and validation when I was programming with the old Rackspace Cloud v1 and v2 APIs.
People asked me, “How did you get anything to work? You must have really wanted it!” since the Rackspace sample code, docs and live API didn’t match each other. My secret: I actually guessed URLs in the browser to find the endpoints I needed. Swagger prevents that headache.
http://editor.swagger.io/#/
Swagger UI
idratherbewriting.com: 10 realizations as I was creating my Swagger spec and Swagger UI
Are microservices for you? You might be asking the wrong question.
developer.mozilla.org: List of default Accept values
Spent Labor Day weekend in Waikiki.
I enjoy going there every few years and seeing what’s new.
However, it’s been completely built out as a mall, so looks kind of corporate now. To combat that, plan to climb Diamondhead and go to the zoo.
Also, who would fly a quadcopter drone at one of the most crowded beaches in the world? Not surprised, just saying.
So what’s new in Waikiki?
Waikiki photo vantage points:
If you’re from the mainland, remember that Hawaii is hot and humid. Stay hydrated, wear a hat, and don’t over-exert yourself – especially around noon-time.
Mahalo!
As a commercially-rated airline pilot who reads accident reports, I always tingle when I fly on anything but a USA majors flight in less than perfect weather.
The recent Emirates 777 crash in Dubai is a case in point.
The airliner, with 300 people aboard, crashed into the runway with a sink rate of 900’/minute, and later the center-tank exploded, killing one firefighter. 22 pax and FAs were injured descending the slides (typically, several people are injured during a slide evacuation.)
It’s important for pilots to always be mindful that a landing approach can end in two ways:
Though it would take a lot of painstaking research to say where this particular flight started going wrong, we do know some of the links in the “accident chain”:
Taken together, obviously nobody with a clue was in the cockpit that day. I would rank this accident as bad as the TransAsia GE235 “Oops, I shutdown the good engine” accident in Taipei.
Disbelief at the death of Prince at the relatively young age of 57.
Prince was a musical genius, certainly one of the giants of this century – he wrote, sang, was a virtuoso of 2 dozen instruments, and played guitar at the level of Jimi Hendrix.
He could perform with everybody, or nobody, yet chose to mentor female musicians, introducing them by name in his shows.
I saw one of his shows, but wish I had gone to more.
For business reasons, he never allowed his catalog on YouTube, but there’s a few links from TV performances that indicate his brilliance and show him “bringing the funk”:
Prince & 3RDEYEGIRL Perform ‘She’s Always In My Hair’
Prince Saturday Night Live Full Performance (2014)
Prince playing piano over ‘Summertime’ at Soundcheck, Koshien, Hyogo Prefecture (1990)
PRINCE BET Interview with Tavis Smiley(1998)
“Stand Back” – Stevie Nicks (writer/vocals) with Prince (synths/drum machine), inspired by Little Red Corvette
cnn.com: Prince’s vision for lifting up black youths: Get them to code, Prince’s Death: Latest News
W: Prince
Nicole Scherzinger sings Purple Rain Tribute
Mayte Garcia on the Prince you didn’t know
PRINCE THE MAN BEHIND THE MUSIC ~ FULL DOCUMENTARY
There were a few major earthquakes this weekend:
Hundreds of aftershocks have occurred in Japan.
You would think that Californians, of all people, would be concerned with earthquake safety, but the LA Times has reported on a building safety cover-up involving thousands of older schools and office buildings which will pancake in a major quake.
How Risky Are Older Concrete Buildings?
LA Times FAQ: Concrete buildings, earthquake safety and you
Non-ductile Concrete Buildings
I used to write telemetry collection software for the Space Shuttle, rockets and balloons, but even I watched the SpaceX barge landing with disbelief as the rocket smoothly rotated in all 11 or so degrees of freedom at the same time – no hesitation or staging before the touchdown.
It was like watching a really big lawn dart plant itself. 🙂
twitter: SpaceX
HN: SpaceX Launch Livestream: CRS-8 Dragon Hosted Webcast
theRegister: SpaceX finally lands Falcon rocket on robo-barge in one piece, SpaceX’s Musk: We’ll reuse today’s Falcon 9 rocket within 2 months
A few pieces of MH370, a Boeing 777-200ER, have recently been found on a Mozambique beach, and confirmed as authentic parts.
Their excellent condition and relatively large sizes indicate that the accident wasn’t a high-speed impact with an obstacle or water.
As a commercially-rated airplane pilot, my opinion is that leaves:
An interesting question would be, “In modern airliners, especially Airbus, anti-phugoid software is deployed. How would that affect an uncontrolled airliner?” Sully said that anti-phugoid software in his Airbus prevented him from slowing descent before impacting the Hudson River.
MH370 Debris Storm
Tourist who found debris was searching for MH370
Turbulence V-Speeds
Australia Confirms Mozambique Debris Came from MH370
MH370: Debris found in March ‘almost certainly’ from missing plane
Investigators Report On MH370 Debris Analysis (2016)
ATSB MH370 Report [pdf]
Congrats to the Laser Interferometer Gravitational-wave Observatory (LIGO) team for directly detecting gravitational waves for the first time.
LIGO was the NSF’s most expensive project, and took scientists basically from the 1960’s to 2015 to fully realize – initially nobody believed it was possible to actually build this instrument.
Two detector locations with perpendicular 4 km 4-mirror laser interferometers were able to detect gravitation waves from a billion year-old blackhole collision:
The graph is very interesting and raises the questions:
(The local speed of light in a medium is a constant, while gravitation waves distort space, thus changing an interference pattern.)
Basic science is always valuable, but just a few of the reasons why this experiment is important:
More generally, measurement tools are the highest form of technology, whether for time, space, EM, or gravity. Any investment of time or money in measurement tools is easily repaid 1000x. For example, the GPS system is the result of accurate time measurement using “atomic clocks.”
This decade is an exciting time for science, as several major terrestrial and space instruments come online or are upgraded.
It will be interesting to see if anybody develops a table-top model of LIGO. Experiments in the 60’s with non-laser methods were susceptible to ambient vibrations, but we’ll see. Cryogenics would likely have to be involved since random atomic motions are larger than the signal being acquired.
LIGO Detects Gravitational Waves for Third Time
Gravitational Waves Detected 100 Years After Einstein’s Prediction
SIGNAL PROCESSING WITH GW150914 OPEN DATA
W: LIGO
Reddit AMA
LIGO black hole echoes hint at general-relativity breakdown
On the time lags of the LIGO signals HN
Gravitational waves from a binary black hole merger observed by LIGO and Virgo
LIGO Architects Win Nobel Prize in Physics
LIGO and Virgo announce the detection of a black hole binary merger from June 8, 2017
I watched Superbowl 50 in Sunnyvale – a nice spring-like day with blue skies.
Got a bonus show: I was just going inside as the Blue Angels did a low-altitude formation flyover, followed by a couple solo approaches, toward Levi’s Stadium.
Denver Broncos over Carolina Panthers 24 – 10, with Denver leading the entire game.
Cam Newton, QB for Carolina, got sacked, to varying degrees, 6 times. He sore-loser sulked during the post-game interviews, which generated a lot of controversy.
Peyton Manning, Bronco’s QB, won MVP, amidst the usual narcissistic drama of whether he’d retire on top, or not.
The turf came under scrutiny, as some linebackers were literally sliding across it.
Halftime Show
Beyonce, looking thick, Bruno Mars, nice moves in a rubber suit, and Coldplay (woefully) performed. Must have been a nostalgic Brit on the halftime committee I guess.
According to the media, Beyonce was doing a Black Power protest, but the show wasn’t particularly different than anything MJ or Janet did. And frankly, I wouldn’t blame her if she did.
Ads
Most of the ads were forgettable.
The Amazon ad with Baldwin and Marino was ok.
There were a few annoying prescription ads, though the cartoon intestines with feet one was more than weird.
Municipal Sports Stadium Corruption
I’m local to the Levi’s Stadium, so am aware of the endless tales of corruption (lack of accounting to City Council, failure to make public service reimbursements, destruction of meeting notes and emails, mis-appropriation of a kids soccer park, etc.)
But even I was surprised that the local transit authorities “privatized” the Caltrain and VTA Light Rail for the day, requiring a a SuperBowl 50 ticket and special $40 ticket per passenger to use a taxpayer-funded system. Hmm.
“Event Passengers Must Pre-Purchase VTA Fare Prior to Boarding
All passengers traveling to the Super Bowl must use VTA’s mobile app, EventTIK to purchase a special VTA Super Bowl 50 Day Pass fare AND possess proof of a valid Super Bowl ticket in order to board the special Super Bowl trains.”
Mr. York: next time, pay for your own damn stadium. You can afford it.
http://www.vta.org/superbowlsun
Formation Flyover Photo
Today was the last chance to see Babbage’s Difference Engine at the CHM in Mountain View before the owner makes it private again.
The Computer History Museum has certainly matured into a world-class museum over the years.
The docent talked for about 45 minutes. Unfortunately, it was displayed at the end of a hallway. So 100+ people with kids and strollers jostled to get a view.
It’s very impressive in person – consists of 8,000 parts, weighs five tons, and measures 11 feet long, moderately noisy and mesmerizing to watch. The cranker used a moderately strong rowing motion.
Babbage, in building the first computer, did not have the hindsight to start with a smaller version first. Thus he never finished building a working model despite a decade of funding from the British government and the remaining days of his life working on it.
CHM did a fantastic job on the DEC PDP-1 and IBM 1401 display rooms. Only about 50 PDP-1’s were made, so to have a working model is amazing.
Congrats to Honda for earning FAA Production Certification for their first aircraft, the HondaJet HA-420 light business jet.
I’ve been following the news of the HondaJet for over a decade as they progressed step-by-step towards certification.
The HA-420 is the most technologically advanced, fastest (420 knots) and efficient (by up to 20%) small business jet currently certified. Of interest to owner/operators, it may be flown single-pilot.
The price is $4.5 million, which Honda can finance.
The creation of the HondaJet is an epic story, starting with Honda’s founder dreaming of building an airplane several decades ago, and establishing design facilities 2 decades ago in the USA.
A jet engine, the GE Honda HF120, was also certified for this plane.
The total investment to certify both an airframe and an engine must have been staggering to get to this point. Only a multinational mfg. company with support from top executives like Honda can pull that off in peacetime.
Even so, aviation is a tough business to make money in, especially as a new entrant.
Japanese companies have a long history of interesting work in aerodynamics. Both the Battleship Yamato and Bullet Train used duck-bill shaped leading airfoils for significant drag reduction. The HondaJet developers likewise use laminar flow nose (see top photo) and wings, and winglets (see second photo.)
According to a review by a friend of Philip Greenspun, the airplane has some issues: interior noise in the passenger compartment is 6 DB too high, only 573 pounds of useful load with full fuel, and a 4000′ runway is needed. Also, a lot of pilot ergonomics that should have made it in, didn’t. Also, the high price is comparable to the the next class up, which are much roomier and have more comfortable useful loads.
yt: Kenny G Live at the HondaJet TC Event with Mr.Fujino,
HondaJet FAA Type Certification Celebration
avweb.com: HondaJet Wins FAA Certification
HondaJet Nominated for 2015 Collier Trophy
W: HondaJet
philip.greenspun.com: HondaJet Pilot Review
HondaJet Makes Chinese Debut at ABACE Show
HondaJet Flight Trial 2017 [Video]
Honda Aircraft Adds Canada to HondaJet Approvals
HondaJet Gets Brazilian Nod
HondaJet, Phenom 300 Spar for Top Light Jet Sales Title
Although I’ve walked by TAP Plastics on Castro St. in Mountain View a hundred times, today was the first time I went inside.
Their motto “the fantastic plastic place” is accurate.
They have specialized in plastics sales since 1952 and have 21 stores.
Their web site is a gem, supporting 9 languages using Google Translate.
TAP Plastics Inc.
312 Castro Street
Mountain View, CA 94041
How to do initial firewalld configuration for Cassandra Server and Opscenter on CentOS/Redhat 7 with 2 network interfaces, in my case Dell 1950/2950.
First: verify that your network interfaces are associated with a NetworkManager zone:
# grep -i zone /etc/sysconfig/network-scripts/ifcfg-*
/etc/sysconfig/network-scripts/ifcfg-enp4s0:ZONE=internal
/etc/sysconfig/network-scripts/ifcfg-enp8s0:ZONE=public
# service network restart
Second: add the Cassandra ports to the internal zone (private interface) and public zone (public interface):
#!/bin/bash
# add ports on internal interface for Cassandra server
firewall-cmd --zone=internal --add-port=7000/tcp --add-port=7199/tcp --add-port=9042/tcp --add-port=9160/tcp --add-port=61619-61621/tcp --permanent
# add ports on public interface for Cassandra server
firewall-cmd --zone=public --add-port=80/tcp --add-port=8888/tcp --permanent
firewall-cmd --reload
Edit the files in /etc/firewalld/zones to remove the desktop helper services, then do
service firewalld restart
3. Verify configuration:
firewall-cmd --get-active-zones
firewall-cmd --zone=public --list-ports
firewall-cmd --zone=public --list-services
firewall-cmd --zone=internal --list-ports
firewall-cmd --zone=internal --list-services
Output is:
# firewall-cmd --get-active-zones
internal
interfaces: enp4s0
public
interfaces: enp8s0
# firewall-cmd --zone=internal --list-ports
7000/tcp 7199/tcp 9042/tcp 9160/tcp 61619-61621/tcp
# firewall-cmd --zone=internal --list-services
ssh
# firewall-cmd --zone=public --list-ports
80/tcp 8888/tcp
# firewall-cmd --zone=public --list-services
ssh
4. Verify firewall rules with nmap:
# nmap -sS my.external.interface.com
Starting Nmap 5.51 ( http://nmap.org ) at 2015-10-15 22:34 PDT
Nmap scan report for my.external.interface.com
Host is up (0.075s latency).
Not shown: 997 filtered ports
PORT STATE SERVICE
22/tcp open ssh
80/tcp open http
8888/tcp open opscenter
Nice! 🙂
Troubleshooting
As always, if you experience network issues on linux, disable selinux, firewalld and TCP wrappers first and verify if those are the source of the problem:
setenforce 0
service firewalld stop
cat /etc/hosts.*
To boot into singleuser mode, replace the linux grub line “ro” item with “rw init=/sysroot/bin/sh”.
Fedora introduces Network Zones
fedoraproject.org: Network Zones
Virtualbox 4.3.30 on OS X 10.8.5 with CentOS 7 guest VMs work ok on my notebook for web development, but setup was a little fussy.
I use VMs for:
Terminology
Getting Started
Choose Network Topology
I wanted to run my web site in a VM, viewable from the Mac browser and have the VM be able to run ‘yum update’, so needed host => guest and guest => Internet routing. There’s 2 networking choices that match those requirements:
sudo ipfw add 100 fwd 127.0.0.1,8080 tcp from any to any 80 in
Bridged
NAT
Troubleshooting
Exercises
Network Security
sshd: 10.0.0.0/255.0.0.0 192.168.0.0/255.255.0.0
http: 10.0.0.0/255.0.0.0 192.168.0.0/255.255.0.0
ALL: ALL
Simulating Production
You can update /etc/hosts to have your browser access your web site in a VM:
/etc/hosts:
# NAT
127.0.0.1 www.mysite.com
or
# Bridged
10.0.0.5 www.mysite.com
But I find that Firefox gets less confused with permanent redirects, etc. by prefixing the hostname:
/etc/hosts:
# Virtualbox NAT Topology (don't forget to use ports 2222 and 8000 from host => guest!)
#127.0.0.1 www.test-mysite.com
or
# Virtualbox Bridged Topology
#10.0.0.5 www.test-mysite.com
#10.0.2.15 www.test-mysite.com
Backups
Take advantage of Virtualbox’s clone and snapshot features.
forums.virtualbox.org: What does “Cable connected” checkbox change?
Port Forwarding in Mac OSX Mavericks
Port Forwarding in Mac OS Yosemite
Here’s my Github pull request for improved error handling in Percona’s clustercheck utility, used by haproxy for health-checking a Percona XtraDB Cluster.
It adds two features:
The error detection is done in a low-latency manner using PIPESTATUS, without an additional database connection. Here is colored diff output.
Wed. Keynotes
WebscaleSQL
5 companies
Facebook, Google, Alibaba, Twitter
Percona and MariaDB
Please use apache CLA
MariaDB.com CEO
http://451research.com/dashboard/dpa
Multisource replication from Taobao
Spider Sharding
Atomic writes with Fusion IO/Sandisk
18% faster with 1/4 writes
Coneect Storage Engine for Federation
Galera integrated
Encryption by Google – tablespace and table
Amazon Aurora
Maxscale proxy
Tomas Ulin, Oracle
MySQL 5.7
– Optimizer improvements
– JSON support in pipeline
– SYS
– GIS rewrite
– Innodb improvements
– native partitions. Bug fixes and transportable tabelspaces
– dynamic buffer pool size
– group replication
– Fabric 1.5
– Workbench 6.3
– MySQL Cluster 7.4 GA ???
Robert Hodges, Continuent/VMware
– “VMware is creating a new kind of hybrid cloud”
– vSphere 6 FT – cpu/ram mirror over 10 Gb Ethernet up to 4 vcpus
– but maintenance still needs continuent
– information week 2014 db popularity
– tunsten replication to vertica, redshift, oracle, hadoop
Thursday
Lightning Talks
pv-mysql-pman
m.wang@salesforce.com
At Silicon Valley Users Group (SVLUG) tonite, Daniel Klopp, Senior Technical Consultant, Taos Consulting, gave an intermediate talk on “Docker.”
He had some really informative and detailed slides on using Docker, especially his cgroup commands samples.
Some of the interesting things he mentioned were:
Daniel Klopp
One attendee mentioned that a work-around for the insecure nature of Docker is to combine it with SELinux, though that will involve a fair amount of work.
Over 400 people RSVPed on a related Meetup, and over 150 people attended, a record for this decade.
Great turnout!
Salad, meat lasagna, pasta alfredo, veggie lasagna from Taos!
Thanks to Symantec once again for hosting the event.
DataStax’s OpsCenter is pretty, but sometimes you don’t want to chop holes in your firewall for the server and agents.
So I wrote cass_top. It works like top, but colorizes the output of nodetool status. It also lets you build nodetool commands using menus, run and log the output.
What’s especially nice is that it uses bash (no python required), and uses minimal screen real estate, so you can view all your clusters on one monitor using eterms.
$ cass_top 10.0.1.140
Please leave a comment with your suggestions.
Below is my patch that implements the CREATE [[NO] FORCE] VIEW options against MySQL/MariaDB 10.1.0.
It adds two new options that look like this:
Notes:
# ./mysql-test-run.pl view Logging: ./mysql-test-run.pl view vardir: /usr/local/mariadb-10.1.0/mysql-test/var MariaDB Version 10.1.0-MariaDB-debug TEST RESULT TIME (ms) or COMMENT ------------------------------------------------------------------- main.view [ pass ] 1896 ------------------------------------------------------------------- The servers were restarted 0 times Spent 1.896 of 7 seconds executing testcases Completed: All 1 tests were successful.
$ cat create_force_view.patch
--- ../mariadb-10.1.0/sql/sql_view.h 2014-06-27 04:50:36.000000000 -0700
+++ sql/sql_view.h 2014-09-02 02:35:42.000000000 -0700
@@ -29,10 +29,10 @@
/* Function declarations */
bool create_view_precheck(THD *thd, TABLE_LIST *tables, TABLE_LIST *view,
- enum_view_create_mode mode);
+ enum_view_create_mode mode, enum_view_create_force force);
bool mysql_create_view(THD *thd, TABLE_LIST *view,
- enum_view_create_mode mode);
+ enum_view_create_mode mode, enum_view_create_force force);
bool mysql_make_view(THD *thd, File_parser *parser, TABLE_LIST *table,
uint flags);
--- ../mariadb-10.1.0/sql/sql_lex.h 2014-06-27 04:50:33.000000000 -0700
+++ sql/sql_lex.h 2014-09-02 01:21:10.000000000 -0700
@@ -170,6 +170,12 @@
VIEW_CREATE_OR_REPLACE // check only that there are not such table
};
+enum enum_view_create_force
+{
+ VIEW_CREATE_NO_FORCE, // default - check that there are not such VIEW/table
+ VIEW_CREATE_FORCE, // check that there are not such VIEW/table, then ignore table object dependencies
+};
+
enum enum_drop_mode
{
DROP_DEFAULT, // mode is not specified
@@ -2442,6 +2448,7 @@
};
enum enum_var_type option_type;
enum enum_view_create_mode create_view_mode;
+ enum enum_view_create_force create_view_force;
enum enum_drop_mode drop_mode;
uint profile_query_id;
--- ../mariadb-10.1.0/sql/sql_parse.cc 2014-06-27 04:50:34.000000000 -0700
+++ sql/sql_parse.cc 2014-09-02 02:34:31.000000000 -0700
@@ -4943,7 +4943,7 @@
Note: SQLCOM_CREATE_VIEW also handles 'ALTER VIEW' commands
as specified through the thd->lex->create_view_mode flag.
*/
- res= mysql_create_view(thd, first_table, thd->lex->create_view_mode);
+ res= mysql_create_view(thd, first_table, thd->lex->create_view_mode, thd->lex->create_view_force);
break;
}
case SQLCOM_DROP_VIEW:
--- ../mariadb-10.1.0/sql/sql_yacc.yy 2014-06-27 04:50:37.000000000 -0700
+++ sql/sql_yacc.yy 2014-09-05 17:19:29.000000000 -0700
@@ -1851,7 +1851,7 @@
statement sp_suid
sp_c_chistics sp_a_chistics sp_chistic sp_c_chistic xa
opt_field_or_var_spec fields_or_vars opt_load_data_set_spec
- view_algorithm view_or_trigger_or_sp_or_event
+ view_algorithm view_or_trigger_or_sp_or_event view_force_option
definer_tail no_definer_tail
view_suid view_tail view_list_opt view_list view_select
view_check_option trigger_tail sp_tail sf_tail udf_tail event_tail
@@ -2446,6 +2446,7 @@
VIEW_CREATE_OR_REPLACE);
Lex->create_view_algorithm= DTYPE_ALGORITHM_UNDEFINED;
Lex->create_view_suid= TRUE;
+ Lex->create_view_force= VIEW_CREATE_NO_FORCE; /* initialize just in case */
}
view_or_trigger_or_sp_or_event
{
@@ -15887,6 +15888,15 @@
| event_tail
;
+view_force_option:
+ /* empty */ /* 411 - is there a cleaner way of initializing here? */
+ { Lex->create_view_force = VIEW_CREATE_NO_FORCE; }
+ | NO_SYM FORCE_SYM
+ { Lex->create_view_force = VIEW_CREATE_NO_FORCE; }
+ | FORCE_SYM
+ { Lex->create_view_force = VIEW_CREATE_FORCE; }
+ ;
+
/**************************************************************************
DEFINER clause support.
@@ -15944,7 +15954,7 @@
;
view_tail:
- view_suid VIEW_SYM table_ident
+ view_suid view_force_option VIEW_SYM table_ident
{
LEX *lex= thd->lex;
lex->sql_command= SQLCOM_CREATE_VIEW;
--- ../mariadb-10.1.0/sql/sql_view.cc 2014-06-27 04:50:36.000000000 -0700
+++ sql/sql_view.cc 2014-09-05 19:33:58.000000000 -0700
@@ -248,7 +248,7 @@
*/
bool create_view_precheck(THD *thd, TABLE_LIST *tables, TABLE_LIST *view,
- enum_view_create_mode mode)
+ enum_view_create_mode mode, enum_view_create_force force)
{
LEX *lex= thd->lex;
/* first table in list is target VIEW name => cut off it */
@@ -259,7 +259,7 @@
DBUG_ENTER("create_view_precheck");
/*
- Privilege check for view creation:
+ Privilege check for view creation with default (NO FORCE):
- user has CREATE VIEW privilege on view table
- user has DROP privilege in case of ALTER VIEW or CREATE OR REPLACE
VIEW
@@ -272,6 +272,7 @@
checked that we have not more privileges on correspondent column of view
table (i.e. user will not get some privileges by view creation)
*/
+
if ((check_access(thd, CREATE_VIEW_ACL, view->db,
&view->grant.privilege,
&view->grant.m_internal,
@@ -285,6 +286,11 @@
check_grant(thd, DROP_ACL, view, FALSE, 1, FALSE))))
goto err;
+ if (force) {
+ res = false;
+ DBUG_RETURN(res || thd->is_error());
+ }
+
for (sl= select_lex; sl; sl= sl->next_select())
{
for (tbl= sl->get_table_list(); tbl; tbl= tbl->next_local)
@@ -369,7 +375,7 @@
#else
bool create_view_precheck(THD *thd, TABLE_LIST *tables, TABLE_LIST *view,
- enum_view_create_mode mode)
+ enum_view_create_mode mode, enum_view_create_force force)
{
return FALSE;
}
@@ -391,7 +397,7 @@
*/
bool mysql_create_view(THD *thd, TABLE_LIST *views,
- enum_view_create_mode mode)
+ enum_view_create_mode mode, enum_view_create_force force)
{
LEX *lex= thd->lex;
bool link_to_local;
@@ -425,14 +431,13 @@
goto err;
}
- if ((res= create_view_precheck(thd, tables, view, mode)))
+ if (res= create_view_precheck(thd, tables, view, mode, force))
goto err;
lex->link_first_table_back(view, link_to_local);
view->open_type= OT_BASE_ONLY;
- if (open_temporary_tables(thd, lex->query_tables) ||
- open_and_lock_tables(thd, lex->query_tables, TRUE, 0))
+ if (open_temporary_tables(thd, lex->query_tables) || (!force && open_and_lock_tables(thd, lex->query_tables, TRUE, 0)))
{
view= lex->unlink_first_table(&link_to_local);
res= TRUE;
@@ -513,6 +518,7 @@
}
}
+if (!force) {
/* prepare select to resolve all fields */
lex->context_analysis_only|= CONTEXT_ANALYSIS_ONLY_VIEW;
if (unit->prepare(thd, 0, 0))
@@ -612,6 +618,7 @@
}
}
#endif
+}
res= mysql_register_view(thd, view, mode);
@@ -621,7 +628,7 @@
meta-data changes after ALTER VIEW.
*/
- if (!res)
+ // if (!res)
+ if (!res && !force) /* 411 - solves segfault problems with CREATE FORCE VIEW option sometimes */
tdc_remove_table(thd, TDC_RT_REMOVE_ALL, view->db, view->table_name, false);
if (mysql_bin_log.is_open())
@@ -908,6 +915,8 @@
fn_format(path_buff, file.str, dir.str, "", MY_UNPACK_FILENAME);
path.length= strlen(path_buff);
if (ha_table_exists(thd, view->db, view->table_name, NULL))
{
if (mode == VIEW_CREATE_NEW)
--- ../mariadb-10.1.0/mysql-test/t/view.test 2014-06-27 04:50:30.000000000 -0700
+++ mysql-test/t/view.test 2014-09-06 00:23:32.000000000 -0700
@@ -5263,4 +5263,17 @@
--echo # -----------------------------------------------------------------
--echo # -- End of 10.0 tests.
--echo # -----------------------------------------------------------------
+
+create no force view v1 as select 1;
+drop view if exists v1;
+
+create force view v1 as select 1;
+drop view if exists v1;
+
+create force view v1 as select * from missing_base_table;
+drop view if exists v1;
+
+--echo # -----------------------------------------------------------------
+--echo # -- End of 10.1 tests.
+--echo # -----------------------------------------------------------------
SET optimizer_switch=@save_optimizer_switch;
If you can’t upgrade linux distros, here’s how to install Datastax Cassandra Community Edition and the python cassandra driver on CentOS 5.x.
It’s not difficult, but there’s several steps, including updating java.
(The following steps would make a complete chef or puppet recipe for a non-SSL install with vnodes.)
# setup environment
groupadd -g 602 cassandra
useradd -u 602 -g cassandra -m -s /sbin/nologin cassandra
mkdir /var/lib/cassandra /var/log/cassandra /var/run/cassandra
touch /var/log/cassandra/system.log
chown -R cassandra:cassandra /var/lib/cassandra /var/log/cassandra /var/run/cassandra
mkdir -p /opt && cd /opt
cat >> /etc/security/limits.conf <<EOD
cassandra soft memlock unlimited
cassandra hard memlock unlimited
cassandra soft nofile 8192
cassandra hard nofile 10240
EOD
# upgrade java
yum remove java
# download, then install JDK 7.x from oracle.com
rpm -Uvh jdk-7u67-linux-x64.rpm
# download, then install recent jna.jar from https://github.com/twall/jna
mv jna.jar /usr/share/java
ln -s /usr/share/java/jna.jar /opt/cassandra/lib/
# update envariables
cat >> /etc/profile <<"EOD"
export JAVA_HOME=/usr/java/default
export JRE_HOME=/usr/java/default/jre
export CASSANDRA_HOME=/opt/cassandra
export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin:$CASSANDRA_HOME/bin
EOD
# get Datastax DCE
curl -L http://downloads.datastax.com/community/dsc.tar.gz >dsc-cassandra-2.0.9.tar.gz
tar zxvf - < dsc-cassandra-2.0.9.tar.gz
ln -s /opt/dsc-cassandra-2.0.9 /opt/cassandra
chown -R root:root /opt/cassandra/
bash cassandra/switch_snappy 1.0.4
# open cassandra firewall ports if necessary (not needed if using internal interface on most servers)
vi /etc/sysconfig/iptables
-A INPUT -i eth0 -m state --state NEW -m multiport -p tcp --dport 7000,7199,9042,9160 -j ACCEPT
service iptables restart
# configure /opt/cassandra/conf/cassandra.yaml (at least listen_address, rpc_address, seeds and tokens before starting server. If you need a do-over, clean the cassandra data with # rm -fr /var/lib/cassandra/*)
# download startup script:
wget http://jebriggs.com/php/start_cassandra.txt -O /etc/init.d/cassandra
chown root:root /etc/init.d/cassandra
chmod 755 /etc/init.d/cassandra
chkconfig --add cassandra
# start cassandra server (if it is standalone, or a seed server. otherwise start after the seed servers):
service cassandra start
# cat /etc/redhat-release CentOS release 5.10 (Final) [root@www1 conf]# nodetool status Datacenter: datacenter1 ======================= Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 10.0.1.2 71.87 KB 256 66.8% 8302c6d5-4c88-4695-bbf4-762bc7f24544 rack1 UN 10.0.1.3 136.63 KB 256 69.9% eddb03b2-98d3-46ff-be63-95435414a883 rack1 UN 10.0.1.4 100.08 KB 256 63.3% 2a8dde5e-29b0-4a67-8204-40769376c44a rack1
If you only see the node on localhost, then you have a problem:
Datastax Opscenter 5.0
You can install the binary from yum or tarball, but the important things to know are:
Python also needs to be upgraded if you want to use cqlsh or the python client cassandra driver.
# install python 2.6 and dependencies
yum install gcc python26 python26-devel libev libev-devel
# install python's pip module
curl --silent --show-error --retry 5 https://bootstrap.pypa.io/get-pip.py | python26
# install cassandra driver for python
pip install cassandra-driver
# install blist.py
tar zxvf - < blist-1.3.6.tar.gz
cd blist-1.3.6
python26 setup.py install
cd ..
# cluster.py - test installation
from cassandra.cluster import Cluster
cluster = Cluster(['127.0.0.1'])
def dump(obj):
for attr in dir(obj):
if hasattr( obj, attr ):
print( "obj.%s = %s" % (attr, getattr(obj, attr)))
dump(cluster);
# python26 cluster.py obj.__class__ = <class 'cassandra.cluster.Cluster'> [...]
Troubleshooting connection problems in JConsole
datastax.com: Storing OpsCenter Data in a Separate Cluster
I recently tuned an existing application that used dozens of views and hundreds of stored procedures using MySQL 5.6.
There seems to be three attitudes towards using views and stored procedures (SPs) with MySQL:
Here are some notes on using views:
And some notes on stored procedures (SPs):
Both views and SPs are relatively new MySQL features, so budget some extra development and testing time when using them, especially with replication.
[MDEV-6365] CREATE VIEW Ignores RESTRICT/CASCADE Options
mysqlperformanceblog.com: Using MySQL triggers and views in Amazon RDS
I did a talk at the Silicon Valley Linux Users Group (SVLUG) tonite on “Devops and Release Canaries with Linux, CloudStack and MySQL Cluster.”
Thanks again to Symantec for hosting.
Ravello Arms Deutsche Telekom with On-Demand Cloud Flexibility
Deutsche Telekom’s Enterprise DevOps Journey with VMware, AWS, Jenkins, Chef & Ravello, Slides
The O’Reilly Velocity Web Operations & Performance Conference is June 24-26 in Santa Clara.
Next to the messages/jobs board was a Web Ops & Performance Tips board:
– use source maps to debug compressed JS and CSS
– use ::before to optimize font rendering
– use local storage to persist markup and templates to reduce requests and payload
– avoid CSS block rendering in chrome by not using screen media type until after. Then put screen back to element
– use gatling stress tool for load generation/perf testing (Apache Licence 2.0)
– learn curl
– learn POSIX before recreating another tool that already exists. Bill Joy (?)
– “if you do it more than twice a week, automate”
– it takes no skills to do NoOps! 🙂
I happened to be in SF today, so I went to the Amazon Web Services pop-up loft on Market St.
Amazon rented an empty storefront for 4 weeks for lecture sessions upstairs, and a computer lab and an ‘Ask an Architect’ bar downstairs.
One of the hosts said the loft was a shell in May, and they had to build out everything: the kitchen area, 2 bathrooms and various partitions.
I asked the experts about new EBS and RDS features, and they had answers as well as a $100 AWS credit.
The weather was sunny and warm in SF.
Lots of street performers and hustlers, including a very smooth male R&B singer. A young rapper named Rap2K15 was selling hand-made CDs.
Update 2014 06 23: Apparently a drawing was held, and I was one of 3 winners of a free general pass to the AWS:Reinvent Conference 🙂
Update 2014 06 24:
AWS Bootcamp
Full-day AWS overview, including EC2, S3, RDS, VPC and IAM, with 2 labs.
“Provisioning and Managing AWS Infrastructure with Chef” with special guest George Miranda, Chef Technical Consultant, Chef
George talked about using Chef tools like chef metal, knife and chef zero and a minimal amount of ruby to make an AMI and provision a MySQL server and 5 Nginx web servers.
Update 2014 06 26:
Dealing With Obstacles at Scale, Bob Hagemann, Twilio
To reduce pain:
– UTC timezone
– UTF8
– use thin AMI and chef/puppet instead of thick AMI
– wrote boxconfig a few years ago (like netflix asgard)
– remote admin mainly
– small teams 3-8
– services should run in 3 AZs
– monitoring with nagios, cron, pingdom
– haproxy on each host as proxy
– MySQL, MHA, LVM. Manual failover.
– SQS DLQ
– global low latency with route53
– http://github.com/twilio
– @bobzilla42
– Uses freeswitch plus own telcom sw
– billing system 100s QPS
– Ops team is about 8 people
– VPNs to HQ and carrier-approved colo
– three founders, one came from Amazon.
925 Market Street, SF
June 4 – 27, 2014 (likely closed on the 27th for dismantling)
Free registration, tshirts and lunch. Closes 5:30 pm, 6:00 pm or 8:00 pm daily.
Muni 30 and 45 return from Market St. and 5th to Caltrain.
@AWSstartups #AWSloft
I recently did some work with liquibase. Here’s some techniques for advanced users to workaround limitations to calculate query cost.
Liquibase Introduction
Liquibase is an Open Source (Apache 2.0 License) Java utility and API for specifying and versioning schema changes (DDL) for several popular databases. It is commonly introduced to projects by programmers, rather than DBAs.
What liquibase can do:
What liquibase can’t do:
How liquibase works:
How to Make Liquibase Consider Cost for MySQL
After some experimentation, there’s a couple liquibase features you can use to do more advanced things:
changeset.xml:
<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.1.xsd">
<changeSet id="1" author="james">
<sql>
create table if not exists `profiling` ( `connection_id` int(11) not null default 0, `query_id` int(11) not null default '0', `state` varchar(40) default '', KEY (query_id));
truncate table profiling;
set profiling=1;
alter table department add column test2 int default null;
insert into profiling (connection_id, query_id, state) select connection_id(), query_id, state from information_schema.profiling where query_id=2;
</sql>
<rollback>
<sql>alter table department drop column test2</sql>
</rollback>
</changeSet>
<changeSet id="1-post" author="james">
<preConditions onFail="HALT">
<sqlCheck expectedResult="0">SELECT count(*) from profiling where state='copy to tmp table'</sqlCheck>
</preConditions>
</changeSet>
</databaseChangeLog>
Notes:
The workaround for those 2 issues is to combine the two techniques in a shell script:
#!/bin/bash
liquibase tag rel0
liquibase update changeset.xml || {
# fail the build pipeline to not propagate changeset to next stage
# (ie. don't run in production)
liquibase rollback rel0
mysql -e 'alter table test.department drop column test2'
exit 1
}
The above looks a little kludgy, but provides a stepping stone for the reader to customize in their particular environment. (The preConditions and bash script can be easily autogenerated with a Perl or Python script.)
An alternative to XML is using the Java API to set everything up.
Please leave a comment if you have any suggestions or a Java API program.
The Percona Live MySQL Conference was held once again in Santa Clara from April 1-4, 2014.
Executive Summary:
Wed. Keynotes
Percona Live 2014 opening keynote with Percona CEO Peter Zaitsev
Robert Hodges – Getting Serious about MySQL and Hadoop at Continuent
(Continuent needs to pivot into another market as MySQL’s new built-in features displace their replication products.)
‘Raising the MySQL Bar’ with Oracle’s Tomas Ulin, VP of Engineering for MySQL, Oracle
Adventures in MySQL at Dropbox, Renjish Abraham
Wed. Talks
Online schema changes for maximizing uptime, David Turner, Dropbox, Ben Black, Tango
– MySQL 5.6 has online schema change capability, however there’s no way to throttle IO consumed during the operation and the single-threaded slave will lag
– David has tested the ALTER ONLINE in MySQL 5.6.17 and will use it when ported to Percona Server
– for now uses Percona Online Schema Change utility for its throttling feature.
– tools have been created by Percona to recover Innodb data if you don’t have backups and you’re out of business otherwise. Call them! 🙂
Galera Cluster New Features, Seppo Jaakola, Codership
– reviewed features in Galera Cluster versions 3 and 4
– looking good.
MySQL Cluster Performance Tuning, Johan Andersson, severalnines.com
- Disable NUMA - echo 0 > /proc/sys/vm/swappiness - bind data node threads to CPUs - cat /proc/interrupts ThreadConfig LDM = cores/2 TC = LDM/4 RealTimeScheduler=1 Numoffraglogparts=LDM Tune redo log Fragmentlogsize=256M Nooffragmentlogfiles=redobuffer=64M
Practical sysbench, Peter Boros, Percona
– prefers “latency” graph style with transparent dots vs. line charts
– uses R and ggplot2 for graphing
– attendees tried to guess SSD performance on Peter’s notebook for different block sizes, most were proven totally wrong by sysbench
Birds of a Feather (BoF) Sessions
“Meet MySQL Team (at Oracle)” BoF
– discussion again this year about parallel query execution (same as at MariaDB BoF last year), with Peter Zaitsev also bringing it up again
– discussion about raw partitions (belief is that they will be 20% more space-efficient and 30% faster, and avoid Linux endless limitations and bugs)
– internal “development roadmap” only extends about 12 months at a time, subject to customer demands
– I griped about FK panic/data loss issues in MySQL Cluster 7.3.3. Tomas Ulin, Vice President, MySQL Engineering, said that was news to him. (See SR 3-8717994851 and SR 3-87646727311)
– Mark Callaghan, Facebook, said he was working on MongoDB now, but requested named keys in flexible schema in MySQL.
– Peter Zaitsev, Percona, said several clients are using GTIDs and they seem to work.
– Oracle pleaded with users to drop MyISAM. I mentioned the main reason was that legacy systems used older compression methods, but InnoDB could be used since it has compression too
– The Oracle MySQL Fabric project is an attempt to counter MongoDB’s automatic slave promotion.
Thursday
Thursday Keynotes
‘9 Things You Need to Know…’, Peter Zaitsev, Percona
The Evolution of MySQL in the All-Flash Datacenter, Nisha Talagala, Fusion-IO
MySQL, Private Cloud Infrastructure and OpenStack, Sean Chighizola, Big Fish Games
Keynote Panel: The Future of Operating MySQL at Scale
Thu. Talks
Benchmarking Databases for Scale, Peter Boros and Kenny Gryp, Percona
Question: “What is Percona’s secret to professional benchmarks?”
Answer: “Benchmark absolutely everything multiple times, time permitting.”
MySQL 5.7: Performance & Scalability Benchmarks, Dimitri KRAVTCHUK
– comprehensive micro-benchmarking graphs of 5.7 to gain a deeper understanding of parts
– the challenge remains: how to tune the whole database to perform well?
– good comparison of relational data modelling and C* data modelling, lots of similarities
– note that MariaDB has a Cassandra plugin
RDS for MYSQL, Tips, Patterns and Common Pitfalls, Laine Campbell, Blackbird (formerly PalominoDB)
Write Conflicts in Multi-Master Replication Topologies, Seppo Jaakola, Codership
– it’s good to see that Codership is paying attention to the details of replication
MySQL Community Awards
Shlomi has a comprehensive post on this years winners.
MySQL Lightning Talks (5 minutes each)
Truncating Sub Optimal DBA Verbal Responses Vectors, David Stokes (Oracle)
MySQL 5.6 Global Transaction IDs: Benefits and Limitations, Stephane Combaudon (Percona)
mysqlfailover
mysqlrpladmin
Zero database downtime using the Federated storage engine and Replication, prasad mani (BBC)
Scaling via adding a Table, Rick James (self)
Rick knows some clever ways to optimize solutions with MySQL. He’s doing consulting now, so contact him.
IPs
Lat/Long
Mysql.rjweb.org
Extra Table Saves the Day: Slides
No es ‘ano’, es ‘año’! A take on encoding in your DB, Ignacio Nin (Vivid Cortex)
What Not to Say to the MySQL DBA, Gillian Gunson (Blackbird (formerly PalominoDB))
“I’ll code around it. ”
“Stop micro-optimizing. ”
“Use passive master for QA”
“MySQL is a toy database. ”
This conference is a support group. ”
Hall of Shame, Shlomi Noach
Triple active-replication in gaming anecdote: don’t do that.
The bash slave-prefetch oneliner, Art van Scheppingen (Spil Games)
Unsung Relay Log, Vishnu Rao, FlipKart
Com_relaylog_dump for tungsten and mysql 5.5
Unique User Count — Rollup, Rick James (self)
Formula for user visit estimation by counting bits.
Logical Backups in the Cloud, Bill Karwin, Percona
Backups for PHP designers
PHP class Mysql/Dump
How to Squat, Kyle Redinger (VividCortex, Inc)
Iron DBA Replication Challenge, Attunity
Humor
Friday
Friday Keynotes
Percona CMO Terry Erisman opens the 3rd and final day of Percona Live 201
– one of the best conference keynotes ever, and a great primer on Open Source marketing … up there with the O’Reilly Open Source Conference keynote on the importance of Android – before it shipped.
Friday Talks
Global Transaction ID at Facebook, Evan Elias, Santosh Banda and Yoshinori Matsunobu, Facebook
– just write your own MySQL branch if a feature is too hard to deploy 🙂
R for MySQL DBAs, Ryan Lowe and Randy Wigginton, Percona
– R has about 1,000 interesting sample databases (demos included diamonds and cars)
– good interface for quick graphing, not so great for complex programs
– Percona usess R and ggplot graph module for most of the graphs you see now.
MariaDB for Developers, Colin Charles, Chief Evangelist, MariaDB
Closing Prize Drawing
About 30 high-end gifts were handed out.
Some nice prizes contributed by exhibitors, including Nexus 7 tablets, $250 AWS gift certificates, SQLyog and Monyog licenses, and a quad drone!
Exhibits
The exhibits are one of my favorite things at the conference each year because of how strong the MySQL third-party community is.
Some notable absences were Clustrix and Violin memory, but those were offset by new exhibitors. Webyog was a sponsor but I didn’t see a booth. PalominoDB changed their name to Blackbird, and appear to be offering DevOps as well as DBA services.
And of course, as the organizers, Percona had a large, central spread. 🙂
Thanks to the sponsors and exhibitors for making a conference like this financially possible.
Facebook Debuts Web-Scale Variant Of MySQL
Facebook’s Yoshinori Matsunobu on MySQL, WebScaleSQL & Percona Live
Twitter’s Calvin Sun on WebScaleSQL, Percona Live
Slides
Tweets about PerconaLive
Percona Live MySQL Conference Highlights
Most of the Cassandra rollouts I’ve heard about at conferences have been “Devopsed” – written by Dev and productionized by Dev, with hand-off to Operations long afterwards.
That’s the opposite to how RDBMS projects are usually deployed in large companies.
As Cassandra becomes more mature, this hand-off will occur earlier after development ends.
Here is a checklist for handing off a Cassandra database to Operations (I only consider non-trivial rings of 3 or more nodes in production with a full data set):
| Node Impact | |||||
|---|---|---|---|---|---|
| Item | Comments | Performance/ | Space/ | Time/IOPs/BW | |
| Cassandra Server Version | Should be exactly the same minor version across cluster except briefly during server updates | ||||
| Token or vnodes? | needs to be configured before first start of server | ||||
| Cassandra Client/Connector Version | Thrift or CQL? | ||||
| Snitch name? Why? | several choices | ||||
| Replication Factor (RF)? Why? | usually RF=3 for SoT* data, defined at keyspace level | ||||
| Compaction method? Why? | Size or Level, defined at CF level | ||||
| Read Consistency Level? Why? | Netflix recommends CL=ONE. ALL seldom makes sense. | ||||
| Write Consistency Level? Why? | ALL seldom makes sense. | ||||
| TTL? Why? | Defined at row level. | ||||
| Expected Average Query Latency | 10 ms is reasonable, 1 ms is tough. | ||||
| nodetool repair/scrub | needed weekly | yes | more space | more | |
| Bootstrapping a new node | yes | yes | |||
| Java gcpause | stop the world | yes | yes | ||
| Are there any wide columns? do they get wider over time? | pathological case for Cassandra | yes | more space | more | |
| Backup | in case of application bug or a disaster. Opscenter, Priam, custom. | yes | slightly more for incremental backups, double for local cold copy | more | |
| Restore | requires Cassandra node shutdown | yes | |||
| If a storage volume fills, howto fix it? | Especially a problem with multiple JBOD volumes, which fill unevenly. | yes | less space | less | |
| If a storage volume fails, howto fix it? | yes | less space | less | ||
| What is the total data size now? Projected in 12 months? | affects most operations | yes | yes | yes | |
| What is the acceptable query latency? | affects network and hardware choices | ||||
| What is the best maintenance window time each week? | |||||
| What are the business and practical SLAs? | |||||
| What training is needed for your Operations team? | Datastax Admin and Data Modelling Classes (recommend most recent Cassandra version) | ||||
| What partitioner is used? | Opscenter only supports random partitioner or murmur 3 partitioner for rebalancing | ||||
| What procedures need to be written for your Operations team? | |||||
| What monitoring tools? |
|
||||
| What bugs have been encountered? Which ones still apply? | |||||
| What lessons can Devops share with the Operations team? | |||||
SoT = Source of Truth
About Data Consistency in Cassandra
ConstantContact techblog: Cassandra and Backups
stackoverflow.com: Do I absolutely need a minimum of 3 nodes/servers for a Cassandra cluster or will 2 suffice?
Cassandra Parameters Calculator
Adding vnodes to an existing cluster
Adding a new statement or command to the MySQL server is not difficult.
First, decide if you want to modify the server source code, or if a User-Defined Function (UDF) will meet your needs.
Since I just added the SHUTDOWN server command, I thought I would be helpful to outline the steps needed to add a new command.
Prerequisites:
# CentOS yum install cmake gcc make bison # Ubuntu apt-get update apt-get install cmake gcc make bison
# unpack the MySQL source code:
tar zxvf - < mariadb-5.5.30.tar.gz
# most of the files you need to modify are in this directory:
cd mariadb-5.5.30/sql
# add the token(s) (commands and arguments you think you will need) and verify the syntax:
bison -v sql_yacc.yy
# if you get warnings, fix %expect in sql_yacc.cc
# cut-and-paste a code block from a command with similar syntax in sql_yacc.cc to implement your new command, and build a test version of MySQL
# build your new server in a sandbox:
make.sh:
#!/bin/bash cd mariadb-5.5.30 cmake . -DCMAKE_INSTALL_PREFIX:PATH=/usr/local/mariadb-5.5.30 make --with-debug sudo make install
# test your new server with 3 terminal windows:
start.sh:
#!/bin/bash killall mysqld /usr/local/mariadb-5.5.30/bin/mysqld_safe --user=mysql --debug & tail -f /tmp/mysqld.trace | grep Got & tail -f /var/log/mysqld.log & mysql -u root -p # login, then test your new command while watching the log and trace
# read /var/log/mysqld.log and /tmp/mysqld.trace for errors and panics like this:
Version: '5.5.30-MariaDB-debug' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution mysqld: /home/james/mariadb-5.5.30/sql/sql_parse.cc:4477: int mysql_execute_command(THD*): Assertion `0' failed. 130515 11:25:19 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware.
The above panic was caused by the SQLCOM_ switch falling through, because the new command was not defined yet.
# When you’re done, make a test
vi mysql-test/t/my_new_command.test
# Create a patch file:
mv mariadb-5.5.30 mariadb-5.5.30-new tar zxvf - < mariadb-5.5.30.tar.gz cd mariadb-5.5.30/src >patch.txt for i in sql_parse.cc sql_yacc.yy sql_prepare.cc mysqld.cc sql_lex.h; do echo $i diff -u $i ../../mariadb-5.5.30-new/sql/ >>patch.txt done # don't forget mysql-test/t/my_new_command.test
# apply your patch file:
patch -b < patch.txt
# do a build and test your patch before distributing it.
Easy peasy, right! 🙂
Sergei Golubchik wrote on the MariaDB developers list: "Reserved words are keywords (listed in the sql/lex.h) that are
not listed in the 'keyword' rule of sql_yacc.yy (and 'keyword_sp' rule, that 'keyword' rule includes)."
How can I get the output of the DBUG_PRINT
How to find shift/reduce conflict in this yacc file?
MariaDB Contributor Agreement (MCA) Frequently Asked Questions
wikipedia: diff
MySQL Internals Manual
mysqlperformanceblog.com: XtraDB / InnoDB internals in drawing
Overloading Procedures
innodb_diagrams project
Understanding MySQL Internals By Sasha Pachev (O'Reilly)
DTrace can tell you what MySQL is doing
MySQL C Client API programming tutorial
MySQL 5.1 Class Index
Keywords: MariaDB, MySQL server programming, tutorial, patch.
At the OSCON 2011 MariaDB Birds-of-a-Feather (BoF) session, I suggested adding a MySQL SHUTDOWN statement to Monty, which was written up as WL#232. Other databases have this feature, and it’s very handy when automating management of a cluster of MySQL servers.
And at the Percona Live MySQL Conference 2013, Monty suggested to MariaDB BOF attendees that a good way to get a new feature added is to to write a patch to pave the way for a committer to start with.
Phase 1
So … I sat down last nite and wrote the patch against MariaDB 5.5.30.
Basically it meant telling mysql’s lex/yacc files to parse “shutdown”, then calling the existing MySQL API shutdown kill_mysql() function.
This code is released under the Open Source BSD-new License, according to the MariaDB Contributor Agreement.
shutdown_0.1.patch.txt – MariaDB 5.5.30:
--- sql_parse.cc 2013-03-11 03:29:13.000000000 -0700
+++ /home/james/mariadb-5.5.30-new/sql/sql_parse.cc 2013-05-15 13:17:05.000000000 -0700
@@ -1305,7 +1305,6 @@
my_ok(thd);
break;
}
-#ifndef EMBEDDED_LIBRARY
case COM_SHUTDOWN:
{
status_var_increment(thd->status_var.com_other);
@@ -1333,7 +1332,6 @@
error=TRUE;
break;
}
-#endif
case COM_STATISTICS:
{
STATUS_VAR *current_global_status_var; // Big; Don't allocate on stack
@@ -3736,6 +3734,31 @@
lex->kill_signal);
break;
}
+ case SQLCOM_SHUTDOWN:
+ {
+ // jeb - This code block is copied from COM_SHUTDOWN above. Since kill_mysql(void) {} doesn't take a level argument, the level code is pointless.
+ // jeb - In fact, the level code should be removed and Oracle Database statements implemented: SHUTDOWN, SHUTDOWN IMMEDIATE and SHUTDOWN ABORT. See WL#232.
+
+ status_var_increment(thd->status_var.com_other);
+ if (check_global_access(thd,SHUTDOWN_ACL))
+ break; /* purecov: inspected */
+
+ enum mysql_enum_shutdown_level level;
+ level= SHUTDOWN_DEFAULT;
+ if (level == SHUTDOWN_DEFAULT)
+ level= SHUTDOWN_WAIT_ALL_BUFFERS; // soon default will be configurable
+ else if (level != SHUTDOWN_WAIT_ALL_BUFFERS)
+ {
+ my_error(ER_NOT_SUPPORTED_YET, MYF(0), "this shutdown level");
+ break;
+ }
+ DBUG_PRINT("SQLCOM_SHUTDOWN",("Got shutdown command for level %u", level));
+ my_eof(thd);
+ kill_mysql();
+ res=TRUE;
+ break;
+ }
+
#ifndef NO_EMBEDDED_ACCESS_CHECKS
case SQLCOM_SHOW_GRANTS:
{
--- sql_yacc.yy 2013-03-11 03:29:19.000000000 -0700
+++ /home/james/mariadb-5.5.30-new/sql/sql_yacc.yy 2013-05-15 11:12:03.000000000 -0700
@@ -791,7 +791,7 @@
Currently there are 174 shift/reduce conflicts.
We should not introduce new conflicts any more.
*/
-%expect 174
+%expect 196
/*
Comments for TOKENS.
@@ -1645,6 +1645,7 @@
definer_opt no_definer definer
parse_vcol_expr vcol_opt_specifier vcol_opt_attribute
vcol_opt_attribute_list vcol_attribute
+ shutdown
END_OF_INPUT
%type call sp_proc_stmts sp_proc_stmts1 sp_proc_stmt
@@ -1796,6 +1797,7 @@
| savepoint
| select
| set
+ | shutdown
| signal_stmt
| show
| slave
@@ -13715,6 +13717,17 @@
;
+shutdown:
+ SHUTDOWN
+ {
+ LEX *lex=Lex;
+ lex->value_list.empty();
+ lex->users_list.empty();
+ lex->sql_command= SQLCOM_SHUTDOWN;
+ }
+ ;
+
+
set_expr_or_default:
expr { $$=$1; }
| DEFAULT { $$=0; }
--- sql_prepare.cc 2013-03-11 03:29:11.000000000 -0700
+++ /home/james/mariadb-5.5.30-new/sql/sql_prepare.cc 2013-05-15 03:07:00.000000000 -0700
@@ -2173,6 +2173,7 @@
case SQLCOM_GRANT:
case SQLCOM_REVOKE:
case SQLCOM_KILL:
+ case SQLCOM_SHUTDOWN:
break;
case SQLCOM_PREPARE:
--- mysqld.cc 2013-03-11 03:29:14.000000000 -0700
+++ /home/james/mariadb-5.5.30-new/sql/mysqld.cc 2013-05-15 01:20:11.000000000 -0700
@@ -3333,6 +3333,7 @@
{"savepoint", (char*) offsetof(STATUS_VAR, com_stat[(uint) SQLCOM_SAVEPOINT]), SHOW_LONG_STATUS},
{"select", (char*) offsetof(STATUS_VAR, com_stat[(uint) SQLCOM_SELECT]), SHOW_LONG_STATUS},
{"set_option", (char*) offsetof(STATUS_VAR, com_stat[(uint) SQLCOM_SET_OPTION]), SHOW_LONG_STATUS},
+ {"shutdown", (char*) offsetof(STATUS_VAR, com_stat[(uint) SQLCOM_SHUTDOWN]), SHOW_LONG_STATUS},
{"signal", (char*) offsetof(STATUS_VAR, com_stat[(uint) SQLCOM_SIGNAL]), SHOW_LONG_STATUS},
{"show_authors", (char*) offsetof(STATUS_VAR, com_stat[(uint) SQLCOM_SHOW_AUTHORS]), SHOW_LONG_STATUS},
{"show_binlog_events", (char*) offsetof(STATUS_VAR, com_stat[(uint) SQLCOM_SHOW_BINLOG_EVENTS]), SHOW_LONG_STATUS},
--- sql_lex.h 2013-03-11 03:29:13.000000000 -0700
+++ /home/james/mariadb-5.5.30-new/sql/sql_lex.h 2013-05-15 01:19:17.000000000 -0700
@@ -193,6 +193,7 @@
SQLCOM_SHOW_RELAYLOG_EVENTS,
SQLCOM_SHOW_USER_STATS, SQLCOM_SHOW_TABLE_STATS, SQLCOM_SHOW_INDEX_STATS,
SQLCOM_SHOW_CLIENT_STATS,
+ SQLCOM_SHUTDOWN,
/*
When a command is added here, be sure it's also added in mysqld.cc
To apply:
tar zxvf - < mariadb-5.5.30.tar.gz cd mariadb-5.5.30/sql wget http://jebriggs.com/php/shutdown_0.1.patch.txt patch -b < shutdown_0.1.patch.txt
make.sh:
#!/bin/bash cd mariadb-5.5.30 cmake . -DCMAKE_INSTALL_PREFIX:PATH=/usr/local/mariadb-5.5.30 make --with-debug sudo make install
start.sh:
#!/bin/bash killall mysqld /usr/local/mariadb-5.5.30/bin/mysqld_safe --user=mysql --debug & tail -f /tmp/mysqld.trace | grep Got & mysql -u root -p
mysql client (with mysqld.log and mysql.trace entries overlaid):
mysql> shutdown; ERROR 2013 (HY000): Lost connection to MySQL server during query mysql> 130515 13:20:38 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
/tmp/mysql.trace:
T@4 : | | | >parse_sql T@4 : | | | <parse_sql T@4 : | | | >LEX::set_trg_event_type_for_tables T@4 : | | | <LEX::set_trg_event_type_for_tables T@4 : | | | >mysql_execute_command T@4 : | | | | >deny_updates_if_read_only_option T@4 : | | | | <deny_updates_if_read_only_option T@4 : | | | | >stmt_causes_implicit_commit T@4 : | | | | <stmt_causes_implicit_commit T@4 : | | | | SQLCOM_SHUTDOWN: Got shutdown command for level 16 T@4 : | | | | >set_eof_status T@4 : | | | | <set_eof_status T@4 : | | | | >kill_mysql T@4 : | | | | | quit: After pthread_kill T@4 : | | | | <kill_mysql T@4 : | | | | proc_info: /home/james/mariadb-5.5.30/sql/sql_parse.cc:4507 query end
/var/log/mysqld.log:
130515 13:20:08 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql 130515 13:20:08 InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!! 130515 13:20:08 InnoDB: The InnoDB memory heap is disabled 130515 13:20:08 InnoDB: Mutexes and rw_locks use GCC atomic builtins 130515 13:20:08 InnoDB: Compressed tables use zlib 1.2.3 130515 13:20:08 InnoDB: Initializing buffer pool, size = 128.0M 130515 13:20:08 InnoDB: Completed initialization of buffer pool 130515 13:20:08 InnoDB: highest supported file format is Barracuda. 130515 13:20:09 InnoDB: Waiting for the background threads to start 130515 13:20:10 Percona XtraDB (http://www.percona.com) 5.5.30-MariaDB-30.1 started; log sequence number 1597945 130515 13:20:10 [Note] Plugin 'FEEDBACK' is disabled. 130515 13:20:10 [Note] Event Scheduler: Loaded 0 events 130515 13:20:10 [Note] /usr/local/mariadb-5.5.30/bin/mysqld: ready for connections. Version: '5.5.30-MariaDB-debug' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution 130515 13:20:37 [Note] Got signal 15 to shutdown mysqld 130515 13:20:37 [Note] /usr/local/mariadb-5.5.30/bin/mysqld: Normal shutdown 130515 13:20:37 [Note] Event Scheduler: Purging the queue. 0 events 130515 13:20:37 InnoDB: Starting shutdown... 130515 13:20:38 InnoDB: Shutdown completed; log sequence number 1597945 130515 13:20:38 [Note] /usr/local/mariadb-5.5.30/bin/mysqld: Shutdown complete 130515 13:20:38 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
A possible test would be like this, but it would interfere with operation of the test mysqld instance:
mysql-test/t/shutdown.test:
shutdown;
Phase 2
My above patch applies cleanly within the existing MySQL shutdown framework, which implements a feature like Oracle Database's SHUTDOWN IMMEDIATE command.
However, my patch is a Pyrrhic victory, since there's so much wrong with MySQL's existing shutdown framework that it will take an internals committer to sort it out.
The shutdown framework is badly designed, if it was designed at all, since it fails the "does this feel programmed on purpose?" test, and in fact doesn't work reliably:
I guess I'll have to pay myself the worklog bounty of $100. 🙂
This is actually my second MySQL patch contribution. In 1997 or 1998 I submitted a patch for the installer, which was one of the most troublesome components at that time. Monty rewrote it, but I liked my version better.
Update: Sergei Golubchik committed this patch to MariaDB 10.0.4 on 2013-06-25. Thanks, Sergei!
shutdown_0.1.patch.txt
MySQL's Missing Shutdown Statement
WL#232
Bug #63276: skip sleep in srv_master_thread when shutdown is in progress
Lion Air flight 610 crashed on October 29, killing all 189 people on board. Also, one rescue diver involved in the recovery drowned.
Having flown in Jakarta, Indonesia before, I can comment on the challenges that airlines face there:
When you combine the above with an advanced, new aircraft type like the 737 MAX, you get an accident like this.
It appears that the AOA sensor was faulty, requiring disengagement of both the AP and the autothrottles to override, then manual flying using an AI indicator. Update 2018-11-13: Boeing appears to have made a change to the autopilot to push the nose down without documenting it for pilots, even with the autopilot disengaged. Possibly this was to increase the appearance of commonality, thus reducing training requirements.
The average North American airline pilot can easily handle this situation, as they have significant VFR pilotage and systems knowledge.
Fantastic presentations by American Airlines Training Captain Warren VanderBurgh:
aviationtoday.com: FAA Issues Boeing 737 AOA Directive After Lion Air Crash
avweb.com: Lion Air Investigation Continues
W: Boeing 737 MAX
pprune.org: Indonesian aircraft missing off Jakarta
ainonline.com: Indonesian Airlines Draw More Scrutiny in Wake of Latest Crash
Obituary: Captain Warren “Van” Vanderburgh
Revealed: The 21 ‘worst’ airlines in the world
Indonesian plane grounded after passengers complain about stinky durian
ainonline.com: Boeing Guidance on 737 Max Stall Protection System Under Fire
wsj.com: Boeing Withheld Information on 737 Model, According to Safety Experts and Others
U.S. aircraft operators are required to have digital position reporting (ADS-B Out) installed, working and certified by Jan. 1, 2020.
There are over 100,000 airplanes in the USA …
But with only a year to go, there’s some very interesting wrinkles per AVweb reader comments:
I think the lessons are:
The ADS-B NPRM has a fascinating economic analysis of the roll-out cost for operators. The FAA was well-aware of the significant costs involved to re-equip aircraft. If you like horror movies, check it out. 🙂
avweb.com: ADS-B Apocalypse
cincinnatiavionics.com: What’s the difference between 1090 and 978?
ainonline.com: FAA To Mandate ADS-B Equipage by 2020 (2007)
faa.gov: NPRM for Automatic Dependent Surveillance – Broadcast (ADS-B) Out performance requirements to support Air Traffic Control (ATC) service
Hi, I'm James Briggs, an Internet computer programmer/DBA/SRE in Silicon Valley. I've been engineering for over 20 years for companies like eBay, Netflix, Yahoo!, Apple, Amazon and startups.
I am available for MySQL/Cassandra DBA/SRE work in Silicon Valley or remote. Click here to contact me.
I am a MariaDB/MySQL code contributor and Perl CPAN author. Check out my github. My most recent project is cassandra-top.
In my spare time I do Open Source programming and like flying airplanes, karate and studying Asian languages.
This blog talks about events, places and things in my life.
San Jose, San Jose International Airport, CA
Last Updated on Nov 20 2018, 4:53 pm PST
Weather by NOAA
Current Conditions: Overcast with Haze
Temp: 61°F
Wind: NW at 4mph
Humidity: 50%
Windchill: 61°F
RackPing Graphs
