James Higginbotham

API Architect

GlueCon 2013 Notes: The Free Lunch Is Over, Again – Andy Gross, Chief Architect at Basho

1 minute read

The Free Lunch Is Over, Again – Andy Gross, Chief Architect at Basho (@argv0)

  • The Free Lunch Is Over, 2005 Paper by Herb Sutter – “A fundamental turn towards concurrency in software”
  • Today: not just concurrency, but distributed systems
  • Design
    • We’re all distributed systems people now, due to:
      • larger problems e.g. data, computation
      • increased expectations e.g. more uptime, investors require cloud rather than bare metal
    • But, distributed systems is hard
  • Implementation
    • We have a “Hacker News-driven development” approach now, with lots of vendors and language platforms
    • The conversation has shifted from web frameworks to more academic topics and verification of approaches to the async network model
    • There is now a resurgence of functional, declarative programming (even C++ hs closures now!)
    • Explosion of new languages and DSLs (e.g. Bloom for distributed programming http://www.bloom-lang.net/)
    • Use existing tools (compilers, static analysis) to verify our programs
    • Distributed consensus is difficult – Paxos is a protocol with multiple papers on theory and simplification, ZooKeeper an implementation
    • There is no LibPaxos yet – why not built into the OS? (VMS had DLM)
    • Write-ahead logging and transaction recovery should also be a reusable primitive (LibARIES), but historically hard to implement (Statis is a good start http:/code.google.com/p/stasis)
    • Why don’t systems/database developers come to distributed systems conferences and vice-versa? We all need to widen our focus
    • Riak core is Dynamo abstracted (framework, library)
    • Not specific to a database
    • Multiple large production system deployments and a few university systems classes
    • Lacking documentation
  • Testing
    • Can we test distributed systems to prove correctness? Unit tests are grossly insufficient
    • QuickCheck is an improvement, allowing for high-level assertions (“properties”) that a function should fulfill, then generates millions of random test cases to try to falsify the property
    • QuickCheck case study: Poolboy Erlang connection pool library
      • Passed unit tests, integration tests passed, seemed to be working
      • QuickCheck found bugs in every major piece of functionality
    • Paper: “Hansei: property-based development of concurrent systems” by Joe Blomstedt of Basho – http://dl.acm.org/citation.cfm?id=2364505&dl=ACM&coll=DL&CFID=219097490&CFTOKEN=10777869
      • Ask Andy and ye shall receive

You May Also Enjoy