GlueCon 2013 Notes: The Free Lunch Is Over, Again – Andy Gross, Chief Architect at Basho
The Free Lunch Is Over, Again – Andy Gross, Chief Architect at Basho (@argv0) The Free Lunch Is Over, 2005 Paper by Herb Sutter – “A fundamental turn towards concurrency in software” Today: not just concurrency, but distributed systems Design We’re all distributed systems people now, due to: larger problems e.g. data, computation increased expectations e.g. more uptime, investors require cloud rather than bare metal But, distributed systems is hard Implementation We have a “Hacker News-driven development” approach now, with lots of vendors and language platforms The conversation has shifted from web frameworks to more academic topics and verification of approaches to the async network model There is now a resurgence of functional, declarative programming (even C++ hs closures now!) Explosion of new languages and DSLs (e.g. Bloom for distributed programming http://www.bloom-lang.net/) Use existing tools (compilers, static analysis) to verify our programs Distributed consensus is difficult – Paxos is a protocol with multiple papers on theory and simplification, ZooKeeper an implementation There is no LibPaxos yet – why not built into the OS? (VMS had DLM) Write-ahead logging and transaction recovery should also be a reusable primitive (LibARIES), but historically hard to implement (Statis is a good start http:/code.google.com/p/stasis) Why don’t systems/database developers come to distributed systems conferences and vice-versa? We all need to widen our focus Riak core is Dynamo abstracted (framework, library) Not specific to a database Multiple large production system deployments and a few university systems classes Lacking documentation Testing Can we test distributed systems to prove correctness? Unit tests are grossly insufficient QuickCheck is an improvement, allowing for high-level assertions (“properties”) that a function should fulfill, then generates millions of random test cases to try to falsify the property QuickCheck case study: Poolboy Erlang connection pool library Passed unit tests, integration tests passed, seemed to be working QuickCheck found bugs in every major piece of functionality Paper: “Hansei: property-based development of concurrent systems” by Joe Blomstedt of Basho – http://dl.acm.org/citation.cfm?id=2364505&dl=ACM&coll=DL&CFID=219097490&CFTOKEN=10777869 Ask Andy and ye shall receive
The Free Lunch Is Over, Again – Andy Gross, Chief Architect at Basho (@argv0)
- The Free Lunch Is Over, 2005 Paper by Herb Sutter – “A fundamental turn towards concurrency in software”
- Today: not just concurrency, but distributed systems
- Design
- We’re all distributed systems people now, due to:
- larger problems e.g. data, computation
- increased expectations e.g. more uptime, investors require cloud rather than bare metal
- But, distributed systems is hard
- We’re all distributed systems people now, due to:
- Implementation
- We have a “Hacker News-driven development” approach now, with lots of vendors and language platforms
- The conversation has shifted from web frameworks to more academic topics and verification of approaches to the async network model
- There is now a resurgence of functional, declarative programming (even C++ hs closures now!)
- Explosion of new languages and DSLs (e.g. Bloom for distributed programming http://www.bloom-lang.net/)
- Use existing tools (compilers, static analysis) to verify our programs
- Distributed consensus is difficult – Paxos is a protocol with multiple papers on theory and simplification, ZooKeeper an implementation
- There is no LibPaxos yet – why not built into the OS? (VMS had DLM)
- Write-ahead logging and transaction recovery should also be a reusable primitive (LibARIES), but historically hard to implement (Statis is a good start http:/code.google.com/p/stasis)
- Why don’t systems/database developers come to distributed systems conferences and vice-versa? We all need to widen our focus
- Riak core is Dynamo abstracted (framework, library)
- Not specific to a database
- Multiple large production system deployments and a few university systems classes
- Lacking documentation
- Testing
- Can we test distributed systems to prove correctness? Unit tests are grossly insufficient
- QuickCheck is an improvement, allowing for high-level assertions (“properties”) that a function should fulfill, then generates millions of random test cases to try to falsify the property
- QuickCheck case study: Poolboy Erlang connection pool library
- Passed unit tests, integration tests passed, seemed to be working
- QuickCheck found bugs in every major piece of functionality
- Paper: “Hansei: property-based development of concurrent systems” by Joe Blomstedt of Basho – http://dl.acm.org/citation.cfm?id=2364505&dl=ACM&coll=DL&CFID=219097490&CFTOKEN=10777869
- Ask Andy and ye shall receive