These are my raw notes from talks held on Monday at LCA2014. May contain errors, mis-heard quotes. Also completely un-reviewed or spell checked:
Keynote:
Dr Suelette Dreyfus - The Surviellance State
- rights of citizens and state are out of balance
- Not enough awareness by the citizens of the survielleance state
- world whistleblowing survey n=2349
- 68% of people see whistle blowers as normal of heros, not rats.
- 50% of Australian think there is too much secrecy
- 81% support whistleblowers actions
- 87% - 90% support protection for going to the media
- A period of universal deceit
- Open source people are revolutionary by there very actions <- said this faulteh
- Telling the truth is revolutionary
- Irony of Russia being a safe haven from the UK, US, Australia
- So much moeny being spent they can't figure out how to spemndit all
- Insight Platform - Cradle to Graduation tracking of children
- Tender process / details are not transparent
- DELL PowerEdge 1950 2950 are compromised with GODSURGE
- iOS is 100% successfully hacked.
- iOS is completely compromised
- shitload of hardware hacks acheived via interdiction
- Call out unacceptable use
- State surviellance is being used for individual financial benefit by ministers and friendly companies
- What can you do?
- Get political - speak up for provacy protections
- Write user friendly privacy enhancing software - help people detect intrusions by govt.
- Get involved in NFPs that give tech support to journos, NGOs etc
- Have the courage to speak out, particularly if you work for government
- Slides
- The Foreman is a complete lifecycle management tool for physical and virtual servers.
- Foreman supports libvirt provisioning
- Provides a web interface for configuration management
- Monitors and logs facts and history
- foreman-ansible - An Ansible ENC (external node classifier) to get, group and classify nodes based on the Puppet facts in The Foreman.
- Slides
- Focussed on using the Assimilation Project
- Use the process of discovery to locate "forgotten" machines
- Discovers:
- IP and MAc addresses
- Services and service details
- Switches, connections and settings
- Installed services
- OS Configuration
- anything else we like
- Slides
- Vagrant manages VMs
- Uses Ansible to configure VMs it builds
- Excellent testing framework
- One command for building and testing environments
- Slides
- Uses PXE boot and Debian's live-boot to install images
- Trimclient Admin keeps tables of images, realms and hosts
- Allows the grouping of hosts in realms
- Slides
- Without understanding how your hardware buffers, applying QoS can cause great congestion
- Some devices have buffers so large you need to apply QoS
- Low Latency == small buffers, frequent drops
- High Throughput == large buffers, few drops
- In practice, balance both.
- Software Routers
- Buffer sizes are config options in Linux
- Take a look at cerowrt
VOIP does not need specific QoS, standard QoS classes are adequate.
- Usually due to bad vendor networks
- Slides
- Carambola2
- No video etc
- Runs OpenWRT
- Uses this combination for monitoring devices (aircon, solar panels etc)
- Extremely low power usage
- Supports loads of IO inputs
- Slides
- Australia lags IPv6 deployment (no surprise)
- Husk is a netfilter wrapper
- Log and drops by default
- Supports hooks
- Not a complete abstraction
- Not automatic
- Human readable rules
- Named interfaces (ie: ppp0 -> NET)
- Helpers are built in
- Atomic loads
- Logged to syslog
- Need to see how that compares to UFW.
- BoD reckons Husk is better for production deployments
- Slides
- virtual memory is not SWAP
- anon mempry roughly corresponds to memory used by applications
- file memory roughly corresponds to memory ins use by file systems and IO
RatticDB - Elizabeta Sørensen
- Slides
- Encryption is done on the filesystem
- Needs HTTPS
- Any DB supported by Django
- Demo U: admin P:rattic
- Manages audit log of what passwords people have seen
- easy way to manage change when staff leave
- Chef recipes are strictly DSL whereas Puppet feels more like XML
- Chef handles dependencies better that Puppet
- Documentation is generally better in Chef.
- Chef is more strict
- Puppet has a better UI
- Chef has better tools
- Better community support
- Chef uses Chef to install Chef
Howto Reliably Replicate Block Devices, even over Long Distances - Thomas Schöbel-Theuer
- DRBD vs Mars Light
- Mars Lights is more tolerant of network faults
- DRBD has decreasing thoughput over time after disconnects
- 15 pilot clusters
- Rolling out to > 250 clusters
- Looking for more developers
- New domain coming http://mars.technology/ (?)
- IO latency over networks is much better in Mars Light
Better Living Through Statistics - Jamie Wilkinson
- Slides
- Making monitoring suck less
- Blackbox vs Whitebox
- Blackbox:
- Only boolean
- No insight into "why"
- Problems with check+alert
- THresholds vary
- Extra work to add targets
- Monitoring sucks because existing models do not scale.
- Should base monitoring on rates and derivitives
- Duration of abnormality should be considered.
- Aggregate groups of machines data for analysis
- rate of errors vs rate of queries
- What t LAert on:
- Rate of Change of QPS outside normal
- Ratio of errors to queries
- Latency
- Make sure alerts are actionable
- and Documented
- Blackbox testing is still required
- Keep counters in stead of guages
- Compare them
- Run statistical packages to timeseries DB and experiment
- The tools don't exist yet
Performance Co-pilot, checkmk are possible solutions. monitoring@lists.linux.org.au is a useful disucssion place
Running virtualized Galera instances - Raghavendra Prabhu
- Sheepdog does storage virtualisation for QEMU.
The Six Stages of systemd - Rodger Donaldson
- Denial
- Claiming there's nothing wrong with the existing init system
- Anger
- Raging against the machine
- Bargaining
- but No one is realing looking with sysV
- Depression
- Change is scary
- May even use BSD (no!)
- Acceptance
- Tinkering
- Learning
- Lead to...
- Enthusiasm
- Discovering the benefits of the new system
- What do I want?
- Reliability
- Deterministi View
- Enforceable polict
- Life's to short for sysV script repition
- "Enterprise" software and "trendy" new frameworks never have sysV init scripts
- Value spending time efficiently
- systemd has:
- efficient parameters
- includes allow efficient variation (a-la apache2)
- Simplfies service management
- Reduces risk
- Need Service Views
- Want to manage multi-tenant systems
- Maximise Reliability and Efficiency
- Minimise overhead
- makes serice co-tenant
- Easy to Measure
- cgroups makes measuring and accountability easier
- Linux competitors are already adopting alternatives
Scaling Graphite - Devdas Bhagat
- Graphite is a web based graphing system for time series data series plots
- Written in Python
- Uses multiple daemons
- Has it's own storage backend
- Moving parts
- Whisper/Cere - storage backend
- Webapp - web front end and API provider
- Relaying daemons
- Sppeding up IO
- SSDs
- performance improved bu not enough
- One SSD failure brough down the whole system
- User simple naming conventiosn
- Collectd ran into memory problems (bug)
- Switched to Diamond instead
- Relaying
- CPU still a bottleneck
- rewrote relays in C
- They collect all the business metrics they can
- Graphite doesn't support hashing well but it works
- Switched to RAID0
- They're hiring: http://www.booking.com/jobs
Thinking outside the box - Steven McDonald
- What's the "box"?
- A confined thinking space
- confined by the person doing the thinking
- Inside the box is past experiences, common wisdom, other existing knowledge
- Why outside the box?
- Inside the box is quicker, easier and produces the same result
- You can automate box thinking
- Need to know when to leave the box
- Sometimes you need to use an imperfect solution to restore a service for most users
- Gave examples
- Too easy to focus on the root cause rather then what the customer cares about.
- Problems need to be clearly defined before it is solved, otherwise you're heading in the wrong direction before you begin.
Practical Crypto BoF
- Many crypto passphrase caching agents (like GPG agent) are vulnerable to memory dump attacks via ptrace
- Redphone for VOIP encrypted calls
- No crypto talks as LCA2014...