craige/ blog/ 2014/
LCA2014 - Monday

These are my raw notes from talks held on Monday at LCA2014. May contain errors, mis-heard quotes. Also completely un-reviewed or spell checked:


Dr Suelette Dreyfus - The Surviellance State

  • rights of citizens and state are out of balance
  • Not enough awareness by the citizens of the survielleance state
  • world whistleblowing survey n=2349
  • 68% of people see whistle blowers as normal of heros, not rats.
  • 50% of Australian think there is too much secrecy
  • 81% support whistleblowers actions
  • 87% - 90% support protection for going to the media
  • A period of universal deceit
  • Open source people are revolutionary by there very actions <- said this faulteh
  • Telling the truth is revolutionary
  • Irony of Russia being a safe haven from the UK, US, Australia
  • So much moeny being spent they can't figure out how to spemndit all
  • Insight Platform - Cradle to Graduation tracking of children
  • Tender process / details are not transparent
  • DELL PowerEdge 1950 2950 are compromised with GODSURGE
  • iOS is 100% successfully hacked.
  • iOS is completely compromised
  • shitload of hardware hacks acheived via interdiction
  • Call out unacceptable use
  • State surviellance is being used for individual financial benefit by ministers and friendly companies
  • What can you do?
    • Get political - speak up for provacy protections
    • Write user friendly privacy enhancing software - help people detect intrusions by govt.
    • Get involved in NFPs that give tech support to journos, NGOs etc
    • Have the courage to speak out, particularly if you work for government

System Administration Miniconf

How to keep track of puppet with Foreman - Glen Ogilvie

  • Slides
  • The Foreman is a complete lifecycle management tool for physical and virtual servers.
  • Foreman supports libvirt provisioning
  • Provides a web interface for configuration management
  • Monitors and logs facts and history
  • foreman-ansible - An Ansible ENC (external node classifier) to get, group and classify nodes based on the Puppet facts in The Foreman.

Open Sourcing your entire Puppet configuration - Elizabeth Krumbach Joseph

Providing a continuously updated ITIL CMDB - Alan Robertson

  • Slides
  • Focussed on using the Assimilation Project
  • Use the process of discovery to locate "forgotten" machines
  • Discovers:
    • IP and MAc addresses
    • Services and service details
    • Switches, connections and settings
    • Installed services
    • OS Configuration
    • anything else we like

Ansible and Vagrant - Daniel Hall

  • Slides
  • Vagrant manages VMs
  • Uses Ansible to configure VMs it builds
  • Excellent testing framework
  • One command for building and testing environments

Scalable SOE deployments - Matthew Cengia

  • Slides
  • Uses PXE boot and Debian's live-boot to install images
  • Trimclient Admin keeps tables of images, realms and hosts
  • Allows the grouping of hosts in realms

'Quality of Service', a common misconception - Julien Goodwin

  • Slides
  • Without understanding how your hardware buffers, applying QoS can cause great congestion
  • Some devices have buffers so large you need to apply QoS
  • Low Latency == small buffers, frequent drops
  • High Throughput == large buffers, few drops
  • In practice, balance both.
  • Software Routers
    • Buffer sizes are config options in Linux
    • Take a look at cerowrt VOIP does not need specific QoS, standard QoS classes are adequate.
    • Usually due to bad vendor networks

Custom equipment monitoring with OpenWRT and Carambola - Andrew McDonnell

  • Slides
  • Carambola2
  • No video etc
  • Runs OpenWRT
  • Uses this combination for monitoring devices (aircon, solar panels etc)
  • Extremely low power usage
  • Supports loads of IO inputs

Dualstack Firewalling with husk - Phillip Smith

  • Slides
  • Australia lags IPv6 deployment (no surprise)
  • Husk is a netfilter wrapper
    • Log and drops by default
    • Supports hooks
    • Not a complete abstraction
    • Not automatic
  • Human readable rules
  • Named interfaces (ie: ppp0 -> NET)
  • Helpers are built in
  • Atomic loads
  • Logged to syslog
  • Need to see how that compares to UFW.
    • BoD reckons Husk is better for production deployments

Optimizing Linux memory usage - Sander van Vugt

  • Slides
  • virtual memory is not SWAP
  • anon mempry roughly corresponds to memory used by applications
  • file memory roughly corresponds to memory ins use by file systems and IO

RatticDB - Elizabeta Sørensen

  • Slides
  • Encryption is done on the filesystem
  • Needs HTTPS
  • Any DB supported by Django
  • Demo U: admin P:rattic
  • Manages audit log of what passwords people have seen
  • easy way to manage change when staff leave

Coming up to speed on Chef within AWS - Josh Mesilane

  • Chef recipes are strictly DSL whereas Puppet feels more like XML
  • Chef handles dependencies better that Puppet
  • Documentation is generally better in Chef.
  • Chef is more strict
  • Puppet has a better UI
  • Chef has better tools
  • Better community support
  • Chef uses Chef to install Chef

Howto Reliably Replicate Block Devices, even over Long Distances - Thomas Schöbel-Theuer

  • DRBD vs Mars Light
    • Mars Lights is more tolerant of network faults
  • DRBD has decreasing thoughput over time after disconnects
  • 15 pilot clusters
  • Rolling out to > 250 clusters
  • Looking for more developers
  • New domain coming (?)
  • IO latency over networks is much better in Mars Light

Better Living Through Statistics - Jamie Wilkinson

  • Slides
  • Making monitoring suck less
  • Blackbox vs Whitebox
  • Blackbox:
    • Only boolean
    • No insight into "why"
  • Problems with check+alert
    • THresholds vary
    • Extra work to add targets
    • Monitoring sucks because existing models do not scale.
  • Should base monitoring on rates and derivitives
  • Duration of abnormality should be considered.
  • Aggregate groups of machines data for analysis
  • rate of errors vs rate of queries
  • What t LAert on:
    • Rate of Change of QPS outside normal
    • Ratio of errors to queries
    • Latency
  • Make sure alerts are actionable
  • and Documented
  • Blackbox testing is still required
  • Keep counters in stead of guages
  • Compare them
  • Run statistical packages to timeseries DB and experiment
  • The tools don't exist yet

Performance Co-pilot, checkmk are possible solutions. is a useful disucssion place

Running virtualized Galera instances - Raghavendra Prabhu

  • Sheepdog does storage virtualisation for QEMU.

The Six Stages of systemd - Rodger Donaldson

  • Denial
    • Claiming there's nothing wrong with the existing init system
  • Anger
    • Raging against the machine
  • Bargaining
    • but No one is realing looking with sysV
  • Depression
    • Change is scary
    • May even use BSD (no!)
  • Acceptance
    • Tinkering
    • Learning
    • Lead to...
  • Enthusiasm
    • Discovering the benefits of the new system
  • What do I want?
    • Reliability
    • Deterministi View
    • Enforceable polict
    • Life's to short for sysV script repition
    • "Enterprise" software and "trendy" new frameworks never have sysV init scripts
    • Value spending time efficiently
  • systemd has:
    • efficient parameters
    • includes allow efficient variation (a-la apache2)
    • Simplfies service management
    • Reduces risk
  • Need Service Views
    • cgroups help with this
  • Want to manage multi-tenant systems
    • not multi VM hordes
  • Maximise Reliability and Efficiency
    • Minimise overhead
    • makes serice co-tenant
  • Easy to Measure
    • cgroups makes measuring and accountability easier
  • Linux competitors are already adopting alternatives

Scaling Graphite - Devdas Bhagat

  • Graphite is a web based graphing system for time series data series plots
    • Written in Python
    • Uses multiple daemons
    • Has it's own storage backend
  • Moving parts
    • Whisper/Cere - storage backend
    • Webapp - web front end and API provider
    • Relaying daemons
  • Sppeding up IO
    • SSDs
      • performance improved bu not enough
    • One SSD failure brough down the whole system
  • User simple naming conventiosn
  • Collectd ran into memory problems (bug)
    • Switched to Diamond instead
  • Relaying
    • CPU still a bottleneck
    • rewrote relays in C
  • They collect all the business metrics they can
  • Graphite doesn't support hashing well but it works
  • Switched to RAID0
  • They're hiring:

Thinking outside the box - Steven McDonald

  • What's the "box"?
    • A confined thinking space
    • confined by the person doing the thinking
    • Inside the box is past experiences, common wisdom, other existing knowledge
  • Why outside the box?
    • Inside the box is quicker, easier and produces the same result
    • You can automate box thinking
    • Need to know when to leave the box
  • Sometimes you need to use an imperfect solution to restore a service for most users
  • Gave examples
  • Too easy to focus on the root cause rather then what the customer cares about.
  • Problems need to be clearly defined before it is solved, otherwise you're heading in the wrong direction before you begin.

Practical Crypto BoF

  • Many crypto passphrase caching agents (like GPG agent) are vulnerable to memory dump attacks via ptrace
  • Redphone for VOIP encrypted calls
  • No crypto talks as LCA2014...
This site is powered by ikiwiki.