OpenZFS – Communities co-operating on ZFS code and features

OpenZFS is a new community founded around open-source, cross-platform ZFS projects. 

open-zfs freebsdZFS is arguably the world’s most advanced file-system and has been in active development for over a decade. It is the popular and highly-advanced 128-bit file-system with enhanced error detection and correction capabilities designed for Solaris during the Sun Microsystems days.

Different projects have continued developing ZFS, such as illumos, FreeBSD and Oracle, but to avoid further fragmentation a number of companies and communities with an interest in ZFS have joined forces and set up OpenZFS.

Prior to the formation of OpenZFS there was little or no co-ordination [1] between the different ZFS related projects and the implementations on different operating system, but OpenZFS is to change that and to promote collaboration between cross-project developers and stakeholders.

The high-level goals of OpenZFS are:

  • to raise awareness of the quality, utility, and availability of open source implementations of ZFS
  • to encourage open communication about ongoing efforts to improve open source ZFS
  • to ensure consistent reliability, functionality, and performance of all distributions of ZFS.

OpenZFS has been founded by members of the Linux, FreeBSD, Mac OS X, and illumos communities, including Matt Ahrens, one of the two original authors of ZFS.

The OpenZFS community brings together over a hundred software developers and companies with the aim to improve and further develop ZFS. Some well known companies taking part are iXsystem, HybridCluster, Nexenta and PogoLinux.

The notion “open” in OpenZFS should to be stressed. Oracle has further developed ZFS (e.g. v35) but hasn’t made the code changes public, but the OpenZFS project will be open, share and encourage co-operation.

 “The goals of the project are to raise awareness, encourage open communication and to ensure consistent reliability, functionality and performance across multiple platforms.”

Ahrens said.

This is what Justin Gibbs, president of the FreeBSD Foundation, had to say:

freebsd_foundation 100.x100This is a cross-platform effort to ensure the continued evolution of the ZFS file system. For developers and users of FreeBSD, the formation of OpenZFS clarifies the future of ZFS support for our platform.  The FreeBSD project is now an equal partner in defining the course for ZFS. OpenZFS combines the man power of the FreeBSD, Illumos, Linux, and MacOS communities to provide a level of test coverage, feature development, documentation, and support that wasn’t possible with our separate efforts.  Most importantly, OpenZFS will improve platform interoperability and reduce fragmentation of ZFS implementations. Today is an exciting day for ZFS and the FreeBSD platform.  I encourage you to browse http://www.open-zfs.org and to get involved. You are officially invited to help make the future of OpenZFS!

Matt Ahrens will co-present with Martin Matuska a presentation on OpenZFS at the upcoming EuroBSDCon 2013: OpenZFS:  Upcoming Features and Performance Enhancements with Illumos and FreeBSD joining Forces.

All in all, this is a very welcome development in the future of ZFS.

Links:

[1] With the exception of the illumos – FreeBSD co-operation.

BSD Conference: vBSDcon, October 25 – 27, 2013

A reminder for the upcoming vBSDcon 2013 has gone out today:

vBSDcon_banner

“Registrations for vBSDcon 2013 remain open until October 23rd, 2013. Register now for this BSD conference scheduled to take place at the Dulles Hyatt in Herndon, VA from October 25 – 27, 2013. Registrations are being accepted at www.vbsdcon.com.

Users and developers from across the BSD communities are encouraged to attend the event intended bring together members of the BSD community for a series of roundtable discussions, educational sessions, best practice conversations, and exclusive networking opportunities. Those interested in such an opportunity to learn, experience, and meet others involved in the BSD communities should plan to attend vBSDcon 2013.

vBSDcon is proud to bring prolific speakers such as:

  • David Chisnall, a member of FreeBSD’s Core Team which is charged with deciding FreeBSD’s overall goals and directions, speaking on the migration from GCC to LLVM/CLANG within the FreeBSD project.
  • Luigi Rizzo, FreeBSD source committer and project developer for netmap, speaking on the Evolution of the Netmaps Framework.
  • Baptiste Daroussin, FreeBSD source committer and project developer for PkgNG, speaking on the topic of PkgNG.
  • Henning Brauer & Reyk Floeter, OpenBSD developers, speaking on deep packet inspection with OpenBSD and PF.
  • Scott Long, FreeBSD source committer, speaking on “Disspelling the Stigma of the “Dot-oh” Release.
  • Devin Teske, FreeBSD source committer, with “A Comprehensive Look at bsdconfig”.
  • Kris Moore, PC-BSD Director of Development, speaking on automating deployment of FreeBSD and PC-BSD systems.
  • John Hixson, of iXsystems, speaking on the topic of FreeNAS plugins.

vBSDcon is being hosted at the Dulles Hyatt in Herndon, VA making it extremely convenient for attendees who book their room at the venue. The venue is also just minutes from Dulles International Airport with regular shuttles to/from the hotel and airport terminal during the day. Breakfast, lunch, and snacks will be provided on-site by the hotel’s on premise restaurant.

The schedule includes a reception dinner at the Dulles Hyatt on the evening of October 25th provided by Verisign and a mid-conference social the following evening celebrating 20 years of FreeBSD. Space for off hours hacker lounges and doc sprints will be available in the conference facilities with complimentary wireless internet access. The BSD Certification Group will also be hosting a BSDA certification exam on Saturday evening following the completion of conference activities for the day.

All are invited to take part in this event and are encouraged to register at the vBSDcon web site at www.vbsdcon.com. Simply click the “Register now” button to begin your registration! We look forward to meeting you all there!”

FreeBSD 10′s New Technologies and Features

freebsd_logo-100x100FreeBSD 10 has been in the works for a while. FreeBSD 9 became available on 12 January 2012 and now 20 months later, FreeBSD 10 is shaping up nicely, with two alpha releases available for testing.

According to the Release Schedule  FreeBSD 10 will receive the RELEASE status in November, but since the developers aim for quality of product over speed of release, this may slip into Dec 2013 / Jan 2014.

There has been a lot of maturing technologies in FreeBSD 10, with many new features which make this release, I think, the most exciting one in years. A lot of development has gone into virtualisation support. Virtualisation with FreeBSD Jails has been available for a long time, but not so much “full virtualisation”.

Let’s have a look at the some of the most talked about, most requested and most interesting features that have found their way into or are planned for “10.0″, but may not make the deadline. (More details and links to feature commits can be found FreeBSD 10 Wiki page).

General

pkgngpkgng is the new package management tool for FreeBSD

llvm-clangLLVM/Clang – FreeBSD 10.0 will deprecate the GPL-licensed GCC and switch to the BSD-licensed LLVM/Clang compiler (v3) by default. GCC is still in the source tree. It’s been switched off on platforms that LLVM supports fully (amd64, arm, armv6, i386), but can easily be built; just add WITH_GCC=YES to /etc/src.conf and ‘make -C/usr/src buildworld installworld’. (Thanks @DES for this clarification)

Unmapped VMIO buffers – The use of the unmapped buffers eliminate the need to perform TLB shootdown for mapping on the buffer creation and reuse, greatly reducing the amount of IPIs for shootdown on big-SMP machines and eliminating up to 25-30% of the system time on i/o intensive workloads

Libc++Libc++ has been integrated in FreeBSD 10. The libc++ library is focused upon C++11 support, is licensed under the MIT/UIUC license (rather than GPL) and will now be used instead of libstdc++

Variable symlinks – The support for variable symbolic links (varsym) supports automatic expansion of per-process, per-jail or system-wide variables in symbolic file links (may not make it in final release)

Kernel

Tickless kernel – FreeBSD 10.0 now supports a truly tickless kernel, enhancing battery performance on laptops and general resource effectiveness in virtual machines

freebsd_amdAMD Kernel Mode-Setting – AMD GPUs kernel mode setting supports the use of newer xf86-video-ati drivers and AMD GPUs

New iSCSI stack – The new iSCSI stack is kernel-mode and focused on reliability and interoperability

freebsd_intelRDRAND – Intel’s “Bull Mountain” RDRAND CPU instruction set on Ivy Bridge and Haswell CPUs for random number generator access will be supported in FreeBSD 10

KMS – A new X.Org stack with initial KMS support. Kernel mode-setting support in FreeBSD is still not at the level of support found on Linux for Intel, Radeon, and Nouveau hardware, but making good progress.

Startup

UEFI boot-loader support (may not make it in final release)

Animated boot splash support

Networking

BIND replacementUnbound and LNDS will replace BIND as the system’s DNS resolver. BIND will still be available from the ports.

PF firewall with SMP support. The PF firewall is now SMP-friendly. It supports fine-grain locking and better utilisation of CPUs on multi core machines. This means greater performance due to the multi-threading

NetMap – The NetMap framework for high-performance raw Ethernet packet access (paragraph updated – Thanks @Ed)

CARPCARP support has been overhauled and rewritten from the ground up

Wifi improvements – Improved 802.11n WiFi/WLAN wireless networking stack with support for new features and new drivers (e.g. Atheros PCI/PCIe 802.11n WiFi adapter from Qualcomm ,SMP/concurrency races, 802.11n TX aggregation)

Hardware Support

PCI hot-plug support (may not get included)

Apple Thunderbolt connectivity support (may not make it in final release)

USB Audio 2.0 – USB Audio support has been revamped, supporting new devices, higher bandwidth support and increased sampling frequency (may not make it in final release)

Architectures

FreeBSD/arm – Greatly improved support for the ARMv6 and ARMv7 platforms, incl SMP, thread-local storage (TLS) and superpages. Support for new ARM SoCs like the Texas Instruments OMAP4 and MV78x60 included

Raspberry Pi support.

Filesystems

ZFSZFS – FreeBSD 10.0 brings with it support for ZFS TRIM and it also supports LZ4 compression support which compresses much better (up to 50%) than the default LZJB compression. L2ARC compression support was also added, as well as NOP-write optimisation (ported from Illumos). All this will be great for using FreeBSD or FreeNAS as your NAS file server.

UFS live file-system resizing support. A UFS formatted filesystem can now be enlarged with growfs(8) when mounted in read-write mode. This should come in useful when adding extra storage to virtual machines without interrupting the service. Newly created UFS file systems will have faster fsck operation

FUSE – FreeBSD 10.0 brings in FUSE file-system support in user-space. FUSE file systems can now be accessed under FreeBSD without installing the “fusefs-kmod” kernel module from ports. This will make accessing these file systems (Linux) more stable.

Virtualisation

bhyvebhyvebhyve is the new native BSD Hypervisor and present in FreeBSD 10, developed from the ground up to offer a light-weight low-level HVM virtualization on FreeBSD.  bhyve supports the latest AMD and Intel virtualisation extensions, and is able to run the FreeBSD kernel fully virtualised. bhive also supports VirtIO for para-virtualization

Virtio – “virtio” is the name for the para-virtualisation interface developed for the Linux KVMVirtio drivers are developed alongside byhve. Any hypervisor supporting this interface should run without problems (Qemu/KVM, VirtualBox and BHyVe). The BSD-licensed implementation of the virtio kernel drivers support PCI, memory balloon, network IO, block and SCSI interfaces.

freebsd_xenXen – FreeBSD 10 comes with Xen DomU x86_64 support, Xen i386 PV support, plus many other Xen related items (USB PV drivers, SCSI pass-through, PowerPC/ARM/MIPS support, PV SMP support). Before you ask: No, there’s no Xen Dom0 support, yet

VPS – Virtual Private Systems for FreeBSD is an extension of the VIMAGE concept to the rest of the kernel (OS-level virtualisation, similar to jails), allowing live migration of VPSs from one host to another. This project is still in SVN and may not make it in final release)

All in all this is an impressive list of new features. This makes the release of FreeBSD definitely something to look forward to.

Which of the above or other features are you eagerly waiting for?

FreeBSD 9.2-RC4 now Available

freebsd_logo_textGlen Barber (@G_J_B_) has announced, on behalf of the FreeBSD Release Engineering team, the availability of the fourth release candidate (RC) builds of the 9.2-RELEASE release cycle.

Changes between -RC3 and -RC4 include:

  • Revert the tribute boot loader logo as the default logo.
  • Fix a filesystem bug that would cause removed files to fail to dereference vnodes until the file system was forcibly unmounted and remounted.
  • Fix a rtadvd(8) segmentation fault on service reload.
  • Create and correct ownership and permissions of /var/authpf in the standard mtree.
  • Fix a NFS deadlock.
  • Stop SIOCSIFADDR, SIOCSIFBRDADDR, SIOCSIFDSTADDR and SIOCSIFNETMASK at the socket layer rather than pass them on to the link layer without validation or credential checks.
  • Prevent cross-mount hardlinks between different nullfs mounts of the same underlying filesystem.
  • Fix the length calculation for the final block of a sendfile(2) transmission which could be tricked into rounding up to the nearest page size, leaking up to a page of kernel memory.

The RC4 downloads are now available on the FTP servers for the amd64, i386, ia64, powerpc, powerpc64, and sparc64 architectures.

 

 

BSDnow.tv Episode 002: Engineering and Powder Kegs

Allan Jude (@AllanJude) and Kris Moore (@pcbsdkris), founder of the PC-BSD project, have set up and are hosting a weekly video podcast at bsdnow.tv.

The show is dedicated to spreading the word about the BSD family of operating systems, and keeping those who are already aware up to date with the latest news and developments.

The second episode is now available. Allan and Kris discuss the following subjects:

  • 64bit time in OpenBSD
  • AESNI pipelining gets a speed boost
  • OpenBSD 5.4 Preorders
  • GCC no longer built by default on FreeBSD -CURRENT
  • Patch to update Xorg and MESA on FreeBSD
  • Interview – Glen Barber
  • Making your own binary repository
  • iXsystems hosts FreeBSD Anniversary party
  • NetBSD gets basic support for the cubieboard 1 & 2
  • Rayservers ditches Linux for BSD
  • HPN for OpenSSH 6.2
  • Call for testing: OpenSSH-6.3
  • pkgsrc gets signing support
  • FreeBSD vs. Linux: 10 points of superiority

FreeBSD Security Advisories (sendfile, ifioctl, nullfs)

software-bug-signThe FreeBSD Security Team has identified some issues in sendfile, ifioctl and nullfs, and has issued the following security advisories:

Please read the advisories and take the recommended action(s) if you’re system is affected.

Relating to security and cryptography is a post worth reading by Colin Percival, a former FreeBSD Security Officer, who wrote today about Trust/NSA/Tarsnap: Don’t trust me: I might be a spook.

BSD Magazine (September 2013): Day-to-Day BSD Administration

bsdmag-2013-09September’s issue of the BSD Magazine is now available: Day to Day BSD Administration (free PDF download).

This issue is dedicated to day-to-day BSD administration with the use of MidnightBSD custom installations and Live CDs, BSD server maintenance, directory encryption using PEFS and much more.

  • It’s lonely at the top
  • MidnightBSD Custom Installations and Live CDs
  • BSD Server Maintenance
  • Re-purposing an Abandoned Mac Mini as a Wireless Router with OpenBSD
  • Monit – Monitoring solution for enterprise and SOHO servers with FreeBSD
  • FreeBSD Programming Primer – Part 8
  • A closer look at the changes in PC-BSD/TrueOS 9.2 – Part 2 – Directory encryption using PEFS
  • Intro to ZFS
  • FreeBSD on XenServer

Download and read Day-to-Day BSD Administration

bsdnow.tv Podcast – Episode 1

bsdnow1Allan Jude and Kris Moore, founder of the PC-BSD project, have set up and are hosting a weekly video podcast at bsdnow.tv.

The show is dedicated to spreading the word about the BSD family of operating systems, and keeping those who are already aware up to date with the latest news and developments.

The first episode is now available. Allan and Kris discuss the following subjects:

  • Radeon KMS commited
  • VeriSign Embraces FreeBSD
  • fetch/libfetch get a makeover
  • FreeBSD Foundation Semi-Annual Newsletter
  • The place to B…SD
  • Interview – Peter Hessler
  • Using stunnel to hide your traffic from Deep Packet Inspection
  • NetBSD 6.1.1 released
  • Sudo Mastery
  • Documentation Infrastructure Enhancements
  • FreeBSD FIBs get new features
  • FreeNAS 9.1.0 and 9.1.1 released
  • BSD licensed “patch” becomes default