Posts

Adding ZVOL VIRTIO disks to a guest running on a host with the FreeBSD BHYVE hypervisor

On atlantis.domain (my FreeBSD BHYVE hypervisor host), I use these commands:

Install FreeBSD on a 10G disk.

vm create -s 10G guestname

vm install FreeBSD-12.1-RELEASE-amd64-memstick.img

vm console guestname

Add 8 30GB zvol-based disks

vm add -d disk -t zvol -s 30G guestname
vm add -d disk -t zvol -s 30G guestname
vm add -d disk -t zvol -s 30G guestname
vm add -d disk -t zvol -s 30G guestname
vm add -d disk -t zvol -s 30G guestname
vm add -d disk -t zvol -s 30G guestname
vm add -d disk -t zvol -s 30G guestname
vm add -d disk -t zvol -s 30G guestname

You will need to do

vm poweroff guestname
vm start guestname
because the guest does not seem to see the 8 30GB without power cycling. (rebooting won't work)

***

If you want the boot disk (the one create by "vm create") to be a zvol, you must edit the template:

root@atlantis:~ # zfs snapshot tank/vm-bhyve@2020-06-11-2145

root@atlantis:~ # mount -t zfs tank/vm-bhyve@2020-06-11-2145 /tank/vm-bhyve@2020-06-11-2145

root@atlantis:~ # vim /tank/vm-bhyv…

Changing the capacity of each VDEV in a ZPOOL without losing data and no downtime with ZFS

Before we start, let's define some acronyms.

FreeBSD: FreeBerkeley Software Distribution
ZFS: Zettabyte File System
ZPOOL: ZFS POOL
DEV: DEVice
VDEV: Virtual DEVice
GEOM: disk GEOMetry
BHYVE: BSD HYperVisor
ZVOL: ZFS VOLume

***

I have been tinkering with FreeBSD, ZFS, GEOM, BHYVE, and other interesting technologies.

***

Let's experiment on how to change the capacity of each VDEV of a ZPOOL without losing data and no downtime with ZFS.

To do so, you need a set of disks that are bigger.
Example, if the ZPOOL has 2 mirror VDEVs with 2 10GB disks each (total: 4 10GB disks), and you want to upgrade to 20 GB disks, than you need 4 20GB disks for the upgrade.


Figure 1: Live upgrade of disks in a ZPOOL

ZPOOL tank                           ZPOOL tank
    VDEV mirror-0                        VDEV mirror-0
        DEV /dev/vtbd1                    DEV/dev/vtbd9
        DEV /dev/vtbd2    --->         DEV/dev/vtbd10
    VDEV mirror-1                       VDEV mirror-1
        DEV /dev/vtb…

A new beginning in computational geometry

It has certainly been a while since the last post was published on this blog.

At the beginning of year 2019, I somehow started to reflect on my career, where I was, and where I wanted to go in my journey. At the end of 2019, I decided to look for new opportunities, after spending more than 4 years doing software development in bioinformatics at Gydle.

I was mainly looking for a new job where the computational aspect that I liked would still be a thing. I wanted also a sizable team, in order to be in an environment with social interactions. I wanted a workplace that values great software development tools and best practices. Finally, I wanted something that was not totally disconnected from what I did so far in life. That thing I did so far in life is the frontier between biology and computer science.

On 13 January 2020, I started to work for Bodycad. Bodycad aims at "The Pursuit of Orthopaedic Perfection". In a nut shell, Bodycad designs and manufactures personalized patient…

A list of interesting papers on optical mapping

At GYDLE inc., I work on optical mapping problems, and sometimes I find solutions to these problems. To tackle efficiently any problem, the error model for the problem must be understood. To write software that deals well with a given type of data, this type of data has to be understood, and one must be able to separate the good data from the bad data.

Once in a while, I read scientific papers to keep myself up to date with the optical mapping technologies. I created a Mendeley group where I maintain a list of papers on optical mapping. The list is available here. Some of these papers describe error models for optical mapping.

The Bioinformatics Adventure continues.

According to  the social network LinkedIn, I have been working on optical mapping problems at Gydle Inc. (with Philippe and Marc-Alexandre) for 8 months so far. I previously worked at Argonne National Laboratory (with Fangfang and Rick and other people).

The Bioinformatics Adventure continues. Yes, I am still doing bioinformatics. No, I no longer work on assemblers.

I have not worked on any genome assemblers for a while. As of now, I am more an aligner person than an assembler person (see "Sense from sequence reads: methods for alignment and assembly").

At Gydle Inc, we are 3 employees (the CEO, a computer scientist, and myself). Our CEO worked at Illumina at some point. Working in a small company is very fun. We get to do a lot of different things:

algorithm design,software design,software development,software quality control,test development, ticket management,data analysis,data curation.
I schedule my time around activities related to the development of optical mapping s…

The Spark Notebook from creator Kate Matsudaira

Image
I received my Spark Notebook. What is the Spark Notebook, you may ask.
The Spark Notebook combines form and function. The Spark Notebook project raised funding on Kickstarter too.

The way I see it, the Spark Notebook is an agenda (a 6-month agenda) with additional features. These features include (from the guide) the yearly planning pages, accomplishments, the monthly planning pages, the weekly planning pages, the inspiration pages, and the note pages. The monthly planning pages include something called the 30-day challenge. According to creator Kate Matsudaira, the 30-day challenge feature is useful to help start (or break) a habit.

The Spark Notebook comes with a guide. On this website, some of the text is white with a background image which makes it hard to read. For example, consider the screenshot below.







How did I learn about the existence of the Spark Notebook ?

One of my hobbies consists in watching videos on the Amazon Web Services (AWS) YouTube channel. I like these videos be…

The convergence of HPC and cloud computing

An exascale computer is an imaginary system that can sustain one exaflops (10^18 floating point operations per second.) Such an object is needed in science and engineering, mostly for simulating virtual versions of objects found in the real world, such as proteins, planes, and cities. Important requirements for such a computer are 1) memory bandwidth, 2) floating point operation throughput, 3) low network latency, and so on.

2 of the many challenges for possibly having exascale supercomputers by 2020 are 1) improving fault-tolerance and 2) lowering energy consumption. (see "No Exascale for You!" An Interview with Berkeley Lab's Horst Simon).

One typical solution to implement fault tolerance in HPC is the use of the checkpoint/restart cycle whereas in most cloud technologies fault tolerance is instead implemented using different principles/abstractions such as load balancing and replication (see the CAP theorem). The checkpoint/restart can not work at the exa scale because …