Block Device Interface Change And Related Pain

Alexander Viro announced that the block device interface would be changing, and that some of these changes had made it into 2.3.38; he listed:

1. New type (struct block_device) is defined. We have a cache of such objects, indexed by dev_t. struct block_device * is going to replace kdev_t for block devices. Handling of the cache is done in fs/block_dev.c
2. They have methods (struct block_device_operations). Currently the set is { open, release, ioctl, revalidate, check_media_change }. For now (and it's going to change) types are the same as in file_operations. However, in the near future they are going to become

   int (*open)(struct block_device *bdev, mode_t mode, unsigned flags);
int (*release)(struct block_device *bdev);
int (*ioctl)(struct block_device *bdev, unsigned cmd, unsigned long arg);
int (*revalidate)(struct block_device *bdev);
int (*check_media_change)(struct block_device *bdev);

3. ->revalidate() and ->check_media_change() disappeared from file_operations.
4. register_blkdev() takes block_device_operations instead of file_operations now. For one thing, it means that block devices are more or less insulated from all future changes in file_operations (Good Thing(tm)). For another, it means that drivers should be modified. I did the change for all drivers in the main tree, see the patch for details. It's pretty easy.
5. blkdev_open() doesn't change ->f_op. def_blk_fops has all needed methods (open, release and ioctl call the methods from block_device_operations, indeed).
6. Inodes got a new field: i_bdev. Filesystems should not worry about it - just remember to call init_special_inode() when you are initializing device/fifo/socket in-core inode (in foo_read_inode() or in foo_mknod(); all filesystems in the tree are doing it now). Contents of this field: pointer to struct block_device if it is a block device inode, NULL otherwise.
7. Superblocks got a new field: s_bdev. Handled by code in fs/super.c, points to the struct block_device if the mount is device-backed, NULL otherwise (i.e. for NFS, CODA, procfs, etc.).
8. do_mount() first argument is struct block_device * now. It does the right thing for non-device mounts - just pass NULL and it will work (allocate the anonymous device, etc.)
9. Instead of calling get_blkfops(), use ->bd_op in struct block_device. Moreover, better use blkdev_get()/blkdev_put()/ioctl_by_bdev() (see examples in mm/swapfile.c, drivers/char/raw.c, fs/super.c, fs/isofs/inode.c, fs/udf/lowlevel.c).
10. Thing that is probably going to happen RSN: instead of struct gendisk per major we may want to go for struct gendisk per _disk_. It would mean that at some point near ->open() we will put the pointer to it into the struct block_device. One obvious consequence being that partitions-related ioctls() will become completely generic.

Notice that it is _not_ the same as devfs (and not a beginning of moving devfs into the main tree). It just provides the backplane - no namespace, no nothing. Inodes (either in normal filesystems or in devfs) point to such animals. That's it. Eventually things like ->b_dev, ->b_rdev, ->i_dev, ->rq_dev, etc. are going to become pointers to such objects, but it will be done step-by-step - otherwise we'll end up with a moby patch and moby breakage in bargain...

Character devices are not affected at all - IMO using the same type both for block and character device was a mistake. So their handling remains as-is. Probably something should be done for them too, but that's completely different story.

Richard B. Johnson picked himself up off the floor and said:

Good grief Charley Brown! You, in a few key-strokes, just blew away major portions of the work done over the past few years by software engineers who ported their drivers to Linux. Linux will never be accepted as a 'professional' operating system if this continues.

It's enough of a problem putting one's job on-the-line convincing management to risk new product development to Linux. Once these products are in Production, and bugs are discovered in the OS, we must be able to get the latest version of the OS and have our drivers compile. If this is not possible, you do not have an operating system that is anything other than an interesting experiment.

For instance, there was a simple new change in the type of an object passed to poll and friends. This just cost me two weeks of unpaid work! Unpaid because I had to hide it. If anyone in Production Engineering had learned about this, the stuff would have been thrown out, the MicroCreeps would have settled in with "I told you so..", and at least three of us would have lost our jobs.

Industry is at war. You can't do this stuff to the only weapons we have. Once you claim to have a "Professional Operating System", its development must be handled in a professional way. If major kernel interface components continue to change, Linux is in a heap of trouble as are most all of those who are trying to incorporate it into new designs.

The industrial use of Linux is not at the desktop. It involves writing drivers for obscure things like machine controllers (read telescope controllers), Digital signal processors (read medical imaging processors), and other stuff you can't buy at the computer store. It doesn't matter if you fix all of Donald Becker's drivers to interface with the new kernel internals. You have still broken most everything that counts.

There were a number of replies to this. Alexander found Richard's post clueless and Monty-Pythonesque. On a serious (though annoyed) note, he explained, "one of the worst things about block drivers-to-kernel interface is that they share it with files. I.e. _any_ change in file_operations or in struct file or in struct inode and you are deep in it. Change the size of any field prior to ->i_dev and you are in for recompile. Change <gasp> device number bitness and even recompile may be of little help. Removing those dependencies (not all of them are removed yet, more will follow) is going to save _your_ ass a year later."

Also replying to Richard, Victor Khimenko said, "Drivers MUST be changed with new kernel release (and thus via development branch: development kernels are just snapshots of development process after all). It was true from the start and it'll be true tomorrow. It's true for most OSes available. It's ESPECIALLY true for Linux where drivers are linked directly in kernel. If you expected something other then you made wrong choice choosing Linux."

Gregory Maxwell said to Richard:

We all know your position on compability. :) Many people, including myself, usually understand and agree with it.

However, you are going a little far on this one.

The change is going into 2.3.x, and that *IS* the approiate place to break interfaces. These kinds of changes should certantly not be introduced into 2.2.x.

This should cause you little difficulity, as your example of having to upgrade to fix a bug should not apply. When you upgade to fix a bug then you should just be increasing patchlevel. If there is not a patch for a bug in 2.2.x which is fixed in 2.4.x then there is a bug in the Linux development process.

In order to move forward, we *must* break things. To make up for this we continue to maintain old versions. There are still bugfixes being made against 2.0.x and there will be bugfixes against 2.2.x. RedHat even still issues updates against RH4.2..

So if this were to have occured within a stable kernel version, or if it had severly affected userspace, I would agree.

Rik van Riel put it this way to Richard:

Industrial use of Linux usually doesn't involve the kernels which are marked as `development', ie. where the `middle' version number is odd and where major things are expected to change.

People venturing out on that terrain can know what they're heading into (see http://kt.linuxcare.com/) and shouldn't come whining when some actual development happens in the development branch of the kernel. The should only whine when development stops, not when useful changes are taking place...

But David Parsons objected to Rik, "Except, of course, that when the changes go in they are never backed out so the interfaces remain stable for the production kernels. That's the *really* annoying thing about this line of argument; when else should someone complain that an interface has been turned into gravel? If you wait until the development tree has become a production tree, enough code will be modified to work with the New! And! Improved! interfaces that your complaints (cf: old-style fcntl locking) will be dismissed sight unseen by the Core Team." He added, "The big support providers are the ones who benefit from interface churning. It's the small shops that get bitten in the ass because they don't have enough money to buy programmers or enough time to do the patches." There was no reply to this.

Alan also replied to Richard with the quote of the day, saying, "Linux isnt at war. War involves large numbers of people making losing decisions that harm each other in a vain attempt to lose last. Linux is about winning."

At some point, Richard posted again, having received many private emails in addition to the slew on the list. He said:

I have gotten a lot of mail on this so I will reply only once.

Many of the professional industrial uses of Unix were previously covered using Sun boards, boxes and SunOs. If you ever dial 10 before a long-distance number to get a cheaper rate, that's voice over IP and we make that stuff. This was developed on Suns, runs on them, but will soon be running on cheap Intel clones.

If you ever have to go to the hospital and have a CAT-Scan or a MRI, you are using equipment developed by us, even though the name on the box may be Phillips, General Electric, Toshiba, or various other companies. You can look http://www.analogic.com and see what we do for a living here.

The Sun driver interface has been constant. Unfortunately, you have to install it, meaning link it and reboot. When Installing a system, meaning the complete software package, the end-user's technician installs the OS from a CDROM. Then the application with its drivers are installed from another CDROM. This works on Suns and has been the De-facto standard way of doing things.

Linux was not suitable for the applications running on Suns until Linux provided the installable device driver. The ability to install a hardware-interface module into a kernel was my main selling point for using Linux to replace SunOs, and, indeed the whole Sun architecture.

Incidentally, the cost is the same. A CDROM for Solaris is essentially the same cost as a CDROM for Linux. Once you start distributing an operating system and supporting the distributors, a "free" operating system is no longer free.

By the time a decision was made to produce our new Exact Baggage Scanner, marketed by Lockheed-Martin, engineering management was dragging its feet on the use of Linux. They wanted something that was "everything to everybody", but didn't want the cost of using Suns. Further, it had to be completely under company control.

I was unable to convince anybody to use Linux so I had to write my own Operating System. It is called ARTOS (Analogic Realtime Operating System). Our Sky Computer Division, which produces the world's fastest (still) digital signal processor, made the high-speed stuff, a lowly Intel Pentium with my OS is used as the system controller, and an Alpha Workstation is used for the user interface.

When this was completed, we went on to producing our third generation CAT Scanner. This uses a Pentium as the main system controller and Linux as the operating system. The User Interface uses Windows-NT. It was felt that Linux was sufficiently well-hidden in the bowels of the machine so nobody would care.

The drivers in this machine comprise both block and character devices. One of major building blocks is the driver that interfaces to the Digital Signal Processor. This DSP board comprises up to 32 TMS-320C20 DSPs plus an i960 for interface. It is made by our CDA Division.

Completed data, available within a 32k window, a 512x512x16bit chunk, must be transferred to the User Interface within 1/4 second to make the specification. It does.

Now, our legal department has defined the criteria we must meet to use Linux. They presume that we will provide a "current distribution" of Linux to every end-user. They also defined that, since drivers may be deemed to modify the operating system, we have to provide driver source-code to the customer if they request it. Application code continues to be proprietary.

Changing the kernel interface to drivers is counter productive. In fact it makes the usual field installation impossible. The usual installation would automatically and transparently compile the interface modules, using the new Operating System. This is no longer possible because the compilation will fail.

Again, if Linux is to become other than an interesting experiment, one cannot change these interfaces without understanding the whole picture.

Distributors don't care. The more changes there are, the greater the obsolescence, the more money they make selling new boxes of CDROMs. Therefore there is no controlling negative feedback to be obtained from the distribution channel. You can reject what I say out-of-hand, and continue as an experiment, or you can listen and make a significant contribution to providing jobs worldwide.

It is, of course, possible to fragment Linux. A company could be started, called StableLinux that distributes only Linux n.n.n and performs bug-fixes and maintenance on that version only. This is not helpful to the greater Linux community. Instead, we need to minimize the changes that affect the interfaces to world-wide applications. Just as POSIX attempted to stabilize the API so that one could write "portable" code, the interface to hardware that hasn't even been invented yet has to be stable.

Chris Adams and Horst von Brand suggested that "current distribution" refered to even-numbered minor version numbers only. Horst expanded, "OK, "current distribution" means 2.2.x kernel today, and was 2.0 sometime back. It will be 2.4 in a few months time, and perhaps 2.6 in a year and a half. You are supposed to distribute the machine and source to drivers &c _when shipped_, I'd assume. Check the code, test it to breaking *and keep it*. Ship that to customers, and either offer upgrades to 2.4 if needed for some reason, or stay put."

Elsewhere, replying to Richard's original post, Jamie Lokier said, "If you need a stable API, you chose the wrong operating system. It's no secret that Linux APIs change. You can't blame the kernel developers for doing exactly what they said they will do. If you want, you can blame the people who incorrectly assumed the APIs would stay the same, for not investigating the obvious." And Ted added, "If you told your management that Linux kernel interfaces never change across versions, then you were sadly mistaken. However, the mistake is on your end, I'm afraid."

To this, Richard replied:

No. According to our Legal Department, to satisfy the GPL requirement that we provide source to the end-user, they required that we supply a "current" distribution of Linux if the end-user requests it.

This seemed, by them, to be an easy solution to possible problems. Unfortunately, for Engineering, this means that we have to keep everything "current" during development so that, by the time equipment is shipped, it will run with the "current" distribution (whatever this is).

The obvious solution, given these constraints, is that we just ignore all changes until shipping time, then attempt to compile with the latest distribution, fixing all the problems at once. However, we then end up shipping untested software which ends up being another problem. Checking to see if it "runs" isn't testing software in the cold cruel world of industry.

So, presently, I have 13 drivers I have to keep "current". Yesterday they all got broken again. A week before, half of them were broken because somebody didn't like a variable name!

That said, a major problem with changes that I see, is that the changes are made without the notion of a terminating condition. For instance, new parameters are being passed to existing interface functions.

If you are going to break an interface, you should plan on only breaking it once rather than opening the door for more changes and leaving it open. For instance, once you have to pass more than (depends upon the machine) about 3 parameters, it's best to put them all in a parameter- list (structure) and pass only the address of the parameter list (pointer).

From that time on, you only have to add structure members to the parameter list if you have to add changes. If I had seen these kinds of changes I would not have complained. It means I have to rework stuff only once.

So `read(f,.......)` should have been changed to `read(params *)` and you are done with it forever as long as you don't change structure member names and functions for kicks.

This time it was Alexander's turn to pick himself up off the floor; and in response to the first paragraph of Richard's post, said, "Oh. My. God. They are requiring you to do WHAT??? Do you mean that you really ship 2.3.x to your customers? Arrggh. "Source" == "source of what we are shipping". And not "anything that was written by other guys who started from the same source". It's utter nonsense. _No_ license can oblige you to include the modifications done by somebody else. Otherwise you'ld have those drivers in the main tree, BTW - _that_ much should be clear even for your LD." But David Lang put in, "he is not saying that he has to ship a 2.3 kernel, he is reacting to the fact that he will have to ship a 2.4 kernel. the blame for this lies squarly on the legal department who decided that they had to ship a "current" disto. There is some semblance of reason for this as they want to try and limit the support costs by not using "obsolete" versions, but given the way many of the major distros patch the kernel before shipping it you still may have problems. The answer is to figure out some way to educate the legal department to allow for a more gradual change."

Thu Jul 24 11:01:40 PDT 2008		linux/breakage.src
Updated: Thu Aug 18 2005 9:31.42		Viewed: never