|Author:||Tom L Nguyen email@example.com 11/03/2004|
|Copyright:||© 2004 Intel Corporation|
2.1. About this guide¶
This guide describes the basics of the PCI Express Port Bus driverand provides information on how to enable the service drivers toregister/unregister with the PCI Express Port Bus Driver.
2.2. What is the PCI Express Port Bus Driver¶
A PCI Express Port is a logical PCI-PCI Bridge structure. Thereare two types of PCI Express Port: the Root Port and the SwitchPort. The Root Port originates a PCI Express link from a PCI ExpressRoot Complex and the Switch Port connects PCI Express links tointernal logical PCI buses. The Switch Port, which has its secondarybus representing the switch’s internal routing logic, is called theswitch’s Upstream Port. The switch’s Downstream Port is bridging fromswitch’s internal routing bus to a bus representing the downstreamPCI Express link from the PCI Express Switch.
A PCI Express Port can provide up to four distinct functions,referred to in this document as services, depending on its port type.PCI Express Port’s services include native hotplug support (HP),power management event support (PME), advanced error reportingsupport (AER), and virtual channel support (VC). These services maybe handled by a single complex driver or be individually distributedand handled by corresponding service drivers.
2.3. Why use the PCI Express Port Bus Driver?¶
In existing Linux kernels, the Linux Device Driver Model allows aphysical device to be handled by only a single driver. The PCIExpress Port is a PCI-PCI Bridge device with multiple distinctservices. To maintain a clean and simple solution each servicemay have its own software service driver. In this case severalservice drivers will compete for a single PCI-PCI Bridge device.For example, if the PCI Express Root Port native hotplug servicedriver is loaded first, it claims a PCI-PCI Bridge Root Port. Thekernel therefore does not load other service drivers for that RootPort. In other words, it is impossible to have multiple servicedrivers load and run on a PCI-PCI Bridge device simultaneouslyusing the current driver model.
To enable multiple service drivers running simultaneously requireshaving a PCI Express Port Bus driver, which manages all populatedPCI Express Ports and distributes all provided service requeststo the corresponding service drivers as required. Some keyadvantages of using the PCI Express Port Bus driver are listed below:
- Allow multiple service drivers to run simultaneously ona PCI-PCI Bridge Port device.
- Allow service drivers implemented in an independentstaged approach.
- Allow one service driver to run on multiple PCI-PCI BridgePort devices.
- Manage and distribute resources of a PCI-PCI Bridge Portdevice to requested service drivers.
2.4. Configuring the PCI Express Port Bus Driver vs. Service Drivers¶
2.4.1. Including the PCI Express Port Bus Driver Support into the Kernel¶
Including the PCI Express Port Bus driver depends on whether the PCIExpress support is included in the kernel config. The kernel willautomatically include the PCI Express Port Bus driver as a kerneldriver when the PCI Express support is enabled in the kernel.
2.4.2. Enabling Service Driver Support¶
PCI device drivers are implemented based on Linux Device Driver Model.All service drivers are PCI device drivers. As discussed above, it isimpossible to load any service driver once the kernel has loaded thePCI Express Port Bus Driver. To meet the PCI Express Port Bus DriverModel requires some minimal changes on existing service drivers thatimposes no impact on the functionality of existing service drivers.
A service driver is required to use the two APIs shown below toregister its service with the PCI Express Port Bus driver (seesection 5.2.1 & 5.2.2). It is important that a service driverinitializes the pcie_port_service_driver data structure, included inheader file /include/linux/pcieport_if.h, before calling these APIs.Failure to do so will result an identity mismatch, which preventsthe PCI Express Port Bus driver from loading a service driver.
This API replaces the Linux Driver Model’s pci_register_driver API. Aservice driver should always calls pcie_port_service_register atmodule init. Note that after service driver being loaded, callssuch as pci_enable_device(dev) and pci_set_master(dev) are no longernecessary since these calls are executed by the PCI Port Bus driver.
pcie_port_service_unregister replaces the Linux Driver Model’spci_unregister_driver. It’s always called by service driver when amodule exits.
220.127.116.11. Sample Code¶
Below is sample service driver code to initialize the port servicedriver data structure.
Below is a sample code for registering/unregistering a servicedriver.
2.5. Possible Resource Conflicts¶
Since all service drivers of a PCI-PCI Bridge Port device areallowed to run simultaneously, below lists a few of possible resourceconflicts with proposed solutions.
2.5.1. MSI and MSI-X Vector Resource¶
Once MSI or MSI-X interrupts are enabled on a device, it stays in thismode until they are disabled again. Since service drivers of the samePCI-PCI Bridge port share the same physical device, if an individualservice driver enables or disables MSI/MSI-X mode it may resultunpredictable behavior.
To avoid this situation all service drivers are not permitted toswitch interrupt mode on its device. The PCI Express Port Bus driveris responsible for determining the interrupt mode and this should betransparent to service drivers. Service drivers need to know onlythe vector IRQ assigned to the field irq of struct pcie_device, whichis passed in when the PCI Express Port Bus driver probes each servicedriver. Service drivers should use (struct pcie_device*)dev->irq tocall request_irq/free_irq. In addition, the interrupt mode is storedin the field interrupt_mode of struct pcie_device.
2.5.2. PCI Memory/IO Mapped Regions¶
Service drivers for PCI Express Power Management (PME), AdvancedError Reporting (AER), Hot-Plug (HP) and Virtual Channel (VC) accessPCI configuration space on the PCI Express port. In all cases theregisters accessed are independent of each other. This patch assumesthat all service drivers will be well behaved and not overwriteother service driver’s configuration settings.
2.5.3. PCI Config Registers¶
Each service driver runs its PCI config operations on its owncapability structure except the PCI Express capability structure, inwhich Root Control register and Device Control register are sharedbetween PME and AER. This patch assumes that all service driverswill be well behaved and not overwrite other service driver’sconfiguration settings.
by Alessandro Rubini
This article is meant to show the internal structure of devicedrivers for serial ports, and how they can be perform a variety ofservices including ppp and slip. The discussion isbased on 2.4 source code, but most of the material applies equallywell to 2.2 and 2.0.
The usual view of a serial port
When discussing the software implementation of a serial port, thefirst thing that comes to mind is
/dev/ttyS0, as this isthe most known character of serial communication, at least on PC-classcomputers. Since
/dev/ttyS0 is a file special file oftype ``char', a serial driver is often considered a conventional chardriver, and phrases like ``char drivers are exemplified by serialports' come to mind. Unfortunately, the exemplification is basicallywrong.
If you look in the real code, you'll see how the ``char driver'idea only scratches the surface of what a serial driver is, and that'swhy a serial driver doesn't quite lend itself to be the prototypicalexample of char drivers.
Actually, the ``char driver' abstraction doesn't correctlydescribe serial device drivers because there is not specific majornumber associated to each of them. Actually, you can have add-onserial ports that plug in your computer and are managed by specifickernel modules but do not get assigned a new major number.
/proc/devices you'll find that major 4 isassociated to the
ttyS driver, but that's a white lie:text-mode console devices belong to major 4 too, and, actually,Linux-2.0 used the more general ``
ttyp' name for majornumber 4.
What makes a serial port different from a more conventional chardriver like the printer port or a tape drive is its being part of thetty abstraction. Since a serial communication channel can be used toplug an alphanumeric terminal, a serial device driver must beintegrated in the terminal emulator layer, called the ``tty'abstraction, from the name of ancient tele-type devices (still in wideuse when Unix was being written)
Overview of tty management
Flexible and powerful tty handling is made up of several buildingblocks. You must consider that there exist a huge range of ttydevices, from VGA and frame-buffer based text consoles up to serialcommunication channels and virtual terminals as exemplified by (butnot limited to) the
Figure 1 shows the various building blocks that are involved withoperation of a serial driver. Most files live in drivers/char,if not, the directory specified is relative to the root directory ofthe kernel source tree. The figure shows how each building block isregistered (registration is there in order to allow each block to beimplemented as a kernel module).
The image is available as PostScript here
The file fs/devices.c exports the interface used by mostsystem resources to register device drivers, each identified by amajor number). This is how
tty_register_driver gets holdof a major number if it needs it to support the new tty driver (anobject that is introduced below). The function is defined bytty_io.c, which also defines the file operations that are usedto act on tty devices. File operation are the driver-specificimplementations of read, write and the other systemcalls that relate to file access.
What serial.c or other serial drivers do in order to runtheir own code is registering a
struct tty_driver object.The ``driver' declares the major number and the range of minornumbers it is going to manage, as well as a number of operations itsupports. The operations are concerned with input and output of dataas well as flow control and communication with higher layers. Theimplementation of these operations, together with interrupt handlingand actual input/output of data is the scope of hardwaredrivers for serial ports.
The data flow between user space and the serial device driver,therefore, is mediated by the tty layer, that implementsfunctionalities that are common to all tty-type devices. However, notall of tty management is defined be tty_io.c, most of the policyis define by the line discipline, a software module thatdisciplines how a physical tty I/O line is used. The default linediscipline for Linux is called
N_TTY, a name that will beexplained later. If n_tty is active, input data reaches userspace via the usual /dev/ interface and the standard terminalI/O handling (i.e, all the features defined by
<termios.h>, that make terminal handling sopowerful and so difficult).
The red line in figure 1 shows the logical flow of data, from thehardware channel up to the user-accessible device special file andbackwards. What keeps it all together is the
struct ttydata item, that in itself includes a pointer to all three relevantobjects: the
file_operations structure that is used incommunicating with user space, the
tty_driver structurethat hosts functions to control real hardware, and the
tty_ldisc structure that lists all entry points to thecurrent line discipline.
Why so complex?
Sure this kind of arrangement may look exceedinglycomplex. However, as usual, the extra complexity is meant to makethings ultimately more flexible and powerful. Adding support for newserial hardware may be different from writing a conventional chardriver, but this setup guarantees full tty emulation on all serialports without any code replication or unneeded complexity in thelow-level driver.
Another advantage of this kind of arrangement, possibly even moreimportant than generalized tty support, is in the ability to changethe line discipline associated to each tty device. Unlike typicaldevice drivers, whose task is connecting an hardware device with userspace, a serial driver has nothing to do with user space; data itreceives from the hardware is passed over to the line discipline, anddata it receives from software comes via a line discipline method.
A serial driver, therefore, is not concerned in any way with datatransfer to/from user space. The task is left to the line discipline,together with all the hairy
termios handling. This makesit possible for serial data to be steered to a different user-spaceaccess facility than its associated ttyS device special file.
PPP and slip
Whenever you dial your modem with ppp to connect to theInternet, or use the simpler slip communication protocol toconnect the PC to your Linux palm-top, you are exploiting thecomplexity just shown. Both ppp and slip implement theirown line discipline; when either of them is run, the tty device isswitched to a different line discipline in order to detach/dev/ttyS0 from the serial port and keep all of serialcommunication within the kernel.
Figure 2 shows the conceptual layout in the slip case. Ichose not to use ppp in the example to avoid extra complexityor incorrect simplification. In version 2.4 of the Linux kernel theppp software implementation is split in several files (onceagain, it's more structured to be more powerful and avoid codereplication across similar devices). The two protocols behaveotherwise in the same manner.
The image is available as PostScript here
The role of the slip driver, as shown, is registering both anetwork device (depicted as
slip0 and a line disciplinefor tty devices (called
N_SLIP). When the tty device isswitched to the new line discipline, TCP/IP communication can begin.The new line discipline sets up data transfer between serial hardwareand the network layer; when it is active, nothing can be read from orwritten to the associated /dev/ttyS device. As soon as thedevice (
/dev/ttyS0 or equivalent) is closed, the defaultline discipline is restored. Actually, that's the main reason whyneither slattach nor pppd can exit after setting up thenetwork channel.
The individual data structures
There are three main data structures involved in tty management(and thus, serial communication):
struct tty_struct: this is the data structure that stays at the core of tty management. It includes both of the following structures. An instance of
tty_structis created any time a new tty device is opened, and exists until it is last closed. Note that actual code (in
tty_io.c) is complicated by the need to preserve
termiossettings across close and open, at least for some of the ttys (like serial ports).
struct tty_driver: this is the low level hardware handling. At open time, the function
get_tty_driverretrieves the driver for the current tty an places it into the
tty_struct, where it is further accessed.
struct tty_ldisc: the structure is referenced by the
tty_struct. At open time the field is initialized to reference n_tty, and user programs can change the current line discipline via ioctl, as explained in a while.
The structures are declared in three different files:
tty_struct is a complex structure defined ininclude/linux/tty.h, a header generally devoted to ttyissues. Actually, it is not as interesting to look at as the othertwo, because user modules rarely need to directly interact with it.
include/linux/tty_driver.h andinclude/linux/tty_ldisc.h are devoted exclusively to thedefinition of the relevant data structures. The files carry aprominent comment block that explains the exact role of most of thefields. Unlike
tty_ldisc are actively used by authors of devicedriver modules.
Typically, a kernel module that supports a new kind of hardwaretransmission will implement a
tty_driver structure, whilea module that uses generic serial hardware for a new purpose willimplement a line discipline. For example, if you have a specialkeyboard that transmits data via a standard RS-232 serial port, you'll need aline discipline that gathers data packets and send them to eitherthe input mechanism (see drivers/input/input.c andinclude/linux/input.h) or to the generic keyboard driver (using
handle_scancode(), exported bydrivers/char/keyboard.c).
Listing 1 shows the most important operations declared by the
tty_driver data structure, while listing 2 depicts thoseexported by the line discipline. Note that those listings are byno means complete, if you look for authoritative information, you should read the relevant header files.
Data flow in reading and writing
The image is available as PostScript here
Figure 3 visualizes how data flows from user space down to hardwareinterfaces and backwards. It refers to the specific case of a standardPC serial port with the default line discipline attached. The logicalstacking of line disciplines (near to user space) and tty drivers(near to the hardware itself) should be apparent.
While writing data is straightforward, the reading process may needsome explanation. Reading is more complex than writing because there'sno direct causal relationship between hardware (that pushes up datawhen it arrives) and user space (that requests data when it needs it).The solution is, as you expected, use of buffering: data received byhardware remains on hold in a kernel buffer until a user space programrequests it; whenever a user program asks for data and the buffer isempty, the program is put to sleep, and is awaken only when the bufferis filled with at least partially. Note that a write buffer exists aswell, however, the write implementation is much morestraightforward because each step towards hardware level is directlydriven by the step above it. There are no uncontrolled delays in datatransmission, and the buffer is only needed to decouple hardwaretransmission from program flow. The figure does not show it forsimplicity.
When tty devices are concerned, the read buffer lives within thetty data structure; while this makes
struct tty_structconsiderably bigger, there is no point in keeping the bufferelsewhere: each tty can't transfer data without a buffer, and ttydevices are dynamically allocated so no memory is wasted in buffersfor unused devices.
Actually, tty-related buffering is organized in two levels: kerneldevelopers chose to provide both a ``conventional' buffer, where datais waiting to be eaten by the line discipline (i.e., in the defaultcase, being transferred to user space), and a ``flip' buffer, used byhardware routines to store incoming data as quickly as possible,without the need to synchronize for concurrent access: flip buffersare exclusive ownership of the hardware device, which eventually callstty_flip_buffer_push to deliver data to the tty buffer, wherethe line discipline pulls it from.
It's interesting to note that the flip buffer is laid out as twophysical buffers that are alternatively written to. This allows morereliable operation, as the interrupt handler will always have a wholebuffer available for writing. The function flush_to_ldisc,called by the low-level driver and part of the tty layer (i.e.,tty_io.c), arranges for the flip buffer to be flipped, beforethe interrupt handler returns. This layout, by the way, is whythe flip buffer is called so.
How to use a custom line discipline
Kernel code, as stated, can register a new line discipline with thetty subsystem, and this is also available to modularized code. Youcould, therefore, write your own line discipline and register it.Each line discipline is identified by a number, and a symbolicname for it is available, as common with C programming. Assignednumbers are given a name by include/asm/termios.h.
The default tty line discipline is identified by a number of
N_TTY, PPP uses
N_PPP and soon. Unfortunately, no line discipline numbers have currently beenreserved for ``local use', so you can only experiment with thenumbers that are not used on your system. Actually, no official drivercurrently used
N_MOUSE, so this is a good bet for yourcustom line discipline.
In order to activate the
N_MOUSE line discipline, theuser space program must use this code:
The role of register_serial
If you noted that drivers/char/serial.c exports a functioncalled register_serial, you may wonder what's its role in thetty architecture just outlined. As a matter of facts, the facility isonly an hook offered by the ``standard' serial tty driver in order toeasy run-time addition of standard serial ports. The ``serial' beingregistered is not a whole software module, but rather, only adefinition or parameters to use for the new serial port. Theparameters are described by
struct serial_struct, whichin turn is defined by include/linux/serial.h; they are used bythe conventional serial driver, exploiting the de-factostandardization that exists on PC serial ports. You can't use thefunction to register a driver for serial hardware of a different kindthan a 16450 or compatible UART. The list of supported hardware forthe PC platform is found in the array
uart_config, inserial.c. Other platforms offer different implementations for
Impact Port Devices Driver License Test
The serial console
Serial ports can also be used as console devices, and this kind offunctionality is separate from tty management. Actually, the consoledevice sits in the lowest levels of Linux, as it must bring criticalinformation out of the system as soon as possible. It just can't beinvolved in all the complexity of tty management. But in version 2.4(as well as 2.2) console management is a world of its own, with itsown data structures and functions that make it very flexible yetreliable. Discussion of such world is best delayed to next month'scolumn.
Verbatim copying and distribution of this entire article ispermitted in any medium, provided this notice is preservedAlessandro is an independent consultant based in Italy. Eventhough he claims to be a programmer, his activity consists mostly inadvocating libre software and documenting what real programmers do.He runs the GNU operating system on his computers, and can thus bereached as