Introduction Chapter 4: StorageBrent Rector December 2003 ContentsWhat Is
WinFS? In some ways, personal computer is an inadequate name. Most people don't use a personal computer to compute. They use a computer to communicate (through e-mail or instant messaging) and to store and organize their personal data (such as e-mail, documents, pictures, and digital music). Unfortunately, while your computer presently stores this data quite well, it does a relatively poor job of allowing you to organize the information so that you can find it later. Disk capacity has been growing at roughly 70 percent annually over the last decade. It's presently possible to buy drives with more than 250 gigabytes (GB) of storage. It's likely that 500-GB drives will become available in the next few years and that many systems will have more than one disk drive. I just did a quick check on the computer on which I'm writing this chapter, and I have 283,667 files in 114,129 folders in only 200 GB of disk space. When I forget exactly where I put a file, it can take quite a while to find it again. In the worst case, I have to search the entire contents of each disk. In a few years, people will be able to store millions of files, most of which, if nothing improves, they'll never see again. One reason people have difficulty finding information on their computer is because of the limited ability for the user to organize data. The present file system support for folders and files worked well originally because it was a familiar paradigm to most people and the number of files was relatively small. However, it doesn't easily allow you to store an image of your coworker Bob playing softball at the 2007 company picnic at a local park and later find the image when searching for documents that:
The hierarchical folder structure doesn't work well when you want to categorize data in numerous ways. Therefore, we have a problem today in that we have lots of stuff to store and no good way to categorize it. In addition to categorizing information, which many people associate with attaching a fixed set of keywords to data, people need to relate data. For example, I might want to relate a picture to the company picnic, or I might want to relate a picture to Bob, who is also a member of an organization to which I donate time and effort, as a contact. Another problem is that we store the same stuff in multiple places in multiple formats. Developers spend much time and effort creating their own, unique storage abstractions for everyday information such as People, Places, Times, and Events. For example, Microsoft Outlook has a definition of a Contact. The Microsoft Windows Address Book also has its own definition of a contact. Each instant messaging application has yet another. Each application stores its definition of a contact in a unique, isolated silo of information. There are a number of problems with current approaches to data storage, including the following:
What Is WinFS?WinFS is the new storage system in Longhorn. It improves the Microsoft Windows platform in three ways. First, it allows you to categorize your information in multiple ways and relate one item of information to another. Second, it provides a common storage format for information collected on an everyday basis, such as information dealing with people, places, images, and more. Third, it promotes data sharing of common information across multiple applications from multiple vendors. WinFS Is a Storage PlatformWinFS is an active storage platform for organizing, searching for, and sharing all kinds of information. This platform defines a rich data model that allows you to use and define rich data types that the storage platform can use. WinFS contains numerous schemas that describe real entities such as Images, Documents, People, Places, Events, Tasks, and Messages. These entities can be quite complex. For example, a person can have multiple names, multiple physical and e-mail addresses, a current location, and much more. Independent software vendors (ISVs) can also define their own new data types and provide their schema to WinFS. By allowing WinFS to manage complex storage problems, an ISV can concentrate on developing its unique application logic and leverage the richer storage facilities of WinFS for its everyday and custom data. WinFS contains a relational engine that allows you to locate instances of storage types by using powerful, relational queries. WinFS allows you to combine these storage entities in meaningful ways using relationships. One contact can be a member of the Employee group of an Organization while concurrently a member of the Household group for a specific address. ISVs automatically gain the ability to search, replicate, secure, and establish relationships among their unique data types as well as among the predefined Windows data types. This structure allows the user to pose questions to the system and ask it to locate information rather than asking the system to individually search folders. For example, you can ask WinFS to find all e-mail messages from people on your instant messenger buddy list for which you don't have a phone number. Using relational queries, you can find all members of a Household for a particular employee with a birthday in the current month. WinFS also supports multiple flexible programming models that allow you to choose the appropriate application programming interface (API) for the task. You can access the store by using traditional relational queries using structured query language (SQL). Alternatively, you can use .NET classes and objects to access the data store. You can also use XML-based APIs on the data store. WinFS also supports data access through the traditional Microsoft Win32 file system API. You can even mix and match—that is, use multiple APIs for a single task. However, for most purposes, developers will use the managed class APIs to change data in the WinFS store. It will often be far more complex to make an update using raw SQL statements as compared to using the object APIs. In addition, WinFS provides a set of data services for monitoring, managing, and manipulating your data. You can register to receive events when particular data items change. You can schedule WinFS to replicate your data to other systems. WinFS Is a File SystemFor traditional file-based data, such as text documents, audio tracks, and video clips, WinFS is the new Windows file system. Typically, you will store the main data of a file, the file stream, as a file on an NTFS volume. However, whenever you call an API that changes or adds items with NTFS file stream parts, WinFS extracts the metadata from the stream and adds the metadata to the WinFS store. This metadata describes information about the stream, such as its path, plus any information that WinFS can extract from the stream. Depending on file contents, this metadata can be the author (of a document), the genre (of an audio file), keywords (from a PDF file), and more. WinFS synchronizes the NTFS-resident file stream and the WinFS-resident metadata. New Longhorn applications can also choose to store their file streams directly in WinFS. File streams can be accessed using the existing Win32 file system API or the new WinFS API. WinFS Isn't Just a File SystemA file system manages files and folders. While WinFS does manage files and folders, it also manages all types of nonfile-based data, such as personal contacts, event calendars, tasks, and e-mail messages. WinFS data can be structured, semistructured, or unstructured. Structured data includes a schema that additionally defines what the data is for and how you should use it. Because WinFS is, in part, a relational system, it enforces data integrity with respect to semantics, transactions, and constraints. WinFS isn't just a relational system, either. It supports both hierarchical storage and relational storage. It supports returning data as structured types and as objects—types plus behavior. You might consider WinFS a hierarchical, relational, object-oriented data storage system—although it actually contains certain aspects of each of those traditional storage systems. WinFS extends beyond the traditional file system and relational database system. It is the store for all types of data on the newest Windows platform. WinFS and NTFSYou can store a file either in the traditional NTFS file system or in the new WinFS data store just like you can store things in FAT32 or on CD-ROMs or in NTFS today. Normally, a file stored in NTFS is not visible in WinFS. Longhorn applications using the new WinFS APIs can access data stored either in WinFS or in NTFS. In addition, Longhorn applications can continue to use the Win32 API to access data stored in the NTFS file system. File PromotionFiles are either in WinFS or not. Any item that has a file stream part can participate in promotion/demotion, which we more generally call metadata handling. When WinFS promotes a file, it extracts the metadata from the known NTFS file content and adds the metadata to the WinFS data store. The actual data stream of the file remains in the NTFS file system. You can then query WinFS regarding the metadata as if the file natively resides within WinFS. WinFS also detects changes in the NTFS file and updates the metadata within the WinFS data store as necessary. File Import and ExportYou can also import a file to WinFS from NTFS and export a file from WinFS to NTFS. Importing and exporting a file moves both the file content and the metadata. After importing or exporting, the new file is completely independent of the original file. WinFS Programming ModelThe WinFS programming model includes data access, data manipulation, WinFS data class extensibility, data synchronization, data change notifications, and event prioritization. Data access and data manipulation allow you to create, retrieve, update, and delete data stored within WinFS and to exercise domain-specific behaviors. Data class extensibility enables you to extend WinFS schemas with custom fields and custom types. Data synchronization allows you to synchronize data between WinFS stores and between a WinFS and a non-WinFS store. The top of the WinFS data model hierarchy is a WinFS service, which is simply an instance of WinFS. One level in the hierarchy from the service is a volume. A volume is the largest autonomous container of items. Each WinFS instance contains one or more volumes. Within a volume are items. WinFS introduces the item as the new unit of consistency and operation, rather than the file. The storage system stores items. You have rich query ability over items. An item is effectively a base type of the storage system. An item therefore has a set of data attributes and provides a basic query capability. People typically organize data in the real world according to some system that makes sense in a given domain. All such systems partition data into named groups. WinFS models this notion with the concept of a folder. A folder is a special type of item. There are two types of folders: containment folders and virtual folders. A containment folder is an item that contains holding links to other items and models the common concept of a file system folder. An item exists as long as at least one holding link references it. Note that a containment folder doesn't directly contain the items logically present in the folder but instead contains links to those items. This allows multiple containment folders to contain the same item. A virtual folder is a dynamic collection of items. It is a named set of items. You can either enumerate the set explicitly or specify a query that returns the members of the set. A virtual folder specified by a query is quite interesting. When you add a new item to the store that meets the criteria of the query for a virtual folder, the new item is automatically a member of the virtual folder. A virtual folder is itself an item. Conceptually, it represents a set of nonholding links to items, as you can see in Figure 4-1.
Figure 4-1. The WinFS data model hierarchy Sometimes, you need to model a highly constrained notion of containment—for example, a Microsoft Word document embedded in an e-mail message is, in a sense, bound more tightly to its container than, for example, a file contained within a folder. WinFS expresses this notion by using embedded items. An embedded item is a special kind of link within an item (named Embedded Link) that references another item. The referenced item can be bound to or otherwise manipulated only within the context of the containing item. Finally, WinFS provides the notion of categories as a way to classify items. You can associate one or more categories with every item in WinFS. WinFS, in effect, tags the category name onto the item. You can then specify the category name in searches. The WinFS data model allows the definition of a hierarchy of categories, thus enabling a tree-like classification of data. Organizing InformationAll these features together allow five ways to organize your information in WinFS:
WinFS APIsWinFS provides three data access APIs: the managed WinFS API, the ADO.NET API, and the Win32 API. The WinFS API is a strongly typed "high level" API. ADO.NET provides a lower level API for working with data as XML or as tables or rows. Using ADO.NET, you can access data stored in WinFS by using Transact-Structured Query Language (T-SQL) and, when you want, retrieve data in XML using the T-SQL's FOR XML capability. The Win32 API allows access to the files and folders stored in WinFS. You might prefer to use multiple access patterns to solve a problem. For example, you can issue a T-SQL query that returns a set of contacts as managed objects of the WinFS Contact type. Regardless of the API you use, each API ultimately manipulates data in the WinFS store using T-SQL. In many cases, you will prefer to use the managed WinFS API. These .NET Framework classes automatically perform the object-relationship mapping needed to translate between object-oriented programming constructs, and they perform the necessary T-SQL to achieve the WinFS data access. Using the Managed WinFS ClassesThe WinFS managed classes reside in the using System.Storage; ItemContextThe WinFS store consists of items organized into folders and categorized. The first step in working with WinFS is to identify the set of items with which you want to work. We call this process binding, and the set of items can be any of the following:
To bind to a set of items, you create a The following code creates an System.Storage.ItemContext ctx = System.Storage.ItemContext.Open (); Alternatively, you can pass a string to the constructor to connect the item context to a specific WinFS store. The following code creates an item context connected to a WinFS share identified by the \\machine\Legal Documents share: ItemContext ctx = null; Be sure to close or dispose of the context object as
soon as you finish using it regardless of exceptions. An using (ItemContext ctx = ItemContext.Open (@"D:\MyStore")) {
Storing a New Item in a WinFS Data StoreEvery item in a WinFS data store must be a member of a
folder of the store. You obtain the root of the folder hierarchy by
calling the extremely well-named static method Getting a FolderIn the following example, I'll find the current user's Personal Contacts folder if it exists and create it when it doesn't exist. Note that this is a somewhat contrived example—the system automatically creates a user's Personal Contacts folder if it doesn't exist when the user first logs into a system—but it gives me a chance to show how to create an expected folder when it doesn't exist. ItemContext ctx = ItemContext.Open (); The preceding code does a number of interesting things.
First, I try to locate an existing folder contained in the user's
personal data folder hierarchy. I'm not looking for the folder by a
well-known name. Instead, I'm locating the folder within the user's
personal data tree that has previously been associated with the
well-known category This folder normally already exists, but when it
doesn't, I retrieve the root folder for the user's data hierarchy. I
create a new item, of type Of course, this is the verbose way to find the Personal
Contacts folder. I wanted to show you how things work. Normally, I'd
use the following code instead. The WellKnownFolder userDataFolder = Creating a New ItemAs I now have the Personal Contacts folder, it seems appropriate to create a new contact in the folder. In the following example, I'll create a number of Person contacts and add them to the folder: Person[] CreateFriends (ItemContext ctx) {
The prior code uses the static
I subsequently update the Let's look more closely at the // Find the PersonalContacts folder Relationship ItemsWinFS defines a relationship data model that allows you
to relate items to one another. When you define the schema for a data
type, you can define zero or more relationships as part of the schema.
For example, the Folder schema defines the FolderMember
relationship. The Organization schema defines the A relationship relates a source item to a target item.
In the previous example, the Personal Contacts folder was the source
item and the Person item was the target item. The When you define a relationship, you define whether the relationship keeps the target item in existence—a holding relationship—or doesn't keep the target item in existence—a reference relationship. When you create a holding relationship to a target item, WinFS increments a reference count on the target item. When WinFS deletes a holding relationship it decrements the reference count on the target item. An item no longer exists in the store when its reference count reaches zero. WinFS never alters the reference count of the target when you create or destroy a reference relationship to the target. Therefore, the target item can disappear from the store when its reference count reaches zero and the relationship might refer to a no-longer-existing item. WinFS defines the Folder ItemsNow that you know about Link items, I can refine my
description of Folder items. A Folder is a WinFS item that has a
collection of Link items. The target of each Link item in the
collection is a member of the folder. The Note this gives a WinFS folder much greater flexibility than traditional file system folders. The members of a folder can be file and nonfile items. Multiple links to a particular item can reside in many folders concurrently. In other words, multiple folders can contain the same item. Other Item TypesGenerally, you create other item types in the WinFS store as you did in the previous examples. Each type occasionally has its own special usage pattern. For example, we can have organizations as members of our Personal Contacts folder, so let's create one: Organization cp = FindOrCreateOrganization (ctx, "Main Benefit"); Now let's add an employee to that organization: enum Names { Monica, Rachel, Chandler, Joey, Phoebe, Ross }
Similarly, we can create households in our Personal Contacts folders. Note that a household doesn't imply a family. A household might be a group of roommates. WinFS has additional schema for families, but I'll leave that as an exercise for the reader. CreateHousehold (ctx, Friends [(int) Names.Chandler], The prior example uses one concept I've not yet
discussed. Note the use of the hhmd.Target_Key = p1.ItemID_Key; Basically, the How to Find ItemsOf course, it doesn't do much good to place items in a
data store if you cannot subsequently find them easily. The The FindResult res = ctx.FindAll (typeof(Person), "DisplayName='Brent%'"); Alternatively, I could use the static FindResult res = Person.FindAll (ctx, "DisplayName='Brent%'"); In both of these examples, the When you know that only a single item will match the
type requested and specified filter criteria, you can use the Person p = Person.FindOne (ctx, "DisplayName='Brent Rector'"); The second string parameter is a filter expression that
allows you to specify additional constraints the returned items must
satisfy. The basic format of the filter expression is a string in the
form " WinFS calls the expression an FindResult Files = File.FindAll (ctx, "Extension='doc' || Extension='txt'"); These expressions can be quite complex. For example, the
following statement returns all Person items that represent employees
of an employer with the string pattern = "Source(EmployeeOf).DisplayName='Main Benefit'"; Here's another one. I want the Person items where the Surname is not "Ranier" and the e-mail addresses don't end with ".edu". string filter = "PersonalNames[Surname!='Ranier'] && Identifying a Specific ItemYou frequently need to create references to items in the WinFS store. Eventually, you use these references to locate the appropriate item. Earlier in this chapter, I showed you how to use a link to reference an item. Links use a friendly string-based identity for the reference, and this string name must be unique within the link's containing folder. In other words, you need both the folder and one of its contained links to identify the referenced item. However, you can create multiple links with the same friendly string name as long as you add the links to different folders so that all names within a single folder remain unique. Note that these multiple links with the same friendly text name don't actually have to reference the same target item. They could, but they don't have to. In such cases, searching for all links with a specific
friendly text name (using We need a way to reference any arbitrary item in the store—for example, suppose I want the 3,287th item in the store. Fortunately, you can do exactly this. Finding an Item by ItemID_Key ValueWinFS assigns each newly created item a GUID-based
identification number, known as its Item GetItem (ItemContext ctx, SqlBinary itemID_Key) {
Common FeaturesWinFS API provides several features across the entire spectrum of data classes. These features are
AsynchronyThe WinFS API allows you to run queries asynchronously. The WinFS API uses the .NET standard asynchronous programming model patterns. TransactionsThe WinFS store is a transactional store. WinFS,
therefore, allows you to make transactional updates to the store using
the using (ItemContext ctx = ItemContext.Open()) {
NotificationsThe WinFS Notification Service uses the concepts of short-term and long-term subscriptions. A short-term subscription lasts until an application cancels the subscription or the application exits. A long-term subscription survives application restarts. WinFS API watchers are a set of classes that allow applications to be selectively notified of changes in the WinFS store and provide state information that can be persisted by the application to support suspend/resume scenarios. The
When a watcher raises an event, it sends watcher state data with the event notification. Your application can store this state data for later retrieval. Subsequently, you can use this watcher state data to indicate to WinFS that you want to receive events for all changes that occurred after the state was generated. The watcher programming model also allows any combination of added, modified, and removed events to be disabled. It can also be configured to raise an initial event that simulates the addition of all existing items, item extensions, relationships, and so on. The WinFS watcher design is broken down into the classes described in the following table.
You use the
Because by default a watcher watches for changes in the
specified item and its descendants, you might want to specify Person p = Person.FindOne (ctx, A Folder is just another WinFS item. You watch for changes in a Folder the same way you do for a Person: Folder f = · · · You can watch for changes in a specified relationship of an item, too: Person p = · · · Blob and Stream SupportBlob and stream support APIs are still in flux at the time of this writing. Check the documentation for the latest information about how to access blobs and streams in the WinFS store. Cursoring and PagingThe various By default, a find operation provides a read-only, scrollable, dynamic cursor over the returned collection. An application can have a fire hose cursor for maximum performance. A fire hose cursor is a forward-only cursor. The application can retrieve a page of rows at a time, but the next retrieval operation will begin with the subsequent set of rows—it cannot go back and re-retrieve rows. In a sense, rows flow from the store to the application like water from a fire hose—hence the name. The Data BindingYou can use the WinFS data classes as data sources in a
data-binding environment. The WinFS classes implement SecurityThe WinFS security model fundamentally grants a set of
Getting Rights on an ItemEach WinFS item class has a method named Setting Rights on an ItemWinFS uses a special Item type, using (ItemContext ctx = ItemContext.Open("\\localhost\WinFS_C$")) {
Extending the WinFS APIEvery built-in WinFS class contains standard methods
such as Domain BehaviorsIn addition to these standard methods, every WinFS type will typically have a set of domain-specific methods unique to that type. (Actually, WinFS documentation often refers to type definitions as schema, reflecting the database heritage of WinFS.) WinFS refers to these type-specific methods as domain behaviors. For example, here are some domain behaviors in the contacts schema:
Value-Added BehaviorsData classes with domain behaviors form a foundation that application developers build on. However, it is neither possible nor desirable for data classes to expose every conceivable behavior related to that data. You can provide new classes that extend the base
functionality offered by the WinFS data classes. You do this by writing
a class whose methods take one or more of the WinFS data classes as
parameters. In the following example, the MailMessage m = MailMessage.FindOne (…); You can then register these custom classes with WinFS. The registration data will be associated with the schema metadata WinFS maintains for every installed WinFS type. WinFS stores your schema metadata as WinFS items; therefore, you can update, query, and retrieve it as you would all other WinFS items. Specifying ConstraintsThe WinFS data model allows value constraints on types.
WinFS evaluates and enforces these constraints when you add items to
the store. However, you sometimes want to verify that input data
satisfies its constraints without incurring the overhead of a roundtrip
to the server. WinFS allows the schema/type author to decide whether
the type supports client-side constraint checking. When a type supports
client-side validation, the type will have a validate method you can
call to verify that an object satisfies the specified constraints. Note
that regardless of whether the developer calls the Using the WinFS API and SQLThe WinFS API enables a developer to access the WinFS store by using familiar common language runtime (CLR) concepts. Throughout this chapter, I used the following coding pattern for WinFS access:
Step 2 is essentially a query to the store. The WinFS
API uses a filter expression syntax based on The following capabilities are present in SQL, but they are not available when using a filter expression:
It is thus essential that a WinFS developer be able to seamlessly transition between the SQLClient API and the WinFS API, using one or the other in various places in the code. Aggregate and Group with SQL, and Then Use WinFS APIA small-business owner, Joe, wants to determine who his top 10 customers are and send gift baskets to them. Assume that Customer is a schematized item type. This means that an ISV has provided a schema for the Customer type to WinFS, and therefore, it also means that a WinFS store can now contain Customer items. A Customer item has a holding link to a schematized Order Item type. Order Item has an embedded collection of Line Orders, as follows: 1. using (ItemContext ctx = ItemContext.Open()) {
In line 1 of this example, I open a context for the root of the system volume. In line 3, I create a SQL command object that I subsequently use to execute a SQL query against the WinFS store. This command object reuses the connection used by the item context. Lines 4 through 12 construct the query, and line 14 executes the query. The query returns the top 10 customers in the following manner: the SELECT statement in lines 6 through 12 generates a grouped table containing the total value of each customer's orders; the ORDER BY clause on line 12, combined with the TOP 10 modifier in line 6, selects only the top 10 customers in this grouped table. The Line 19 uses the When you define a new type in WinFS (known as creating a
new schema), you are actually defining two types: your managed class
and the WinFS store's persistent format of the class. WinFS always adds
the Data suffix to the name of your class to create the name of
the store's type. Therefore, for example, when you define a new
Customer type that resides in the WinFS store, WinFS creates the
parallel The first column of the rowset contains the store's Line 24 adds the customer to the Finally, I use the Navigate in API, and Then Aggregate in SQLAssume that you want to find the average salary (over a 10-year period) for the CEO of each company in my portfolio. Use the following assumptions:
1. using (ItemContext ctx = ItemContext.Open(@"Companies In My Portfolio")) {
Line 1 opens a context for the Companies In My Portfolio
WinFS share. Lines 3 through 11 create a parameterized SQL query that I
can use in the context of the folder. This query returns the average
salary for a given employee (represented by the Line 13 gets a For each organization, line 17 gets the Line 19 prepares for the query and sets the value of the parameter to the appropriate LinkID, and then line 20 executes the parameterized SELECT. Line 21 reads the next and only row from the query result, and lines 22 and 23 print the name of the CEO and the 10-year average salary. SummaryThe WinFS data store provides a far richer data storage model than traditional file systems. Because it supports data, behavior, and relations, it's difficult to categorize WinFS as a file system, a relational database, or an object database. It's a bit of all those technologies in one product. WinFS provides a common definition of ubiquitous information that is globally visible and available to all applications running on Longhorn. Applications can leverage the query, retrieval, transactional update, and filtering capabilities of WinFS; therefore, the developer spends less time developing data access and storage code and more time working on unique application functionality.
|
Top
of Page