Seven Experiences with Contextual Field Research

Edited by Michael Good

Digital Equipment Corporation
Nashua, New Hampshire, USA

Introduction

Contextual field research is a growing area of interest in human-computer interaction. At the CHI ’88 conference, Karen Holtzblatt, Sandy Jones, and myself hosted a Special Interest Group on “Field Research Techniques for Building Usable Products.” This overview of seven different experiences with contextual field research techniques grew out of the network we established at the conference.

These contributions show a wide range of experience with contextual field research, including its use in academic research and the design of diverse computer products. We hope that this snapshot of contextual field work in mid-1988 may excite your curiosity about what may be possible from this perspective, currently out of the mainstream of human-computer interaction research. The bibliography at the end of this article includes sources for more information on contextual methods, as well as references from the individual contributions.

Contextual Field Research in a Usability Engineering Process

Michael Good
Digital Equipment Corporation

Over the past few years, Digital’s Software Usability Engineering group has adapted engineering techniques to the design of usable computer systems. These usability engineering techniques have evolved from an emphasis on laboratory testing of user interfaces to an emphasis on contextual field techniques.

Two main factors motivated us to change our focus to contextual field research. First, our laboratory work was often not sufficient to produce usable, competitive products. Although the lab tests often met their goals, the goals were usually not grounded in customer experience.

Second, we became aware of theoretical and philosophical work (presented, for instance, by Winograd and Flores) which argues for the importance of context in understanding human behavior.

Visiting our customers to understand their needs is now one of our primary techniques for designing and building usable systems. A contextual approach now informs our other work, including the way we collect data in a laboratory setting. Contextual field work has been one of the major techniques used in the design of the XUI user interface style and DECwindows workstation software.

Data collected at the user’s work place provides insight into what users need in both new and modified systems. During interviews of users actually working with their systems, we ask about their work, the details of their system interfaces, and how they perceive various aspects of the system. The user and the engineer work together to reveal how the user experiences the system as it is being used. We find that these visits with users are the best way for us to learn about users’ experiences with computer systems.

Contextual interviews reveal users’ ongoing experience of a system. Other types of interviews, which are not conducted while the user works, reveal users’ summary experience, or experience as perceived after the fact. We find that data on ongoing experience provides a richer source of ideas for interface design than data on summary experience.

We have used data from contextual interviews in three ways. First, we have used this data to begin building a theory of usability that is grounded in user experience. This theory is then used throughout our interface design work. Second, we analyze data about user’s experience with particular computer applications to provide detailed information for the designing of specific products. Finally, we are beginning to use contextual and action research to generate and develop ideas for new products.

For example, data collected from field studies has revealed the importance of interface transparency to users. A transparent interface allows the user to focus on the task rather than on the use of the interface. We have used this principle throughout our interface design, and it underlies our work to improve system performance, reduce unnecessary mouse movements, and support hypothesis testing in DECwindows application programs.

We have found it is important to consider the diversity of environments in which people will use a system when developing new products. Different users in different contexts have different usability needs. In our experience, some important aspects of user’s context have been:

Type of work being performed
Physical work place environment
Interaction with other software systems
Social situation
Organizational culture

All these aspects influence the usability of a system for each individual. As with other products, software systems are used in the field in ways not anticipated by the designers.

Because the context in which a system is used is so important, we interview a variety of users who use particular products to perform different tasks. We look for common elements of usability for groups of people, as well as the distinctive elements of usability for individual users.

Ideally, the number of interviews conducted per product depends on how much data is being generated in each succeeding interview. The interview process stops when new interviews no longer reveal much new usability data. In practice, resource and time limitations may stop the interview process before this point. In any event, our approach is to start with a small number of interviews (four or less) with people in various jobs. We use these interviews to determine how many and what type of users will be most useful for uncovering new usability data.

Whenever possible, we videotape interviews. If users are unwilling to have their work videotaped, we audiotape the session while the second team member takes detailed notes to supplement the taped information. The two team members meet after the interview to reconstruct an accurate record of events.

Even without any taping or note-taking, engineers can learn a great deal from user visits. Although the detail from the interview may not be remembered, the understanding gained during the interview is still a valuable source of insight.

We believe that our contextual approach to interface design has greatly increased our effectiveness in producing more usable systems. We also find contextual field work to be more enjoyable and fulfilling than our earlier laboratory work. We are continuing to develop and evolve our techniques to build even better systems which enrich human experience.

The Elicitation and Interpretation of Experiential Data Within an Iterative Design Framework

Peter C. Wright and Andrew F. Monk
University of York

At York we are currently examining methods for eliciting and interpreting experiential data as part of our research on iterative design. The problem we have chosen is how to provide a cost-effective method for evaluating single prototypes. Our approach is to combine a variety of methods within a hypothesis testing framework. The aim is to identify problems, diagnose their cause and pass on recommendations to designers as rapidly as possible and with the minimum of effort. Each stage in the evaluation will identify a certain class of problems, some of which can be diagnosed immediately and some of which will need to be further explored at the next stage. Experiential data is collected at the end of this process. We argue that without it many important usability problems can not be identified or their causes diagnosed.

It is useful to think about usability problems as forming a hierarchy with each successive level in the hierarchy requiring more contextualized knowledge for its diagnosis. By contextualized knowledge we mean knowledge about a particular user and the goals and priorities which are relevant when the problems are identified. An example of a problem at the bottom of the hierarchy is a user having difficulty in correcting typographical errors because of inadequate delete facilities. This may be a quite general problem independent of the particular user and the task being performed. An example of a problem near the top of the hierarchy is the difficulty that a user might have in understanding how to carry out a database search task on some new system. The problem arises from the user’s experience with the system, with other systems and with the task it is being used for and so depends very much on the context. Such problems often have a large impact on usability. By definition, these problems require information about the user’s knowledge and beliefs in order to diagnose their cause, and to recommend the changes needed.

The work at York has attempted to develop methods which elicit increasing amounts of context-sensitive data which can be used to diagnose problems at successively higher levels of this hierarchy and have identified four diagnostic levels. At the first level are problems that can be identified from system logs recorded from unknown users during free use. An approach was developed which identifies problems by isolating redundant user input. These problems can be diagnosed with minimal user-specific or task-specific information. A second level involved the analysis of log data from users completing tasks which have been set by the evaluator of the system. When these tasks are well constrained the system evaluator can use them to gain partial information about the users’ plans and goals when problems occur. This method is not always adequate since strategic differences amongst subjects can make plans and goals difficult to infer.

A third level used verbal protocol methods to elicit users’ plans and goals directly. Two methods have been used re-enactment and teachback. Re-enactment proved particularly useful for diagnosing problems that could not be diagnosed from log data alone. In this method the user and evaluator discuss problems that the user had experienced during previous sessions. The system logs are used by the user to re-create the problem and the system evaluator questions the user about them.

The problems identified by the above methods do not necessarily require the evaluator to have any information about the user’s knowledge of how the system works or what the system can do. The fourth level used a verbal protocol technique developed from Kato’s (1986) question-asking method to diagnose problems involving this knowledge. For this method the user is encouraged to treat the evaluator as a personal tutor and ask questions whenever there are difficulties. Unlike Kato’s method the evaluator is allowed to ask the user questions as well as the user the evaluator.

In addition the users are encouraged to think of themselves, not as experimental subjects or tutees, but rather as co-evaluators of the system. This question-answer dialogue method provides a means of detecting usability problems where there is no observable problem which could be identified in a system log, for example, but where the user still feels that the system is awkward or obscure.

Our research aims to validate this approach by considering a specific application, a reference database, with a number of identifiable usability problems. This application has been studied by the group for some time and so the problems inherent in its design are relatively well understood. The advantage of this is that the procedures themselves can be evaluated by examining their ability to detect and diagnose different types of usability problem.

Thus far we have completed a detailed study of a single user-evaluator pair and are just starting a larger scale study in which groups of evaluators will be trained in different methods and their effectiveness examined. Early results show that the experiential data provided by the question-answer dialogue method was particularly useful for diagnosing important problems caused by a mismatch between the designer and user’s view of how the system should work. Inadequate displays and missing functionality are examples. It was these context-sensitive problems rather than problems at any of the lower levels which affected the user’s judgments of the usability of the system. Recently there has been increasing recognition of the need for contextualized research in HCI (e.g. Whiteside et al., 1988). Any usability judgments that are made as part of an evaluation process are conditioned by what perspective on usability has been adopted by the evaluators. Thus it would seem that evaluators should strive to encompass as a broad a perspective as possible. This must be reflected in the kind and range of data they collect and methods they use. Our experiences at York have demonstrated the value of the contextualized approach to usability not as an alternative to system-oriented perspectives but as an extension to them.

Experience with the Design of a Real-Time Computer Performance Monitor

Barry Tavlin
Candle Corporation

We at Candle are relatively new to the field of Usability Engineering. A little over a year ago, I helped to set up a Human Factors department. In the last year. we have worked to improve the usability of some of Candle’s products.

At first that work consisted of writing what we call “textware” — help screens, command help texts, and menus for an essentially command-driven product. Our main effort was the menus. However, more recently we began to realize that menus alone are not enough. Then I attended CHI ’88 (my first CHI conference) and was very excited by what I saw there.

Since CHI ’88, I have attempted to implement the Usability Engineering ideas (including field research) that were presented at the conference. This has been very successful. In August, I discussed our results at SHARE 71. (SHARE is a users’ group for large IBM customers). Briefly, here is our story.

The Menu System

Candle’s main product is a realtime computer performance monitor for large IBM mainframes. The data that our product reports is highly technical, and historically it has only been of interest to a small, highly technical group within a data center. This group was able to function effectively with our product’s interface — 800 commands, each a 4-letter acronym.

In the last few years the audience for our products has grown and changed considerably. As a result our company has come to see the need for more than just a command language interface. Initially the R&D management saw menus as a way to solve the problem. So our department led in the design and implementation of the menu system.

The menus were well received by the sales force and new customers, the target audience for the new interface. And though they were a significant improvement, they did not totally solve the usability problem. When I went to our own data center, I was surprised to find that the menus were not being used. It turns out that though they were helpful for new users, they were not well enough tailored to the job function of our audience. As a result, the menus were not being used on a daily basis.

Our Field Research

As we investigated further, we found that the menus were not well enough designed for system monitoring. Most of our users have a single screen that they glance at to monitor their system. We took the information that was on that single screen and distributed it throughout the menu system. We thought we cleaned up the screen. In fact we essentially had hid some valuable data.

Though the people who had written the menus were knowledgeable in the product and talked regularly with the customers, we did not really grasp the key to our product’s usability until we went out into the field and took a look around.

Fortunately, we have been able to improve matters in a product release currently in development. Through our field research we have developed a prioritized list of user job functions that our product is supposed to help address. We have provided this list, this user model, to the development group to assist them in designing the new release’s functions as well as its user interface.

Usability EngineerIng

Since the CHI conference, we have attempted to implement usability objectives in the software development process as well. We have not yet gotten to do our testing in this case, so it is too soon to tell how all that will work out.

Conclusion

Though our initial charter was “textware”, our department has done a lot to help define usability for our company and to incorporate it in the software development process. In this work, we made a serious effort to go out among the users and be willing to be surprised. And surprised we were. We have learned from the process and as much as possible had the developers learn the lessons with us.

We hope to continue and strengthen our work in this area.

Experience with Participant-Observation Methodology

Steve Poltrock
MCC

The Human Interface Laboratory at MCC is developing a collection of tools called HITS for designing, implementing, and evaluating applications with intelligent user interfaces (Hollan. Miller, Rich, & Wilner, 1988). The intended users of HITS are members of large software development organizations. The goal of our research was to determine the communication, coordination, and collaboration activities in these organizations, and to explore how these activities could be supported by HITS. To collect this information about software development organizations, we used a participant-observation methodology. When using this methodology, the investigator becomes an active member of the organization, sharing in the organization’s goals and activities. Concurrently, observations are collected through several means, including recorded interactions or meetings with other participants, and interviews of people throughout the organization.

Two investigators visited development organizations at two different major computer companies. Over a period of about a month, each investigator participated in user interface design in collaboration with other members of the project. The investigators’ time was divided about equally among three kinds of activities: (1) becoming familiar with the technology of the development environment and becoming familiar with the product, (2) observing and participating in user interface design, and (3) conducting and analyzing interviews.

The investigators observed and noted the user interface design process, including uses of technology for communication and prototyping. They recorded the spatial layout of the facilities, including locations of related work groups and conference facilities. Forms were used to record the participants, structure, and content of meetings.

Twenty three structured interviews were prepared in advance, including separate interviews for different kinds of development and testing personnel, marketing and support personnel, and education and technology transfer personnel. Preparation of the interview questions helped establish a focus for each interview. The first few interviews closely followed the planned questions, but soon they were only used as reminders of major topic areas. When they understood our goals, the interview participants were able to tell us more without the structure imposed by the planned questions.

The amount of time spent interviewing was relatively small at the outset of our participation and increased toward the end. Interviews were tape recorded, except for two instances when the respondents objected to recording the interview. The sponsors and other principal informants were interviewed first to obtain background information about the project, organization, and communication channels, and to identify other people to be interviewed.

Next, the investigators interviewed their immediate co-workers about their background, involvement in the project, procedures, interactions with other groups and individuals, meeting attendance, and channels of communication. These interviews with co-workers and informants provided a local view of the organization. Subsequent interviews included people in development and support groups including marketing, documentation, human factors, industrial design, software, and other professions that bear on any phase of the design. The interviews eventually included the supervisors and managers of these designers. As time and opportunity permitted, the interviews were extended to people in projects parallel to the one in which we participated. Each investigator conducted about 25 interviews, generally lasting about an hour apiece.

One focus of our research was on user interface design and development practices. Each organization had a theory about the way they design user interfaces and how users’ needs are accommodated. In each case, their practices differed somewhat from their theory. We also identified ways in which the practices of both organizations posed serious obstacles to the design and development of successful, innovative user interfaces. Other analyses focused on organizational influences on user interface design, the ways the organizational structure shaped communication and collaboration, communication problems, ways in which tools are used, and technology transfer.

Notes About Some Experiences With Contextual Research

Richard Anderson
Pacific Bell

Contextual research is an important part of the work that I do as a human factors consultant within Pacific Bell. In these notes, I briefly describe the nature of some of my contextual research, what I do with findings from this work, and how well-thought-of the contextual research has been. My descriptions identify and focus primarily on questions and concerns that I have about this approach to developing usable products.

Nature of My Contextual Research

I conduct contextual research to identify difficulties, confusions, dissatisfactions, etc. that users experience while interacting with computer systems. Concisely stated, my usual method consists of the recording (via notetaking) of my observations of users during their “normal” use of a computer system in their work environment while they think aloud and are participating knowingly for the purpose of improving the system. Via this method, I attempt to tap “normal” user experience as it happens.

The word “normal” is highlighted above, since, of course, normality of use and experience is altered by elements of the research method. A particular concern of mine is the extent to which one of these elements – my communication to a user during my observation – impacts a user’s experience.

According to Lewis (1982, p. 4), “It is very easy to influence by questions or comments the data participants will give you. The observer must try to remain an observer, and not an interviewer. Let the participants tell you what THEY are thinking. …avoid probes like “Why did you do that?” However, Whiteside, Bennett, and Holtzblatt (1988, pp. 33-34) advocate being an interviewer, promoting use of a “contextual interview … during which the usability engineer observes and questions the user… Together the user and usability engineer interpret the user’s experience…” My approach is to try to limit my communication during most of the user-system interaction to prompts for thoughts when none are being voiced. However, I do sometimes go further by asking for clarification of user comments, asking what, if anything, the user thought of something, and by asking “Why?”.

When do these questions excessively influence users’ thinking? Can one tap a user’s interpretation of his or her experience close to the moment of experience without contaminating further experience? Can usability engineers make good judgments on the fly about when “interview” questions won’t mess things up?

What I Do With My Findings

After collecting data, I carefully separate problems that users experience into categories that emerge from the data (cf. Wixon & Good, 1987). Then, I write a one-page report for each category consisting of a description/definition of the category, a description of the category’s ramifications, and identification of the critical ingredients of design solutions (see Figure 1 for an example).

Absence of or confusions in knowledge of available control options, including

immediate return to menu, aborting activity in progress (via 0 or, in certain situations. RETURN)

access to on-line help

editing arrows & line clear

backing up within a screen (whether and how)

Ramifications:

lots of wasted time [e.g., users step through entire user profile creation process just to be able to respond “no” to the “OK to update (YES)?” prompt]

potential for error (e.g., accidental update is possible when a user is not aware of a means to abort a process; since users are unaware of the potentially helpful on-line help, misunderstandings about fields persist)

Solution component:

display of active control options on the screen

Figure 1. Sample One-Page Report About a Category of Difficulties, Confusions, Dissatisfactions, …

The descriptions of each category’s ramifications are as far as I go with any analysis of the impact of problems on user performance and satisfaction. Without measures of problem times, analyses of impact on total performance or problem time, such as via use of the Pareto Diagram (see Whiteside et al., 1988), are not possible. Hence, the establishment of redesign priorities by myself or by the developers is problematic, further opening the door to the influence of redesign ease.

I do supplement identification of critical components of design solutions with ideas about the forms the design solutions might take. However, in an attempt to not invade developer territory too deeply, I emphasize the critical components, stating that many design solutions may exist.

The package that I present to a product’s developers consists largely of the one-page reports. Other contents include discussions of the research goals, the research method, user interface design standards to which the developers should attend, and recommendations about the use of iterative prototyping.

How Well-Thought-Of the Contextual Research Has Been

Almost all have responded positively to my use of contextual research. However, the resistance that has been experienced is noteworthy.

The absence of quantification, the lack of control over user tasks and the environment, and the small number of “subjects” seem to contribute to a claim that contextual research is unscientific. Is this claim justified, or does it reflect an excessively nanow definition of the word “science”? Is revelation of the findings of this type of field research inappropriate without validation from controlled experiments, or, borrowing words from Carroll and Campbell (1986, p. 244), are these “arguments [that] appeal much more strongly to the demand for scientific credentials than to any benefits for interface design”?

Experiences in Oscilloscope and Video Editor Design

Steve Knox, Eugene Lynch, Wayne Bailey, and Scott Lewis
Tektronix

Easily the most exciting events of CHI ’88 were the Usability Engineering tutorial and the session on contextual research. Much of the research described there served as an affirmation and as a direction for our efforts here at Tektronix.

For the past year we have sought to supplement our traditional laboratory research based on benchmark tasks and time and error measures with video-taped sessions of customers first-time exposure to our products. The analysis of these sessions is more qualitative in nature, as we attempt to probe and understand our customers’ expectations about the product’s functionality and its operation.

Our typical session is broken up into two parts. In the first we ask a customer (who has never seen this specific product before) to invoke state-changes and specific functions which traditionally require the manipulation of a single control. We call these tasks “atomic”, as they comprise basic operations common to almost all applications. The data of interest here is the particular functions the customer expects to utilize and the relative ease (difficulty) with which these functions can be accessed. An experimenter is present at the time who acts as a tutor when necessary and also probes for verbal data when the customer behaviors appear ambiguous.

In the second part, we ask the customer to perform general tasks representative of the product’s applications. The data here is the degree to which the basic operations tested in the first part are retained and generalized to more complex functions. This procedure, which we have called the “directed dialogue”, has served us well, particularly in the early design phase as a means of testing the fit between a product concept and customer needs. Each session will result in modifications to the concept, which are then implemented in either simulations or prototypes. Iteratively the simulations are presented to customers and revised, until we reach a predetermined usability goal (or the mechanical deadline is at hand, whichever comes first).

The special interest session on contextual research was a radical revelation for us. It appeared to offer something which we felt the directed dialogue lacked: the tasks which we asked customers to perform, both atomic and general, assumed that these were relevant and germane to the current problems experienced by our customers in their work.

This summer we have made some initial forays in conducting contextual research and have experienced some success. For example, our laboratory and directed dialogue tasks for oscilloscope usage have always focused on the performance of some sort of measurement. However, four contextual interviews conducted with customers has shown us that displayed waveforms are often not measured but are used to convey qualitative information. That is, rather than making a precise measurement of some property of the waveform, in some applications the information sought concerns the presence of a signal or its particular shape. This kind of information has resulted in some adjustment to the tasks we employ in the directed dialogue.

Additionally, we have conducted a contextual interview with a customer learning to use a video editor manufactured by one of our subsidiaries. From this interview we learned that the underlying premise of the editor interface assumed sophisticated computer literacy on the part of the post-production operator, which is not always the case in established corporate-based media centers. We expect further contextual research to offer insight into the language and visual presentation of the editor interface which will be most apparent to these customers.

As you can see, the contextual interview serves us well in offering insight into customer needs and providing hypotheses to be tested in our directed dialogue sessions. We also feel that the contextual interview has an application for addressing marketing issues, and we plan to expand its use toward that arena.

Extending the Scope of Field Research in HCI

Robert L. Campbell, Robert L. Mack, and Joan M. Roemer
IBM

A major concern in our work at IBM Research is developing research methods that will increase our understanding of users’ needs and their experiences of usability. Quantitative measures, like keystroke counts and performance times on benchmark tasks, leave out most of what is important about usability: the kinds of errors that users make, their satisfaction or dissatisfaction with features of the interface, etc. Such information can only be obtained through qualitative measures, such as various forms of interviews, thinking out loud, and video observation. When used in the laboratory, qualitative methods generate much useful information about users’ interactions with systems, but in an artificial environment in which users work on contrived tasks for limited amounts of time at an unrealistic pace. Because systems need to be usable by users working on real tasks in real work settings, we are devoting increasing attention to field research, and to improving the methods that we use in the field.

Field Studies of Online Help

One area in which we are currently developing our field methods is research on task-oriented online assistance. This work began with a laboratory evaluation of the help system for a command-based text editor. In the laboratory study, we found that thinking out loud, in conjunction with monitoring of help use, provided rich and extensive information about the kinds of difficulties users have with existing help systems, and led to many useful suggestions from the users for reorganizing the help text and reducing its wordiness. Where a help system has to be usable, however, is in users’ offices when they are working on tasks of their own choosing. Users in the laboratory study in fact remarked that the tasks were contrived and that they were learning at an unnatural pace, moving on to new functions without having time to practice the old ones. In consequence, we considered it necessary to move our evaluation of online help into the field.

Online help is a challenging subject for field studies because it forces them to be longitudinal. Learning to use a system takes weeks or months. Spot interviews or one-time observations cannot convey the nature of the learning process. Even repeated interviews miss the details of satisfactory and unsatisfactory interactions with the help system, as users quickly forget these. Monitoring help use produces detailed records over time, but fails to convey the meaning of interactions with the help: what problem the user was trying to solve, and whether the help provided relevant information or not.

In a study now under way, we are using keyboard-activated tape recorders into which users can make spoken comments while using the help for our text editor. In effect, we are asking for localized thinking out loud. We have opted for this approach because other ways of recording comments are too distracting for users who are already distracted when they need to use the help (Campbell, 1988). To the extent that users are willing to make spoken comments, we will have a method for tracking users over time in the field, which could be applied to other features of the interface as well. Of course, we need to be concerned not only with the quality of information we obtain in this way, but also with the value of going to the additional time and expense to do such field studies in the course of a usability engineering approach to product design.

Field Studies of Professional Work

Field studies of professional work have been an important element of the empirical design approach at IBM Research for many years (Gould, 1988). In our past work (Nielsen et al., 1986; Mack & Nielsen, 1987), and again in work in progress, we have used surveys and interviews to understand the computer support needs of business professionals, and to develop requirements for integrated office systems.

Our field work has been carried out as part of the design of advanced office system prototypes, where real-world constraints shape the conduct of that field work. The key constraint is the simple fact that we have very limited access to business professionals. We cannot spend much time with them in face-to-face conversation or observation. These limits are built into the work schedules of business professionals, and we have had to accept them.

In this context, the conduct of field studies does not uncover objective facts, but generates interpretations that blend multiple perspectives, based on multiple instruments, each of which provides incomplete information. No one instrument provides enough understanding. Mack and Nielsen (1987) found that surveys were more valuable when complemented by follow-up interviews, guided by the survey results. The surveys elicited demographic information from professionals, along with a quantifiable summary of standard or generic categories of tasks and computer usage, and provided a background for carrying out a semi-structured interview. The interview, in turn, was aimed at confirming and elaborating the interpretations developed through the survey.

More recently, we have found it useful to develop our interpretations from interviews with multiple sources, in particular with the staff and secretaries of business professionals. We summarize our interpretations as a narrative that we then present to our target professional for comment and elaboration. Although this tends to structure the interview in terms of existing work activities, it is useful in the face of the time constraints we accept in working with the professional. Building a picture like this provides converging evidence for our interpretations, and it focuses our subsequent interview with the target professional.

Developing this composite picture of a professional’s work does not proceed in a linear fashion. There are dialectical interactions between different informants and different levels of analysis. Interview questions can work bottom-up from procedural details to their rationale, or top-down to elaborate general goals into subgoals and procedural details of subgoals. Initially, we tried to formalize this process of tracing out goals and subgoals, working from the bottom up or from the top down, by building on Graesser’s question-asking technique (Graesser & Murray, 1988).

Because of our limited access to professionals’ time and their impatience with drawn out interviews, we found that a formal, mechanical question-asking process would not work. Our interpretations take the form of narrative summaries of work scenarios, focused on procedural aspects of work (e.g., data or information objects manipulated, tools used, temporal characteristics of the work process). In this work, we have focused on what professionals need to know, have or do to accomplish goals. A focus on the social structure of the professional’s work in the context of the work groups (staff, peers, secretaries, etc.) is also possible.

A key question in conducting these interviews is: are we learning anything useful about our interviewees? The answer to this question depends on our goals. We aim at developing requirements for providing software to support (and hopefully improve) the work of those whom we interview. We assume that we understand the professionals’ work well enough to provide useful design guidance in the form of requirements, or anticipated problems with the current design. In our experience the full value of this field work comes from being embedded in a broader empirical design process that also seeks to measure success in terms of measurable or verifiable objectives. We presume our assessments are valid if we help develop something useful, a judgment that can sometimes be problematic when so many uncontrolled factors contribute to the form and success of a prototype.

A challenge we face in working within a usability engineering framework is developing measurable objectives, and assessments for these objectives, that are ecologically valid, and that make contact with the information and understanding we gain from field studies. Standard approaches to usability engineering specify objectives measurable in the laboratory. Yet, as we have argued, field studies are necessary for understanding the users for whom we are developing software: We don’t see how lab studies could substitute. Indeed, the business professionals that we have been studying are unlikely to participate in laboratory studies. But even if they would, we don’t know to what extent defining and evaluating measurable objectives in the lab can provide valid indicators of the usefulness and usability of an evolving system. Should objectives for usability engineering be defined and measured in the field? Or can objectives defined and measured in the lab be shown to be reasonable stand-ins for aspects of usability in the field?

Bibliography

Bjerknes. G., Ehn, P. and Kyng, M. (Eds). Computers and Democracy: A Scandinavian Challenge. Gower Press, Brookfield, VT, 1987.

Boland, R. J., Jr. Phenomenology: A preferred approach to research on information systems. In Research Methods in Information Systems, E. Mumford et al., Eds., North-Holland, Amsterdam, 1985, pp. 193-201.

Campbell, R. L. Evaluating online assistance empirically. IBM Research Report RC 13410, Yorktown Heights, NY, 1988.

Carroll, J. M. and Campbell, R. L. Softening up hard science: Reply to Newell and Card. Human-Computer Interaction, 2, 1986, 227-249.

De Rivera, J. Conceptual Encounter: A Method for the Exploration of Human Experience. University Press of America, Washington, 1981.

Dreyfus, H. L. and Dreyfus, S. E. Mind over Machine. The Free Press, New York, 1986.

Ehn, P. Work-Oriented Design of Computer Artifacts. Arbetslivscentrum, Stockholm, 1988 (distributed by Almqvist & Wiksell International).

Galliers, R. D. and Land, F. F. Choosing appropriate information systems research methodologies. Communications of the ACM, 30 (11), November 1987, 900-902.

Gould, J. D. How to design usable systems. To appear in M. Helander (Ed.), Handbook of Human-Computer Interaction, North-Holland, Amsterdam.

Graesser, A. and Murray, K. A question answering methodology for exploring a user’s acquisition and knowledge of a computer environment. In S. Robertson, W. Zachary, and J. Black (Eds.), Cognition, Computing, and Interaction, Ablex, Norwood, NJ, 1988.

Hollan, J., Miller, J. R., Rich, E. and Wilner, W. Knowledge bases and tools for building integrated multimedia intelligent interfaces. Proceedings of the ACM/SIGCHI Workshop on Architectures for Intelligent Interfaces: Elements and Prototypes, (Monterey, California, March 29 – April 1, 1988), 19-38.

Kato, T. What “question-asking protocols” can say about the user interface. International Journal of Man-Machine Studies, 25, 1986, 659-673.

Lewis, C. Using the “thinking-aloud” method in cognitive interface design. IBM Research Report RC 9265, Yorktown Heights, NY, 1982.

Mack, R. and Nielsen, J. Software integration in the professional work environment: Observations on requirements, usage and interface issues. IBM Research Report RC 12677, Yorktown Heights, NY, 1987.

Nielsen, J., Mack, R. L., Bergendorff, K. H. and Grischkowsky, N. L. Integrated software usage in the professional work environment: Evidence from questionnaires and interviews. Proceedings of the CHI ’86 Conference on Human Factors in Computing Systems, (Boston, April 13-17, 1986), 162-167.

Suchman, L. A. Plans and Situated Actions. Cambridge University Press, Cambridge, 1987.

Whiteside. J., Bennett, J. and Holtzblatt, K. Usability engineering: Our experience and evolution. In M. Helander (Ed.), Handbook of Human-Computer Interaction, North-Holland, Amsterdam, 791-817, 1988.

Winograd, T. and Flores, F. Understanding Computers and Cognition: A New Foundation for Design. Ablex: Norwood, NJ, 1986.

Wixon, D. and Good, M. Interface style and eclecticism: Moving beyond categorical approaches. Proceedings of the Human Factors Society 31st Annual Meeting, (New York, October 19-23, 1987), Vol. 1, 571-575.