Humans should be able work more effectively with artificial intelligence-based systems when they can predict likely failures and form useful mental models of how the systems work. We conducted a study of human's mental models of artificial intelligence systems focusing on a high-performing image classifier. Participants viewed individual labeled images in one of three general classes and then tried to predict whether the classifier would label it correctly. Participants were able to begin performing this task at levels much better than chance, 69% correct. However, after 137 trials with feedback, their performance improved a small, but statistically significant amount to 73%. Analysis of these results and comments indicated that humans were using their own perceptions of the images as first-approximation proxies. 'Projecting' human characteristics onto a computer might be considered a cognitive bias, but in this task, the strategy seemed to yield good initial results. This might be called effective anthropomorphism. Participants sometimes used this strategy both implicitly and explicitly. The paper includes discussion of why this strategy might have worked better than alternatives, why further learning was quite difficult, and what assumptions about similarities between human perception and image classification systems may in fact be correct.

Deep learning has emerged as a powerful tool for feature-driven labeling of datasets. However, for it to be effective, it requires a large and finely-labeled training dataset. Precisely labeling a large training dataset is expensive, time consuming, and error-prone. In this paper, we present a visually-driven deep learning approach that starts with a coarsely-labeled training dataset, and iteratively refines the labeling through intuitive interactions that leverage the latent structures of the dataset. Our approach can be used to (a) alleviate the burden of intensive manual labeling that captures the fine nuances in a high-dimensional dataset by simple visual interactions, (b) replace a complicated (and therefore difficult to design) labeling algorithm by a simpler (but coarse) labeling algorithm supplemented by user interaction to refine the labeling, or (c) use low-dimensional features (such as the RGB colors) for coarse labeling and turn to higher-dimensional latent structures, that are progressively revealed by deep learning, for fine labeling. We validate our approach through use cases on three high-dimensional datasets and a user study.

Anyone in this business knows how fun and exciting hacking can be, but also the emotional and physical toll it can take. Mental health is a longstanding dirty secret in the infosec community, and we are just now learning how to talk about it. The wear and tear of everyday stress combined with an 'always on' aspect of an operational environment creates a perfect storm for burning out. While stress can have a negative impact on job performance, my primary concern is on the health and safety of infosec professionals themselves. Not only does stress have short term effects on cognitive abilities and performance, but recurrent acute stress can have long term effects on health (mental and physical) as well as burnout and turnover. There are many sources of stress in infosec operations, some of which can be managed while others are simply the nature of the job. Activities that require long periods of vigilance and creativity will deplete cognitive resources and increase fatigue. Some of these activities have unpredictable results that can increase frustration. Other times, external factors unrelated to the activity itself may introduce new sources of stress that are not normally present. A certain level of stress is to be expected in these operations because they are considerably difficult, have a high risk vs. reward trade-off, and require a significant amount of knowledge and skill. But, how much stress can you take on and still be a happy hacker? In this talk I will discuss why infosec is so stressful, how this stress affects you and your network, and some things you can do about it. I will also discuss lessons learned from my research study of tactical cyber operations that studied fatigue, frustration, and cognitive workload in operators.

What does it mean for humans and machines to work together effectively on complex analytic tasks? Is human teaming the right analogue for this kind of human-machine interaction? In this paper, we consider behaviors that would allow next-generation machine analytic assistants (MAAs) to provide context-sensitive, proactive support for human analytic work - e.g., awareness and understanding of a user's current goals and activities, the ability to generate flexible responses to abstractly-formulated needs, and the capacity to learn from and adapt to changing circumstances. We suggest these behaviors will require processes of coordination and communication that are similar to but at least partially distinguishable from those observed in human teams. We also caution against over-reliance on human teaming constructs and instead advocate for research that clarifies the functions these processes serve in enabling joint activity and determines the best way to execute them in specific contexts.

Darknet Markets are a hotbed of illicit trade and are difficult for law enforcement to monitor and analyze. Topic Modeling has been a popular method to semantically analyze market listings, but lacks the ability to infer the information-rich visual semantics of images embedded within these listings. In this paper we present a relatively fast method using unsupervised and self-supervised machine learning methods to infer image semantics from large, unstructured multimedia corpora, and demonstrate how it may aid analysts in investigating the content of Darknet Markets.

We present a new and novel graph visualization technique designed specifically for virtual reality (VR). Ring graphs organize graph nodes by categorical attributes along a ring that are placed in a sphere layout. Links between nodes are drawn within the rings using an edge bundling technique. This 3D placement of data takes advantage of the stereoscopic environment that VR offers. We conducted a user study that compared our ring visualization to a traditional node-based graph visualization and found that our ring graph method had higher usability, both in terms of accuracy in completing a set of tasks as well as lower task completion time.

Red and Blue cyberdefense teams provide valuable cybersecurity assessment services to help prevent and defend against malicious intruders. Through interviews, we investigated the methods, tools, and challenges of two specific U.S. Government Department of Defense Red and Blue teams and how they work together during integrated operations. We found examples of successful integration, as well as opportunities for enhanced, shared situation awareness. Based on these findings, we discuss design implications for tools that can facilitate situation awareness among multiple cyberdefense teams by supporting data fusion, change detection, network mapping, and access tracking.

Operator stress is a common, persistent, and disabling effect of cyber operations and an important risk factor for performance, safety, and employee burnout. We designed the Cyber Operations Stress Survey (COSS) as a low-cost method for studying fatigue, frustration, and cognitive workload in real-time tactical cyber operations. The combination of pre- and post-operational measures with well validated factors from the NASA Task Load Index and additional contextual factors provide a quick, easy, and valuable assessment of cognitive stress. We report on our experiences developing and fielding the survey instrument, validation, and describe the use and results of the COSS in four studies of cyber operations across the National Security Agency.

Visual analytic systems, especially mixed-initiative systems, can steer analytical models and adapt views by making inferences from users’ behavioral patterns with the system. Because such systems rely on incorporating implicit and explicit user feedback, they are particularly susceptible to the injection and propagation of human biases. To ultimately guard against the potentially negative effects of systems biased by human users, we must first qualify what we mean by the term bias. Thus, in this chapter we describe four different perspectives on human bias that are particularly relevant to visual analytics. We discuss the interplay of human and computer system biases, particularly their roles in mixed-initiative systems. Given that the term bias is used to describe several different concepts, our goal is to facilitate a common language in research and development efforts by encouraging researchers to mindfully choose the perspective(s) considered in their work.

The goal of our research was to understand the effects of display size on interaction zones as it applies to interactive systems. Interaction zone models for interactive displays are often static and do not consider the size of the display in their definition. As the interactive display ecosystem becomes more size diverse, current models for interaction are limited in their applicability. This paper describes the results of an exploratory study in which participants interacted with and discussed expectations with interactive displays ranging from personal to wall-sized. Our approach was open-ended rather than grounded in existing interaction zone models in order to explore potential differences in interaction zones and distances. We found that the existence of different interaction zones and the distance at which these zones are relevant are dependent on display size. In discussion of the results, we explore implications of our findings and offer guidelines for the design of interactive display systems.

Hacking is a high-risk, high-reward, with a high-cost to human capital. In this session, we will talk about the effects of human factors in cyber operations and why you should care about them. Specifically, we will focus on results of research at the National Security Agency that studied the effects of cognitive stress on tactical cyber operators. A key motivation for this work was the intuition that cognitive stress may negatively affect operational security, work performance, and employee satisfaction. Operator fatigue, frustration, and cognitive workload increases significantly over the course of a tactical cyber operation. Fatigue and frustration are correlated, and as one increases so does the other. The longer the operation, the greater the mental demand, physical demand, time pressure, frustration, and overall effort needed to complete the operation. Operations longer than 5 hours have 10% greater increases in fatigue and frustration compared to shorter operations. We found no link of performance to operation length; that is, from the operator's perspective longer operations did not result in higher success. Knowing how these factors affect cyber operations has helped us make more informed decisions about mission policy and workforce health. We hope that by sharing this with the greater Black Hat community, they will also be able to learn from our study and improve their own cybersecurity operations.

TexTonic is a visual analytic system for interactive exploration of very large unstructured text collections. TexTonic visualizes hierarchical clusters of representative terms, snippets, and documents in a single, multi-scale spatial layout. Exploration is supported by interacting with the visualization and directly manipulating the terms in the visualization using semantic interactions. These semantic interactions steer the underlying analytic model by translating user interactions within the visualization to contextual updates to the supporting data model. The combination of semantic interactions and information visualization at multiple levels of the data hierarchy helps users manage information overload so that they can more effectively explore very large text collections. In this article, we describe TexTonic's data processing and analytic pipeline, user interface and interaction design principles, and results of a user study conducted mid-development with experienced data analysts. We also discuss the implications TexTonic could have on visual exploration and discovery tasks.

Social network graphs are often used to help inform judgments in a variety of domains, such as public health, law enforcement, and political science. Across two studies, we examined how graph features influenced probabilistic judgments in graph-based social network analysis and identified multiple heuristics that participants usedto inform these judgments. Study 1 demonstrated that participants' judgments were influenced by information about direct connections, base rates, and layout proximity, and participants' self-reported strategies also reflected use of this information. Study 2 replicated findings from Study 1 and provided additional insight into the hierarchical ordering of these strategies and the decision process underlying judgments from social network graphs.

Visual analytic systems, especially mixed-initiative systems, can steer analytical models and adapt views by making inferences from users; behavioral patterns with the system. Because such systems rely on incorporating implicit and explicit user feedback, they are particularly susceptible to the injection and propagation of human biases. To ultimately guard against the potentially negative effects of systems biased by human users, we must first qualify what we mean by the term bias. Thus, in this paper we describe four different perspectives on human bias that are particularly relevant to visual analytics. We discuss the interplay of human and computer system biases, particularly their roles in mixed-initiative systems. Given that the term bias is used to describe several different concepts, our goal is to facilitate a common language in research and development efforts by encouraging researchers to mindfully choose the perspective(s) considered in their work.

While the human factors of mission critical systems such as air traffic control and weapons systems have been extensively studied, there has been little work on cyber operations. As with any system, the perfect storm of complex tasks in a high-risk environment takes an incredible toll on human operators, leading to errors, decreased performance, and burnout. An extensive study of tactical cyber operations at the National Security Agency found that operatorfatigue, frustration, and cognitive workload significantly increase over the course of an operation. A discussion of these findings helps us understand the impact that the high-stress, high-risk environment of tactical cyber operations has on its operators.

Cyber defense requires decision making under uncertainty, yet this critical area has not been a focus of research in judgment and decision-making. Future defense systems, which will rely on software-defined networks and may employ "moving target" defenses, will increasingly automate lower level detection and analysis, but will still require humans in the loop for higher level judgment. We studied the decision making process and outcomes of 17 experienced network defense professionals who worked through a set of realistic network defense scenarios. We manipulated gain versus loss framing in a cyber defense scenario, and found significant effects in one of two focal problems. Defenders that began with a network already in quarantine (gain framing) used a quarantine system more, as measured by cost, than those that did not (loss framing). We also found some difference in perceived workload and efficacy. Alternate explanations of these findings and implications for network defense are discussed.

CyberPetri is a novel visualization technique that provides a flexible map of the network based on available characteristics, such as IP address, operating system, or service. Previous work introduced CyberPetri as a visualization feature in Ocelot, a network defense tool that helped security analysts understand and respond to an active defense scenario. In this paper we present a case study in which we use CyberPetri to support real-time situation awareness during the 2016 Cyber Defense Exercise.

Most cyber security research is focused on detecting network intrusions or anomalies through the use of automated methods, exploratory visual analytics systems, or real-time monitoring using dynamic visual representations. However, there has been minimal investigation of effective decision support systems for cyber analysts. This paper describes the user-centered design and development of a decision support visualization for active network defense. Ocelot helps the cyber analyst assess threats to a network and quarantine affected computers from the healthy parts of a network. The described web-based, functional visualization prototype integrates and visualizes multiple data sources through the use of a hybrid space partitioning tree and node link diagram. We describe our design process for requirements gathering and design feedback which included expert interviews, iterative design, and a user study.

The design-first approach focuses on the initial design of a visualization without the overhead of data and system requirements. Two case studies that used this technique show how a nontechnical approach can be successful at eliciting thought-provoking visualization solutions without the need for sample data.

Our research explores how interruptive notifications support task management in a desktop environment. We conducted two user studies with a community of open source software users and developers to explore their experience with interruptive notifications. We found that certain kinds of notifications support multitasking, task prioritization, task management, as well as influence task disruption management. We discuss how these behaviors affect the notification-task management user experience and offer design guidelines derived from these results to inform better design of systems that interrupt through notification.

Randomly browse nearly any publication from nearly any domain in today’s information-saturated world and you’ll read about Big Data. The term permeates scientic journals, technical magazines, newspapers, social sciences, and to some extent pop culture. Many areas of science and government have been wrestling with Big Data for years. However, as the information age fully radiates into nearly all areas of society and application domains, Big Data problems have pushed into these areas as well. It is common to hear people simultaneously propose that they have a Big Data problem as well as a Big Data opportunity

The goal of our research was to understand how knowledge workers use community-curated knowledge and collaboration tools in a large organization. In our study, we explored wiki use among knowledge workers in their day-to-day responsibilities. In this poster, we examine the motivation and rewards for knowledge workers to participate in wikis through the economic idea of costs to contribute.

The goal of the 1st Workshop on Human-Centered Big Data Research was to explore the multi-disciplinary challenges of researching humans in Big Data environments. This paper summarizes the outcomes of the workshop and aims to define potential future work in this area.

This work-in-progress shows a relationship between future use of technology and emotional experience and describes how this relationship can be used as a measurement of the user experience. We found a consistent relationship between future use of technology and emotional experience under different contextual scenarios in two studies of the interruptive notification user experience. Implications for research and future work are also discussed.

Network operations centers are notoriously difficult places to conduct human-centered research. The intense pace and sensitive information environment creates a number of hurdles for researchers. This paper shares the experiences from human-centered research of a government network operations center. The lessons learned from conducting interviews, field observations, and a card sorting study offer guidance to those who may study network operations centers in the future.

A graph visualization offers a novel way to analyze and understand the relationships between cards and the mental models elicited in a card-sorting study. Graph visualizations are graphs that illustrate connections between concepts, such as cards in a card-sorting study. A visualization can quickly show relationships between cards and clusters of cards that represent topics that may not be obvious from traditional card-sort analysis methods. A case study describes how graph visualization can be used to analyze the data. The results of the analysis are compared and contrasted with a popular histogram-matrix analysis method. Strengths and weaknesses of the proposed graph visualization analysis method are discussed.

The 2013 Visual Analytics Science and Technology (VAST) Challenge presented three distinct challenge problems. Mini-Challenge 1 (MC1) asked participants to use visual analytics to predict the success of new movies. Mini-Challenge 2 (MC2) focused on the design of a situation awareness display for monitoring the health, performance, and security of a large computer network. Mini-Challenge 3 (MC3) requested participants to identify the timeline of important network events in two weeks of network data for a ficticious marketing company. The VAST Challenge received 31 final submissions, with MC1 receiving 106 interim submissions over the challenge period. Participants came from 11 different countries, and 14 awards were given.

This poster abstract presents CyberSAVI, a Situation Awareness Visual Interface that provides mission management level situation awareness of the health and status of a network and the people assigned to investigate irregular network events. CyberSAVI focuses on supporting the interaction and coordination between people to provide situation awareness of suspected, known, and on-going network events. The coordinating poster demonstrates how CyberSAVI accomplishes these goals through the description of a realistic network security use case.

This paper offers insights to how cyber security analysts establish and maintain situation awareness of a large computer network. Through a series of interviews, observations, and a card sorting activity, we examined the questions analysts asked themselves during a network event. We present the results of our work as a taxonomy of cyber awareness questions that represents a mental model of situation awareness in cyber security analysts.

Interruptive notifications in a desktop environment are an important service that knowledge workers rely on for maintaining awareness of information and services outside their current focus. Research to date has focused primarily on empirical laboratory-based testing, which is very specific and out of context of a realistic user environment, and broad ethnographic research, which is not specific enough for meaningful notification system design guidelines. This dissertation aims to address the gap between existing laboratory-based and ethnographic research by conducting a series of studies that explored the notification user experience in a both broad and deep way. The results of this research contribute to the following: A catalog of significant contextual factors that affect the notification user experience; a series of models that describe how factors in the notification system context influence the overall user experience; a set of design guidelines, derived from this research but generalized to be applicable to any interruptive notification system.

This report presents the findings of a usability study by NIST's Visualization and Usability Group that examines the usability aspects of smartcards mandated by HSPD-12, as well as users' perceptions and behavior regarding smartcards. We were interested in how users would use smartcards in their everyday work processes; how their work processes might change to accommodate smartcard use; and finally, the benefits and drawbacks they perceive in using smartcards for authentication.

The 2012 Visual Analytics Science and Technology (VAST) Challenge posed two challenge problems for participants to solve using a combination of visual analytics software and their own analytic reasoning abilities. Challenge 1 (C1) involved visualizing the network health of the fictitious Bank of Money to provide situation awareness and identify emerging trends that could signify network issues. Challenge 2 (C2) involved identifying the issues of concern within a region of the Bank of Money network experiencing operational difficulties utilizing the provided network logs. Participants were asked to analyze the data and provide solutions and explanations for both challenges. The data sets were downloaded by nearly 1100 people by the close of submissions. The VAST Challenge received 40 submissions with participants from 12 different countries, and 14 awards were given.

This paper explores the relationship between emotion and the notification experience. We found a strong relationship between the user emotions used to describe interruptive notification experiences and whether the users wanted similar interruptive notifications again in the future. Participants were likely to want similar future interruptive notifications if they described their interruptive notification experiences using positive words. They were likely to not want similar future interruptive notifications if they described their interruptive notification experiences using negative words. The implications for the use of this knowledge in the design of intelligent systems and potential for future work are also discussed.

There are some ways you can make usability testing work in the open source community. Throughout my career in open source, I have run a number of usability tests, and not all have been the conventional laboratory-based testing you often think of when you hear 'usability test.' These three examples help descrive the different ways usability testing can be conducted and how it can fit into the open source community.

A field study of 24 participants over 10 weeks explored user behavior and perceptions in a smartcard authentication system. Ethnographic methods used to collect data included diaries, surveys, interviews, and field observations. We observed a number of issues users experienced while they integrated smartcards into their work processes, including forgetting smartcards in readers, forgetting to use smartcards to authenticate, and difficulty understanding digital signatures and encryption. The greatest perceived benefit was the use of an easy-to-remember PIN in replacement of complicated passwords. The greatest perceived drawback was the lack of smartcard-supported applications. Overall, most participants had a positive experience using smartcards for authentication. Perceptions were influenced by personal benefits experienced by participants rather than an increase in security.

This paper describes a post-hoc analysis of the relationship between the socialness of an interruptive notification and the emotional tone of the words used to describe the experience through a One-Word-Response (OWR). Out of the 89 responses analyzed, 73% of participants used emotional words to describe their notification experiences. There was a significant relationship between the emotional tone of a OWR response and the socialness of an interruptive notification experience and participants were 3.2 more likely to describe social interruptive notifications with positive words than negative words.

In this workshop position paper, two case studies of alternative ways to conduct usability testing in Free/Libre/Open Source Software (FLOSS) projects are described. The first case study involves making the usability test a university project and using students to conduct the testing. The second case study involves using members of the open source user community to help organize the usability test, recruit participants, and conduct the usability test. These two case studies provide a beginning point for discussing ways of adjusting traditional usability methods for FLOSS practices.

There are a number of ways developers try to explain new user interface ideas to the community: ACII art, scanned napkin drawings, Qt designer mockups, annotated screenshots, etc. Some of these methods get the message across better than others. When you try to add details such multiple states, interaction, or animation effects, an image which was meant to help could end up confusing your audience even more. As a designer, I am often asked which tools I use in order to create mockups or annotate existing user interfaces. Although there are some specialized tools for interaction design, the tools I use the most are every day open source software. There are many tips and tricks on how to use these tools to convey user interface ideas. This presentation will explain how to use common open source tools to more effectively communicate your design ideas to the community.

Who says you need a lab in order to do usability testing? The lab provides a stable environment to do serious usability testing, but sometimes you just need to get out there and test a few ideas, not prove a theory. Open source needs more quick and dirty design methods, and this is one of them! This lightning talk will describe the differences between traditional lab testing and quick and dirty testing, give examples of previous testing projects, guidelines for running a test.

A review of case studies about usability in eight Free/Libre/Open Source Software (FLOSS) projects showed that an important issue regarding a usability initiative in the project was the lack of user research. User research is a key component in the user-centered design (UCD) process and a necessary step for creating usable products. Reasons why FLOSS projects suffered from a lack of user research included poor or unclear project leadership, cultural differences between developer and designers, and a lack of usability engineers. By identifying these critical issues, the FLOSS usability community can begin addressing problems in the efficacy of usability activities and work towards creating more usable FLOSS products.

There are some ways you can make usability testing work in the open source community. Throughout my career in open source, I have run a number of usability tests, and not all have been the conventional laboratory-based testing you often think of when you hear 'usability test.' These three examples help descrive the different ways usability testing can be conducted and how it can fit into the open source community.

Although the KDE Usability Project has been around for over 5 years, is KDE much further in terms of usability as it was a few years ago? What are some of the factors which have limited the progress of such an important initiative? In this presentation I will give a state of the KDE Usability Project, including where we came from, what we are doing, and where we are going. I will review some of the history and the motivation for having a usability project in KDE, describe how the project operates with examples of recent and current usability projects (including our successes and failures), and finally, discuss organizational-level changes which we hope will better serve KDE4 and lead us to a more sustainable usability initiative. The KDE community will take away a better understanding of the project which will help fuel a better and stronger KDE Usability Project.

The Modified-Delphi card sort is a pre-design information architecture method which saves time and money, and increases the quality of results gathered. This paper reviews the research basis of the method and provides a practical guide for practitioners who wish to conduct a Modified-Delphi card sorting study.

Open card sorting is used by information architects to gather insights from users to incorporate feedback into aninformation architecture. In theory, it is one of the more inexpensive, user-centered design methods available to practitioners, but hidden costs make it less likely to be conducted properly and affect the quality of results produced. The following proposes a new card sorting method called the Modified-Delphi card sort to replace the Open card sort. The Modified-Delphi card sort is based on a well-known forecasting technique called the Delphi method. Instead of producing individual models that are then analyzed as a whole, participants work with a single model that is proposed and modified throughout the study. The Modified-Delphi card sorting method produces more useful results to aid in the design of an information architecture than the Open card sorting method.

Open source is a software licensing philosophy which believes the human readable code source of a software should be available for the public to freely install, modify, or redistribute. The term "open source" can also refer to the community and development practices of thousands of free/libre/open source software (FLOSS) projects who subscribe to this philosophy and license their software under one of the many available software licenses.

This talk will provide an overview of common UCD methods and when they are the most practical in the development cycle, review different kinds of UCD deliverables and how their information can be used by developers, as well as use examples from exiting projects to help illustrate these points. In the course of the discussion, it will cover: A review of usability design and its methods; Different kinds of usability documents and how they can be used by developers; Different kinds of developer documents and how they can incorporate usability goals; Strategies to help developers and usability designers work together; Strategies to address issues which often come up in OSS/usability development.

In this article, we show some structural settings, or general conditions, of OSS projects and outline their effects on usability in comparison to commercial software development. We describe the challenges of realising and establishing usability in the open source realm for the future. The focus is on two factors: the integration of usability into the daily practice of OSS development, and the availability of usability resources (experts, knowledge, time, availability). If this can be achieved, OSS has an excellent chance of making usability a key market advantage.

This is a report on a collection of research gathered on user's information management habits in order to help develop an alternative to KMenu. The presentation will focus on low-level organization and management principles rather than high-level interface interaction. I will also apply these principles to several low-fidelity prototypes to give examples of possible applications of these principles in KDE.