slim:classes:810:quiz

Question 1 8 / 8 pts List four common pitfalls in research and, in your own words, say why these affect the findings and what researchers might do to avoid each particular mistake. (So, no quotations in this answer, please.)

Your Answer: Failure to understand what’s missing – The veil of ignorance

This is probably the most fundamental pitfall in research because it’s based around something missing, unclear, or unknown to the researcher. When a researcher decides to examine a topic, they are endowed with a certain level of knowledge and understanding that informs what they “see” and how they “query” the world to get their answers. There exists a veil of ignorance covering the research topic to be examined (i.e., this is part of the reason why the researcher is curious about it and wants to study it) and, more fundamentally, there is a veil of ignorance of the researcher (i.e., what don’t they know about the topic, the methodologies of research, best practices, etc.). The veil of ignorance is greater for a novice researcher than it is for an experienced researcher. Because knowing what we don’t know or don’t recognize is extremely difficult to overcome, epistemological bias is a fundamental, existential form of bias. This can result in a failure to adequately develop the research design to be able to measure what the researcher wants to measure, to apply the wrong methodological approach, to frame and word requests for information poorly or in a biased fashion, or to ultimately fail to ask a meaningful research question. To correct for this type of bias, an individual researcher should educate themselves, become part of a community of researchers, and rigorously and ruthlessly question what they know and how they know it. So far as is known, the methods of science are the best general approach to piercing the veils of ignorance.

Failure to examine representativeness

This pitfall arises from not understanding the representativeness of a population sample (what group is part of your population but not represented in your sampling design) or by failing to recognize who didn’t answer your question and what may be possibly different about them (selection bias). The result of this pitfall will be under- or non- represented populations in the study leading to a lack of validity for the study and biased results. Correcting for this pitfall one may try to follow up with non-respondents to see how they differ from respondents as well as looking for absential (forgotten) groups/features that are causal in relation to the research study.

Being misled by numbers

This pitfall arises from a lack of skill in dealing with statistical techniques and measures and is very common even among senior researchers. Not understanding the meanings of statistical measures of significance (or even what significance means), the biases of certain types of numerical methods, or the failure to establish baselines can lead researchers into overgeneralizations and even incorrect conclusions. This pitfall can be further compounded by the conversion of numerical data into charts, graphs, or visualizations. The best corrective for this pitfall is getting a review from a trained statistician as well as being cautious when drawing conclusions.

Overgeneralization of conclusions

This pitfall arises from a failure to confuse causality and correlation as well as confirmation basis and a lack of caution when evaluating as well as failing to consider what might be missing from the study. Correcting for this pitfall should first of all be a humble and cautious approach to the data along with a review of the study for confirmation bias. Avoid drawing conclusions broader than is warranted by the population under study.

Question 2 4 / 4 pts You have to describe bibliometrics to an audience of non-librarians. Use your own words to describe what bibliometrics is and why it is a useful tool for librarians and archivists.

Your Answer: What is bibliometrics?

Bibliometrics is the study of recorded/published forms of communication – including books, web pages, periodicals, databases, broadcasts, etc. – that examines non-intrinsic, countable properties of these information sources such as publication type, format, subject matter, author, publication source, etc. Intrinsic properties are the domain of a different methodology called content analysis. The difference between a bibliometric analysis and an index, is that unlike indexes which are merely tallies of information sources and their extrinsic facets, bibliometric analyses seek to express something about the relationships between information sources, their creators/authors, their distribution channels, their influence, their replicability, their hierarchical distribution as related to various bibliometric laws (Zipf’s, Bradford’s, Lotka’s, etc.) as well as the patterns/characteristics/traits that can be discovered in terms of their usage by various groups. Bibliometrics relies on techniques of numerical analysis including frequency counts, percentage factors, and averages (mean, mode, median).

There are four broad categories of bibliometric studies according to Beck and Manuel (pp. 167-168):

Studies that seek to learn about information sources (i.e., what is a source’s relative value) Studies that seek to learn about institutional trends (i.e., how a collection of sources is used or affected by institutional practices) Studies that seek to learn about people behaviors (i.e., who uses the source and how do they use it; who authors sources) Studies that seek to learn about socio-intellectual phenomenon (i.e., how a source can illuminate trends in academic practice)

Common relationship properties that are examined in bibliometric studies include:

Citation - references made from one source to another Impact factor – relative importance of a source Immediacy index – the velocity of source discovery and citation Hyperlinks – links between sources Library holding – institutional valuation of source worth Circulation counts – proxy of source usage (Beck & Manuel, 176)

Bibliometrics is a useful tool for librarians and archivist to aid in collection development and understanding patterns of collection usage by patrons as well as providing a good introductory field of study for beginning researchers. It can also reveal unique properties of a particular collection.

Question 3 4 / 4 pts Describe the limitations of action research, as discussed by Beck and Manuel. Do you think these limitations are greater or lesser in the library context? Why?

Your Answer: As discussed by Beck and Manuel, action research is constrained by a limited problem scope (it must be solvable), a narrow definition of the actionable goal (improvement in service or practice), internal institutional focus (not generalizable), and subject to numerous stakeholder interests.

Action research in a library context is constrained differently than other institutions because of the types of stakeholders who can be involved in the problem-solving process. Patrons, staff, government entities, library boards, and professional organization such as the ALA which each may impose different constraints and constraint weightings to library problems - balancing the local standards of the community against national standards that may be broader against legal constraints of particular patron segments as well as the resource constraints and personal beliefs / character of staff as well the board designated policy guidelines that must also be navigated to reach a consensus of practice or problem solution that balances all the inputs without compromising the proposed intervention’s effectiveness and the worth of the effort to change existing practices. The stakeholder constraints may be greater or lesser depending on the degree of oversight provided.

The library as an institution is constrained by competing demands, expectations, legal rules, and power relationships whether it is a humble school library, public library branch, and prestigious academic library attached to a law, medical, or business school which is further complicated by the fact that a majority of libraries functions as subordinate parts of larger institutions, serving the needs of the school (elementary, secondary, academic) or institution in the case of business, prison and museums. In this context, the public library acts with somewhat more autonomy as a self-directed institution – but of course, still faces oversight in the form of the library board and funding bodies.

Beck, S. E., & Manuel, K. (2008). Practical research methods for librarians and information professionals. New York: Neal-Schuman Publishers.

Question 4 6 / 6 pts What different grouping of samples does Pyrczak discuss in Chpaters 6 & 7? Why are the evaluation strategies and questions different for each group? Explain.

Your Answer: Pyrczak divides the study groups between those that are generalizing and those that are not.

Stated simply, they serve two different ends and therefore the constraints that are relevant for their evaluation are different. In the generalizing studies, the constraints are derived from methodological soundness based on the metrics of the accepted best practice of population sampling. Population adequacy, randomness of selection, and statistical techniques to avoid bias. Of the four types of non-generalized studies, one type (pilot study – convenience sampling) is an attempt to develop constraints for future research and is designed as an experimental foray into an area where normative expectations have not been measured and are in need of defining – thus guiding future research projects; one type (theory development and testing – convenience sampling) tries to evaluate the predictive value of a theory with convenience samples and measure its empirical validity; one type (qualitative studies) use deliberate purposive samples and don’t rely on generalization, and finally, one type (entire population studies) does use make use of samples at all in its design.

Pyrczak, F. (2014). Evaluating research in academic journals: A practical guide to realistic evaluation. (5th ed.). Glendale, CA: Pyrczak Publishing.

Question 5 6 / 6 pts Discuss some of the ways in which we can reduce bias in our writing, and describe the bias each way avoids.

Your Answer: Why should we reduce bias in our writing?

In a research context, the most obvious reason for reducing bias is to let the researcher and readers of the research get as close to the meaning/truth of the object being studied as possible without being diverted by prejudicial language, conflicts of interest, or cognitive distortions leading to misleading conclusions.

Prejudicial Language

The APA Publication Manual address prejudicial language in section 3.

Scientific writing must be free of implied or irrelevant evaluation of the group or groups being studied. As an organization, APA is committed both to science and to the fair treatment of individuals and groups, and this policy requires that authors who write for APA publications avoid perpetuating demeaning attitudes and biased assumptions about people in their writing. Constructions that might imply bias against persons on the basis of gender, sexual orientation, racial or ethnic group, disability or age are unacceptable (APA, pp. 70-71).

Although the APA uses the term bias here, I think prejudice would have been a better term to use in this context since the bias under discussion is negative in tone. The APA Publication Guide then recommends three general guidelines for reducing prejudicial labeling of people and striking a more neutral tone: 1) Describe at the appropriate level of specificity, 2) be sensitive to labels, 3) acknowledge participation. Specific recommendations for handling prejudicial labeling by topic addresses in turn: 1) gender, 2) sexual orientation, 3) racial and ethnic identity, 4) disabilities, 5) age, and 6) historical and interpretive inaccuracies (APA, pp 71-77). Reducing prejudicial labeling will make an article more publishable and mark the author as a non-bigoted person.

Conflict of interest

This type of bias can arise when a research author doesn’t point out potential conflicts of interest in their writing. Failure to disclose a funding source or showing favoritism to a theory or other research in order to benefit professionally. Pointing out potential sources of conflict can enhance the perceived credibility of the author’s work.

Cognitive distortions

Authors should also correct their writing for ontological bias. Overly favoring what is seen to what is not seen. Pointing out what is not shown or known is a good corrective to over-generalizations in research reporting. This usually covered in the areas that describe limitations of the research design, methodology, theory foundation, etc.

American Psychological Association. (2010). Publication manual of the American Psychological Association. Washington, D.C.: American Psychological Association.

Question 1 8 pts List four common pitfalls in research and, in your own words, say why these affect the findings and what researchers might do to avoid each particular mistake. (So, no quotations in this answer, please.)

Failure to understand what’s missing – The veil of ignorance This is probably the most fundamental pitfall in research because it’s based around something missing, unclear, or unknown to the researcher. When a researcher decides to examine a topic, they are endowed with a certain level of knowledge and understanding that informs what they “see” and how they “query” the world to get their answers. There exists a veil of ignorance covering the research topic to be examined (i.e., this is part of the reason why the researcher is curious about it and wants to study it) and, more fundamentally, there is a veil of ignorance of the researcher (i.e., what don’t they know about the topic, the methodologies of research, best practices, etc.). The veil of ignorance is greater for a novice researcher than it is for an experienced researcher. Because knowing what we don’t know or don’t recognize is extremely difficult to overcome, epistemological bias is a fundamental, existential form of bias. This can result in a failure to adequately develop the research design to be able to measure what the researcher wants to measure, to apply the wrong methodological approach, to frame and word requests for information poorly or in a biased fashion, or to ultimately fail to ask a meaningful research question. To correct for this type of bias, an individual researcher should educate themselves, become part of a community of researchers, and rigorously and ruthlessly question what they know and how they know it. So far as is known, the methods of science are the best general approach to piercing the veils of ignorance.

Failure to examine representativeness This pitfall arises from not understanding the representativeness of a population sample (what group is part of your population but not represented in your sampling design) or by failing to recognize who didn’t answer your question and what may be possibly different about them (selection bias). The result of this pitfall will be under- or non- represented populations in the study leading to a lack of validity for the study and biased results. Correcting for this pitfall one may try to follow up with non-respondents to see how they differ from respondents as well as looking for absential (forgotten) groups/features that are causal in relation to the research study.

Being misled by numbers This pitfall arises from a lack of skill in dealing with statistical techniques and measures and is very common even among senior researchers. Not understanding the meanings of statistical measures of significance (or even what significance means), the biases of certain types of numerical methods, or the failure to establish baselines can lead researchers into overgeneralizations and even incorrect conclusions. This pitfall can be further compounded by the conversion of numerical data into charts, graphs, or visualizations. The best corrective for this pitfall is getting a review from a trained statistician as well as being cautious when drawing conclusions.

Overgeneralization of conclusions This pitfall arises from a failure to confuse causality and correlation as well as confirmation basis and a lack of caution when evaluating as well as failing to consider what might be missing from the study. Correcting for this pitfall should first of all be a humble and cautious approach to the data along with a review of the study for confirmation bias. Avoid drawing conclusions broader than is warranted by the population under study.

1) 4th overgeneralizing which is a founded on ontological bias, rhetorical flourishes, innumeracy, epistemological bias (in the knowledge – causative or correlation), distribution or cause over-generalizing a context, by analogy, superficial similarity, or not taking a long enough timeframe for scope of confusing the accidental for the fundamental (fashion versus functional constraint). Over-generalization to conclusions not directly supported by the research data. ||| failure to ask, correlation or causation, cardinal rule of cautiousness, smallest gap possible, principle of charity, overly critical of “reality” of research, attitude of discussion after finding

Question 2 4 pts You have to describe bibliometrics to an audience of non-librarians. Use your own words to describe what bibliometrics is and why it is a useful tool for librarians and archivists. What is bibliometrics?

There are four broad categories of bibliometric studies according to Beck and Manuel (pp. 167-168): 1) Studies that seek to learn about information sources (i.e., what is a source’s relative value) 2) Studies that seek to learn about institutional trends (i.e., how a collection of sources is used or affected by institutional practices) 3) Studies that seek to learn about people behaviors (i.e., who uses the source and how do they use it; who authors sources) 4) Studies that seek to learn about socio-intellectual phenomenon (i.e., how a source can illuminate trends in academic practice)

Common relationship properties that are examined in bibliometric studies include: • Citation - references made from one source to another • Impact factor – relative importance of a source • Immediacy index – the velocity of source discovery and citation • Hyperlinks – links between sources • Library holding – institutional valuation of source worth • Circulation counts – proxy of source usage (Beck & Manuel, 176)

Question 3 4 pts Describe the limitations of action research, as discussed by Beck and Manuel. Do you think these limitations are greater or lesser in the library context? Why?

As discussed by Beck and Manuel, action research is constrained by a limited problem scope (it must be solvable), a narrow definition of the actionable goal (improvement in service or practice), internal institutional focus (not generalizable), and subject to numerous stakeholder interests.

Question 4 6 pts What different grouping of samples does Pyrczak discuss in Chapters 6 & 7? Why are the evaluation strategies and questions different for each group? Explain.

Pyrczak divides the study groups between those that are generalizing and those that are not. Stated simply, they serve two different ends and therefore the constraints that are relevant for their evaluation are different. In the generalizing studies, the constraints are derived from methodological soundness based on the metrics of the accepted best practice of population sampling. Population adequacy, randomness of selection, and statistical techniques to avoid bias. Of the four types of non-generalized studies, one type (pilot study – convenience sampling) is an attempt to develop constraints for future research and is designed as an experimental foray into an area where normative expectations have not been measured and are in need of defining – thus guiding future research projects; one type (theory development and testing – convenience sampling) tries to evaluate the predictive value of a theory with convenience samples and measure its empirical validity; one type (qualitative studies) use deliberate purposive samples and don’t rely on generalization, and finally, one type (entire population studies) does use make use of samples at all in its design.

Question 5 6 pts Discuss some of the ways in which we can reduce bias in our writing, and describe the bias each way avoids. Why should we reduce bias in our writing? In a research context, the most obvious reason for reducing bias is to let the researcher and readers of the research get as close to the meaning/truth of the object being studied as possible without being diverted by prejudicial language, conflicts of interest, or cognitive distortions leading to misleading conclusions.

Prejudicial Language The APA Publication Manual address prejudicial language in section 3.

Conflict of interest This type of bias can arise when a research author doesn’t point out potential conflicts of interest in their writing. Failure to disclose a funding source or showing favoritism to a theory or other research in order to benefit professionally. Pointing out potential sources of conflict can enhance the perceived credibility of the author’s work.

Cognitive distortions Authors should also correct their writing for ontological bias. Overly favoring what is seen to what is not seen. Pointing out what is not shown or known is a good corrective to over-generalizations in research reporting. This usually covered in the areas that describe limitations of the research design, methodology, theory foundation, etc.

1) The first pitfall is informed by two types of constraints – that come from lack of knowledge and care when performing the framing of a research area and the questions used to collect data. A poorly framed question lacks connection/congruence to the item under observation. Inadequate knowledge of phenomenon biasing of respondents. Not asking the right question or not asking the question in the right way ||| your subjects are typically constrained by your understanding, framing, and wording of the research question Skewed by researcher’s inadequate knowledge of the phenomenon being studies or by their basis assumptions about them. What’ missing? (Bastiat) Context effects of Question framing, wording, open-vs closed, question order 2) Unrepresentative samples or not recognizing response bias – asking the right question of the right persons but also face the constraint of selection bias. What’s missing ontological bias. Using unrepresentative samples or failing to recognize possible xxx bias among those members of the sample who do respond ||| who’s forgotten? Who is represented? Selection bias, what differences are occluded?

2) Third, being misled by the numeric representation of a population into there is meaning or significance in the data. Reifying numbers into eternal verities. Be lacking skill in reading the “signs” of numbers and not applying corrective best practices and biasing your interpretation and there This especially applies to numbers converted to different representational modalities like graphs, charts, or visual representations. Not addressing data’s statistical significance or not providing baseline measures for comparison ||| Numerical measures without analysis, statistical significance, chance, dispersion, missing baseline, temporal or population, numbers change as constraints change

Pyrczak divides the research literature into two general groups – depending on whether the study researchers seek to generalize their findings or not.

The differences in evaluation strategies and questions reflects the different uses and methodological techniques that will be applied to draw inferences from the samples. Pyrcazk essentially describes 5 research projects that can be undertaken. The most typical is the research designed to draw inferences from a population sample to make generalizations about the population it draws from. Standards of best practices have been developed through years of trial and error to be the current consensus. The other 4 research project types serve purpose that are not meant to be generalizable. 2 of the project types are forms of meta-analysis methodological designed to research of their applicability. The other two also have non-generalizable goals and are more narrowly focuses research. One is based around qualitative research and the other studies cover entire populations.

Random sampling, stratified, participation rate, attempt to contact, similarity to non-participants, target group, diverse, discuss limitations, demographics, overall size, subgroups sufficiently large, informed consent

Pilot study, developing and testing theory, developmental tests of a theory, purposive samples, entire population || sufficient detailed descriptions of sample populations, relevant demographics, adequate sample size, orientation (quan. Vs qual.) adequacy, purposive sample, population identify and description, informed consent

Break a constraint. Scope is local, solvable – avoid persona bias accuracy of record keeping 4 goals – social change, empowerment of individual stakeholders, collaboration, knowledge acquisition

Not generalizable, validity and reliability can be loose, process learning, only works with solvable problems, stakeholder buy-in, reflection thought and critical self-evaluation, personal choices on display, PLAN- ACT – REPEAT – stakeholders are key to the success of projects, social mapping to aid, researcher uncertainty, timeline constrains, document, intervention, results, triangulation, matrix organizer Locally focused and solvable, practice problem, multiple stakeholders, improvement or service or process is the goal. Practical consensus building, reflective and emotionally detached, ability to manage a variety of information sources, views the research process as co-participatory, identify stakeholders, work with stakeholders, gather data using a variety of methods

Bias should also avoid would also include judging a domain or technique that an author favors in a non-neutral fashion. There will be a tendency to conflate bias with lack of neutrality and rhetorical flourishes to convey the importance or significance of research findings. Generalization of bias of findings. Intervention bias or theory bias. Non-reporting of limitation is also a source of bias in writing. Another form of bias to be corrected for would ontological bias. Overly favoring what is seen to what is not seen in research terms, this a good correction to over-generalizations and a humble/ discussion of limitation and possible unknown causal agents within the frame of the research project.

This type of bias includes: anchoring, apophenia, attribution bias, confirmation bias, framing, halo effect, self-serving bias, status quo bias

Cognitive biases • 1.1Anchoring • 1.2Apophenia • 1.3Attribution bias • 1.4Confirmation bias • 1.5Framing • 1.6Halo effect • 1.7Self-serving bias • 1.8Status quo bias

Why should we reduce bias in our writing? There are three ways to approach this idea. One is to avoid offending, denigrating or devaluations or their opposition subjects/participants which may lead researchers to make flawed conclusions in their research design, outcomes, and conclusions. This type of bias would compromise the effectiveness and quality of research.

The other reason to reduce bias in the current normative practice of the least offensive labeling practices of segments of studies population to affirm their dignity/worth and to reflect the current intellection/social xxx wherein most texts are judged analyzed through the lens of deconstructivist normative practice of viewing all texts as merely reflections of ultimately meaningless human constructs that are only reflections of unequal power relationships that are imposed consciously/unconsciously by dominant groups to control / dehumanize less dominant groups. This type of bias correctio is based around meditating power relationships and can be antithetical to intellectual progress when it goes beyond confirming courtesy and dignity and delegitimizes viewing human beings in segmented categories.

APA defines bias terms of bias towards sub-groupings of individuals based on traits like gender, race, nationality, sexual orientation, age, religion, work ethic, political affiliation, health, intelligence, etc. in an attempt to avoid bad feelings for non-dominant groups and reduce any tendency to de-humanize them whether consciously or unconsciously. Bias should also avoidance would also include judging a domain or technique that an author favors in a non-neutral fashion. There will be a tendency to conflate bias with lack of neutrality and rhetorical flourishes to convey the importance or significance of research findings. Generalization of bias of findings. Intervention bias or theory bias. Non-reporting of limitation is also a source of bias in writing.

Describe differential aspects in ways that avoid positive or negative expression of their normative value. Go beyond this and avoid the appearance of normative expression when applied to entities like people, cultures, institutions, and creeds.

Describe at the appropriate level of specificity, be sensitive to labels, acknowledge participation, gender, sexual orientation, racial/ethnic identity, disabilities, age, historical/interpretive inaccuracies.

Bibliometrics is the study of a (publication) text’s extrinsic qualities (including – type of subject content, publication type, format), the affiliation of its author, journal, institution and the situatedness of the text within in a variety of network properties including, citation, impact, immediacy, trends, library handing, circulation, but not the intrinsic content which is the subject of content analysis.

Recorded information sources and their countable characteristics (e.g., the number citations or authors. Could be institution specific (e.g., sources cited in dissertations written by students from one’s institution) but often isn’t (e.g., all chemists) ||| ability to see the “countableness” of things and their significance, willingness to invest in learning about the laws of bibliometrics, willingness to conceptualize a problem broadly

Extrinsic facts about publications, broadcasts and other forms of communication, also called scientometrics, webometrics, etc. – authors affiliation, educational credentials, geographic location (SOURCE characteristics), – number of citations or -re-shelving’s of printed materials, usage statistics on electronic resources, which sources cite and are cited by other sources – the number and types of links between pages – long history, frequency % average – rich data on communication patterns and cognitive processes ||| 4 main subgroups – info about sources, learn about institutional trends, learn about peoples behavior, socio-intellectual phenomenon, behaviors laws that constraint the field – Zipf’s Law, Matthew effect, publication bias, Lotka’s law, Bradford’s law, principle of least effort good research topic of 5 reasons